A multimodal visual recognition method for drones based on cross-scene transfer learning

Abstract

The present invention relates to the field of image recognition technology, and specifically to a multimodal visual recognition method for unmanned aerial vehicles (UAVs) with cross-scene transfer learning. The method comprises: synchronously collecting environmental parameter information and multimodal image data in real time, and predicting the scene category of the current flight area using a scene prediction model; selecting the source domain subset with the highest matching degree by calculating the matching degree between the scene category and a source domain subset in a preset source domain data set; dividing a target domain sample into multiple subdomains to obtain target subdomain data; extracting unimodal features of each image in the multimodal image data, and fusing them using a dynamic switching fusion strategy based on the quality assessment results of each image and the scene category to generate a first fused feature; accumulating the first fused feature using a sliding window mechanism, and correcting it according to the historical feature state at the previous moment to obtain a second fused feature; and performing visual recognition on the second fused feature of the current target domain using a visual recognition model trained with staged transfer learning.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a multimodal visual recognition method for drones using cross-scene transfer learning. Background Art

[0002] In recent years, with the widespread application of deep learning technologies, particularly convolutional neural networks (CNNs) and transformer networks, in image recognition, vision-based drone perception systems have rapidly developed. By integrating high-definition visible-light cameras, infrared sensors, and other imaging devices, drones can detect, classify, and identify various objects (such as people, vehicles, infrastructure, and disaster traces) in ground scenes during flight missions. Existing technologies include recognition models pre-trained on large-scale datasets (such as ImageNet, COCO, and DOTA), object detection methods (such as YOLO and Faster R-CNN), and multimodal fusion methods (such as visible-light and infrared fusion recognition), which have been applied in various scenarios such as disaster relief, urban inspections, and agricultural monitoring. In particular, existing methods for image feature extraction, feature fusion, and discriminant classification achieve high recognition accuracy under standard environmental conditions, providing important support for the intelligent execution of drone missions.

[0003] However, existing image recognition methods still face significant challenges in practical drone applications. First, most recognition models rely on training samples from specific source environments (e.g., urban roads and regular buildings) and lack the ability to transfer across diverse flight scenarios (e.g., mountainous areas, farmland, and post-disaster ruins). This results in a significant decrease in target detection and recognition accuracy in these new environments. Second, due to the variability of flight environments (e.g., strong direct sunlight, haze obstruction, low illumination at night, and wind and sand interference), imaging data from different modalities exhibit significant differences in quality and effective information content. Existing multimodal image fusion methods typically employ fixed weighting strategies, making it difficult to dynamically adjust the modal contribution ratio based on environmental conditions, resulting in insufficient robustness in extreme environments. Furthermore, traditional transfer learning methods often rely on supervised learning and require a large number of labeled samples in new scenes, which hinders the rapid deployment and adaptive optimization of drone visual recognition systems in practical applications. Therefore, a multimodal visual recognition method for drones is urgently needed that can dynamically adjust feature fusion strategies and achieve efficient cross-scenario transfer in the absence of labeled data, combined with environmental perception, in the presence of dynamic feature fusion strategies.

[0004] To this end, a multimodal visual recognition method for drones based on cross-scene transfer learning is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal visual recognition method for drones with cross-scene transfer learning, which can dynamically adjust the feature fusion strategy in combination with environmental perception in the absence of labeled data and achieve efficient cross-scene transfer.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multimodal visual recognition method for drones using cross-scene transfer learning, including:

[0008] The sensors onboard the drone collect environmental parameter information and multimodal image data in real time and synchronously, and use the scene prediction model to predict the scene category of the current flight area;

[0009] By calculating the matching degree between the scene category and the source domain subset in the preset source domain data set, the source domain subset with the highest matching degree is selected; based on the environmental feature information of the target domain sample, the target domain sample is divided into multiple subdomains to obtain target subdomain data;

[0010] Extracting unimodal features of each modality image in the multimodal image data respectively, dynamically switching a fusion strategy of the unimodal features based on a quality assessment result of each modality image and a scene category, fusing the unimodal features, and generating a first fused feature;

[0011] For the first fusion features of multiple frames continuously collected during flight, a sliding window mechanism is used to accumulate time series features and correct them according to the historical feature state at the previous moment to obtain the second fusion feature;

[0012] Based on the source domain subset and the target subdomain data, staged transfer learning is performed to train a visual recognition model and the trained visual recognition model is used to perform visual recognition on the second fused features of the current target domain.

[0013] Preferably, the environmental parameter information includes: at least one of light intensity, visibility, temperature, humidity and wind speed; the multimodal image data includes: visible light image, infrared image and lidar point cloud;

[0014] The scene prediction model includes: an environmental parameter processing unit, a preliminary image feature extraction unit, a multi-source information fusion unit and a scene classification prediction unit;

[0015] The environmental parameter processing unit normalizes and / or standardizes the environmental parameter information, encodes the category parameters, and outputs an environmental feature vector;

[0016] The preliminary image feature extraction unit extracts a visual feature vector based on the multimodal image data using a lightweight feature extraction network; the visual feature vector includes scene content information;

[0017] The multi-source information fusion unit fuses the environmental feature vector and the visual feature vector to obtain a scene representation feature;

[0018] The scene classification prediction unit calculates the probability that the scene representation feature corresponds to each predefined scene category through M fully connected layers and a Softmax activation function, and outputs the scene category with the highest probability as a prediction result.

[0019] Preferably, the process of obtaining the source domain subset includes:

[0020] Pre-label the scene category to which each source domain sample belongs in the source domain dataset;

[0021] According to the current scene category predicted by the scene prediction model, the Mahalanobis distance is used to calculate the matching degree between the current scene category and each scene category in the source domain dataset;

[0022] The matching degree is sorted from high to low, and a set of source domain samples whose matching degree exceeds a set threshold is selected as the source domain subset, while source domain samples whose matching degree is lower than the set threshold are eliminated.

[0023] Preferably, the process of acquiring the target subdomain data includes:

[0024] Collect unlabeled sample data of the current target domain; perform cluster analysis on the target domain samples using a clustering algorithm based on the environmental feature vectors obtained synchronously with the unlabeled sample data to obtain N clusters; the number of clusters is adaptively adjusted according to the scene category;

[0025] The target domain samples belonging to the same cluster are divided into a subdomain to obtain N target subdomain data.

[0026] Preferably, the process of generating the first fusion feature includes:

[0027] A feature extraction network is used to extract the depth features of visible light images, infrared images and lidar point clouds in multimodal image data as single-modal features;

[0028] By calculating the clarity score, signal-to-noise ratio, contrast, and point cloud density indicators of visible light images, infrared images, and lidar point clouds in multimodal image data, the quality assessment results of each modality image are obtained;

[0029] Based on real-time environmental parameters, predicted scene categories, and modal image quality assessment results, a fusion strategy switching rule table is set, and one and / or multiple feature fusion strategies are dynamically selected and executed to fuse the single modal features to generate the first fusion feature.

[0030] Preferably, the process of acquiring the second fusion feature includes:

[0031] Setting a time sliding window to accumulate the first fusion features of K frames collected continuously;

[0032] The historical feature state is introduced to correct the first fusion feature in the current time window to obtain the second fusion feature; the correction is based on an error compensation mechanism between the first fusion feature at the previous moment and the first fusion feature at the current moment.

[0033] Preferably, the visual recognition model is a model formed during the staged transfer learning training process and ultimately used to perform a specific recognition task on the second fused feature, specifically comprising: a feature processing and local adaptation unit, a task recognition head unit, and a visual recognition output unit;

[0034] The feature processing and local adaptation unit includes a feature transformation layer and a local adaptation layer; the feature transformation layer includes a series of neural network layers for performing nonlinear processing on the input second fusion features; the local adaptation layer aligns the source domain subset with each target subdomain based on the minimum mean square error loss;

[0035] The task recognition head unit is connected after the feature processing and local adaptation unit, and its specific structure is determined according to the final visual recognition task; the visual recognition task includes image classification task, target detection task and other recognition tasks;

[0036] The visual recognition output unit is connected after the task recognition head unit and is used to output visual recognition results.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention introduces environmental perception and scene prediction mechanisms, enabling drones to understand their specific environment and macro-scene types in real time. This information is crucially applied to multiple subsequent steps: First, when selecting the source domain for transfer learning, the entire source dataset is no longer blindly used. Instead, the most matching source domain subset is prioritized based on the predicted scene category, ensuring the relevance of knowledge transfer; second, when processing target domain data, the target domain is subdivided based on environmental characteristics, achieving more refined local environmental adaptation; finally, when fusing multimodal features, a fixed fusion strategy is no longer adopted. Instead, the fusion method is dynamically switched or adjusted based on real-time environmental parameters, scene categories, and the quality assessment results of each modal data itself. This full-process environmental and scene perception-driven adaptive adjustment enables the visual recognition model to better adapt to complex conditions such as sudden changes in lighting, severe weather, and different terrains. Compared with traditional methods that use fixed models and fixed fusion strategies, the method of the present invention demonstrates higher recognition accuracy and greater stability in various non-ideal or dynamically changing actual flight environments.

[0039] 2. On the one hand, the present invention uses scene prediction and matching calculation to select only the source domain subset most relevant to the current target scene for migration, avoiding interference from irrelevant source domain data; on the other hand, the present invention recognizes that there may be significant environmental differences within the target domain, and innovatively divides the target domain into multiple more homogeneous subdomains based on environmental characteristics, and performs phased and localized adaptation training. This strategy of "precise source selection" and "fine target division + local adaptation" significantly reduces the distribution difference between the source domain and the target subdomain, making transfer learning more focused and effective, maximizing the use of effective knowledge in the source domain, while avoiding the negative effects that may be caused by overall adaptation, thereby improving the recognition accuracy and overall performance of subsequent visual recognition models in each subdomain of the target domain.

[0040] 3. The present invention dynamically adjusts the fusion strategy based on the real-time quality assessment of each modality and the scene context, which ensures that the information of high-quality modalities is fully utilized and the influence of low-quality or unreliable modalities is suppressed, thereby generating a more robust "first fusion feature". The present invention also introduces a time series feature accumulation and optimization mechanism based on the continuity characteristics of UAV flight. By accumulating the first fusion features of multiple frames through a sliding window and correcting them using historical feature states, the mutations and noise of single-frame features are smoothed, effectively reducing the impact of instantaneous interference such as motion blur and short occlusion on the recognition results, and generating a "second fusion feature" that is more stable in time and better reflects the dynamics of the scene. This feature representation method that combines modal quality perception fusion with timing optimization significantly improves the accuracy and reliability of the final visual recognition model under continuous flight and interference conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a flow chart of a multimodal visual recognition method for drones using cross-scenario transfer learning provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a process for generating a second fusion feature according to an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the structure of a visual recognition model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] With the widespread application of drones in security inspections, disaster monitoring, environmental perception and other fields, complex and changing environments have placed higher demands on their visual recognition capabilities. Traditional visual recognition methods are prone to reduced recognition accuracy when faced with scene changes such as different lighting, climate or terrain, especially when the model generalization ability is limited when the data distribution shifts. The drone multimodal visual recognition method based on cross-scene transfer learning can break through the bottleneck of the traditional model's sudden drop in recognition accuracy in new environments, fully utilize the complementary advantages of multimodal data such as visible light, infrared, and laser point clouds, and dynamically adapt the source domain knowledge and target domain features to achieve robust recognition of targets in different geographical regions, weather conditions and lighting environments. This is of great significance for improving the intelligent perception capabilities of drones in complex real-world scenarios such as disaster monitoring, security inspections, and traffic supervision.

[0046] This paper proposes a multimodal visual recognition method for drones with cross-scene transfer learning. This method dynamically adjusts feature fusion strategies by combining environmental perception and achieves efficient cross-scene transfer, even in the absence of labeled data. To illustrate the effectiveness of this method in achieving efficient cross-scene transfer, the following two examples will illustrate its effectiveness.

[0047] Example 1

[0048] In the embodiment of the present application, the method proposed by the present invention is used to describe in detail the process of dynamically adjusting the feature fusion strategy in combination with environmental perception and achieving efficient cross-scene migration in the absence of labeled data. The embodiment of the present application is suitable for multimodal visual recognition of drones in continuous flight environments with dynamic scene changes or interference. Figure 1The content describes the multimodal visual recognition process of drones in detail; among them, Figure 1 The specific flow chart of the method proposed in the present invention includes: real-time synchronous collection of environmental parameter information and multimodal image data by sensors carried by drones, and prediction of the scene category of the current flight area by using a scene prediction model; selecting the source domain subset with the highest matching degree by calculating the matching degree between the scene category and the source domain subset in the preset source domain data set; dividing the target domain samples into multiple subdomains based on the environmental feature information of the target domain samples to obtain target subdomain data; extracting the single modal features of each modality image in the multimodal image data respectively, and dynamically switching the fusion strategy of the single modality features based on the quality assessment results of each modality image and the scene category, fusing the single modality features to generate a first fusion feature; for the first fusion feature of multiple frames continuously collected during the flight, a sliding window mechanism is used to accumulate time series features, and the feature is corrected according to the historical feature state of the previous moment to obtain a second fusion feature; based on the source domain subset and the target subdomain data, performing phased transfer learning to train the visual recognition model and using the trained visual recognition model to perform visual recognition on the second fusion feature of the current target domain. Combined with Figure 1 The following describes the contents:

[0049] The sensors carried by the drone can collect environmental parameter information and multimodal image data in real time and synchronously;

[0050] The environmental parameter information includes: at least one of light intensity, visibility, temperature, humidity and wind speed;

[0051] The multimodal image data includes: visible light image, infrared image and lidar point cloud;

[0052] Specifically, the system is equipped with a visible light camera (1920×1080 resolution, 30 fps), a thermal infrared camera (640×512 resolution, 30 fps), a lidar (16 lines, 10 Hz scanning frequency), and an environmental sensor suite consisting of a digital illuminance meter (0-200 kLux), a visibility sensor, an integrated temperature and humidity sensor, and a 3D wind speed and direction meter. A controller synchronously collects data streams from all sensors in real time. The collected environmental parameter information and preliminary multimodal image data are fed into a pre-trained lightweight scene prediction model.

[0053] Preferably, a scene prediction model is used to predict the scene category of the current flight area;

[0054] The scene prediction model includes: an environmental parameter processing unit, a preliminary image feature extraction unit, a multi-source information fusion unit and a scene classification prediction unit; the environmental parameter processing unit normalizes and / or standardizes the environmental parameter information, encodes the category parameters, and outputs an environmental feature vector;

[0055] The preliminary image feature extraction unit extracts a visual feature vector based on the multimodal image data using a lightweight feature extraction network; the visual feature vector includes scene content information;

[0056] The multi-source information fusion unit fuses the environmental feature vector and the visual feature vector to obtain a scene representation feature;

[0057] The scene classification prediction unit calculates the probability that the scene representation feature corresponds to each predefined scene category through M fully connected layers and a Softmax activation function, and outputs the scene category with the highest probability as a prediction result.

[0058] Specifically, the preliminary image feature extraction unit uses MobileNetV3-Small as the backbone network to process the input multimodal image data separately and extract visual feature vectors containing scene information;

[0059] The multi-source information fusion unit concatenates the environmental feature vector and the visual feature vector, and performs fusion and dimensionality reduction through one or two fully connected layers to obtain scene representation features;

[0060] The scene classification prediction unit consists of three fully connected layers with 128, 64, and 9 neurons respectively (corresponding to 9 predefined scene categories). The last layer uses a Softmax activation function to calculate the probability that the current scene belongs to each predefined scene category and outputs the scene category with the highest probability as the prediction result. The predefined categories include but are not limited to: city-clear, city-rainy, city-nighttime, suburban-clear, suburban-rainy, suburban-nighttime, mountain-fog, mountain-rainy, and post-disaster-smoke and dust.

[0061] By integrating multiple environmental sensors and multimodal imaging devices, the system comprehensively perceives the drone's physical environment and visual scene. A lightweight scene prediction model, incorporating specialized processing units, efficiently integrates environmental parameters and preliminary visual information in real time, accurately and rapidly predicting the current complex flight scenario category. This provides a key basis for subsequent adaptive source domain selection, target domain partitioning, and feature fusion, improving environmental adaptability and responsiveness in rapidly changing scenarios while ensuring the feasibility of onboard computing.

[0062] Preferably, by calculating the matching degree between the scene category and the source domain subset in the preset source domain data set, the source domain subset with the highest matching degree is selected; the process of obtaining the source domain subset includes:

[0063] Pre-label the scene category to which each source domain sample belongs in the source domain dataset;

[0064] According to the current scene category predicted by the scene prediction model, the Mahalanobis distance is used to calculate the matching degree between the current scene category and each scene category in the source domain dataset;

[0065] The matching degree is sorted from high to low, and a set of source domain samples whose matching degree exceeds a set threshold is selected as the source domain subset, while source domain samples whose matching degree is lower than the set threshold are eliminated.

[0066] Specifically, a source domain dataset containing multiple scene categories is pre-built. In this embodiment, the source domain dataset contains 100,000 multimodal images covering 9 scene categories. Each source domain sample is annotated with the scene category and recognition task label; scene category annotation is completed by manual annotation combined with automatic matching to ensure annotation accuracy;

[0067] When the drone enters a new environment, the scene prediction model outputs the current scene category. Assuming the current predicted scene category is "city-rainy day", it is necessary to select a subset of samples with a high degree of match for the "city-rainy day" scene from the source domain dataset as the knowledge source for transfer learning;

[0068] The Mahalanobis distance is used to calculate the matching degree between the current scene category and each scene category in the source domain dataset;

[0069] The calculated matching degrees are sorted from high to low, and the matching degree threshold is set to 0.7; the source domain sample set whose matching degree exceeds the matching degree threshold is selected as the source domain subset; for example, for the "city-rainy day" scene, the matching degrees are from high to low: city-rainy day (0.92), suburb-rainy day (0.78), city-clear (0.71), suburb-night (0.65), etc.; samples of the first three scene categories with matching degrees ≥ 0.7 are selected to form the source domain subset; at the same time, source domain samples with matching degrees below the threshold are eliminated.

[0070] By pre-labeling source domain scene categories and using the Mahalanobis distance to predict the matching degree between the target and source domain scenes, the matching degree between scenes can be more accurately measured. Setting a threshold further ensures that only highly relevant source domain samples are selected. This source domain subset selection strategy based on scene matching effectively avoids the introduction of irrelevant or conflicting source domain knowledge, significantly reduces the risk of negative transfer, and improves the targetedness and efficiency of transfer learning. In particular, recognition accuracy is significantly improved when the scenes vary significantly. Table 1 compares the recognition accuracy of different methods in different scenes.

[0071] Table 1 Comparison of recognition accuracy in different scenarios

[0072] Test scenario Comparison method accuracy (%) (fixed model / fusion) Accuracy of the method of the present invention (%) Accuracy improvement (%) City-Sunny 92.5 94.8 2.3 City at night 75.3 91.2 15.9 Suburban area - rainy day 78.1 89.5 11.4 Mountain-foggy 65.7 85.1 19.4 Post-disaster smoke and dust 70.1 89.6 19.5

[0073] Preferably, the target domain samples are divided into multiple subdomains based on the environmental feature information of the target domain samples to obtain target subdomain data; the process of obtaining the target subdomain data includes: collecting unlabeled sample data of the current target domain; performing cluster analysis on the target domain samples using a clustering algorithm based on the environmental feature vectors obtained synchronously with the unlabeled sample data to obtain N clusters; the number of the clusters is adaptively adjusted according to the scene category;

[0074] The target domain samples belonging to the same cluster are divided into a subdomain to obtain N target subdomain data.

[0075] Specifically, after the UAV enters the target domain environment, it continuously collects unlabeled sample data of the current target domain;

[0076] Based on the flight speed and coverage of the drone, 5,000-10,000 multimodal images are collected in the target domain environment as unlabeled samples. At the same time, the environmental feature vectors obtained synchronously with each image are recorded;

[0077] The K-Means++ clustering algorithm is used to cluster target domain samples. First, a similarity matrix is ​​calculated between samples based on the environmental feature vectors. Then, the density peak points are identified as cluster centers. Finally, the remaining points are assigned to the nearest cluster center to form clusters. The number of clusters, N, is adaptively adjusted based on the scene type. For example, for a "city-sunny" scene, which may have multiple lighting variations, N is set to 5. For a "suburban-nighttime" scene, the environment is relatively simple, so N is set to 2. The specific adjustment rules are as follows:

[0078] Urban scene: N=5+light variation / 20;

[0079] Suburban scene: N=2+light variation / 30;

[0080] Mountainous scene: N=2+visibility change / 5;

[0081] Rainy day scenario: N=2+rainfall level;

[0082] The unit of illumination change is klux, the unit of visibility change is km, and the rainfall level is an integer from 0 to 3.

[0083] The target domain samples belonging to the same cluster are divided into a subdomain, and finally N target subdomain data are obtained.

[0084] Table 2 is a comparison table of the target domain recognition accuracy of transfer learning of different methods.

[0085] Table 2 Comparison of target domain recognition accuracy of transfer learning

[0086] Target domain scenario / subdomain Comparison method accuracy (%) (global migration) Accuracy of the method of the present invention (%) Target domain: urban mixed lighting 80.5 - Subfield 1: Bright Light - 91.3 Subfield 2: Low-light shadows - 86.8 Target domain: Foggy rural areas 72.3 - Subdomain 1: Mist - 89.5 Subdomain 2: Thick Fog - 92.3

[0087] This embodiment achieves fine-grained segmentation of target domain samples through an adaptive clustering method driven by environmental features, capable of capturing differences in data distribution under different environmental conditions within the same scene. At the same time, the number of clusters N is adaptively adjusted according to the scene category, so that the segmentation granularity can match the complexity of the scene. Compared with the traditional method that treats the target domain as a whole, this method fully considers the impact of environmental factors on visual features, and finely depicts the environmental diversity within the target domain, making the subsequent localized domain adaptation process more accurate and helping to reduce the interference caused by intra-domain differences.

[0088] Preferably, the single modal features of each modality image in the multimodal image data are extracted respectively, and the fusion strategy of the single modal features is dynamically switched based on the quality assessment results of each modality image and the scene category, and the single modal features are fused to generate a first fused feature;

[0089] The process of generating the first fusion feature includes:

[0090] A feature extraction network is used to extract the depth features of visible light images, infrared images and lidar point clouds in multimodal image data as single-modal features;

[0091] By calculating the clarity score, signal-to-noise ratio, contrast, and point cloud density indicators of visible light images, infrared images, and lidar point clouds in multimodal image data, the quality assessment results of each modality image are obtained;

[0092] Based on real-time environmental parameters, predicted scene categories, and modal image quality assessment results, a fusion strategy switching rule table is set, and one and / or multiple feature fusion strategies are dynamically selected and executed to fuse the single modal features to generate the first fusion feature.

[0093] Specifically, we used the pre-trained EfficientNet-B3 to extract features from visible light images (output dimension: 1536), the improved ResNet-34 to extract features from infrared images (output dimension: 512), and PointNet++ to extract features from lidar point clouds (output dimension: 1024). Each modality feature was then adaptively pooled to a uniform 512-dimensional feature vector.

[0094] Calculate the quality assessment indicators for each modality image: For visible light images, calculate the clarity score of the Laplacian gradient method, the signal-to-noise ratio based on wavelet transform, and the RMS contrast to obtain the quality assessment results of the visible light image; for infrared images, calculate the clarity index and contrast index based on thermal gradient to obtain the quality assessment results of the infrared image; for LiDAR point clouds, calculate the point cloud density and point cloud completeness index to obtain the quality assessment results of the LiDAR point cloud. Normalize the quality assessment results of each modality to form a modality quality score vector;

[0095] Based on real-time environmental parameters, predicted scene categories and modal image quality assessment results, a fusion strategy switching rule table is set to dynamically select feature fusion strategies.

[0096] This embodiment implements a dynamic feature fusion strategy switching mechanism based on environmental parameters, scene type, and modality quality, resolving the difficulty of traditional fixed fusion strategies in adapting to changing environments. The system adaptively selects the optimal fusion strategy based on current environmental conditions and the quality of each modality's data, leveraging the strengths of each modality, maximizing effective information, and suppressing noise interference, resulting in higher-quality, more environmentally adaptable first-tier fusion features.

[0097] Preferably, for the first fusion features of multiple frames continuously collected during the flight, a sliding window mechanism is used to accumulate time series features, and correction is performed according to the historical feature state at the previous moment to obtain the second fusion feature;

[0098] The process of obtaining the second fusion feature includes:

[0099] Setting a time sliding window to accumulate the first fusion features of K frames collected continuously;

[0100] The historical feature state is introduced to correct the first fusion feature in the current time window to obtain the second fusion feature; the correction is based on an error compensation mechanism between the first fusion feature at the previous moment and the first fusion feature at the current moment.

[0101] Specifically, we set the sliding window size K to 5, which corresponds to approximately 0.25 seconds of data continuously collected by the drone at a frequency of 20 Hz. The sliding window moves forward by one frame each time, keeping the window size constant.

[0102] Store the first fusion feature sequence of the most recent K frames, accumulate the features within the window, and calculate the weighted average, with the weight decaying exponentially over time;

[0103] The difference between the first fusion features of the current frame and the previous frame is calculated, and the first fusion features in the current time window are corrected by a correction mechanism based on error compensation to obtain the second fusion features.

[0104] Table 3 shows the comparison of recognition accuracy of different methods under different interferences.

[0105] Table 3 Comparison of recognition accuracy under different interferences

[0106] Test conditions (continuous flight) Interference Type Comparison method accuracy (%) (simple fusion + single frame recognition) Accuracy of the method of the present invention (%) Normal flight No obvious interference 93.1 97.5 Quick Turn Motion blur (partial frame) 76.5 91.3 Flying low over tree branches Temporary occlusion (partial frame) 72.8 92.7 Infrared signal is weak at night Unimodal quality degradation 79.4 89.1 LiDAR point clouds are sparse in rainy days Unimodal quality degradation 75.6 88.2

[0107] This embodiment accumulates multi-frame features through a temporal sliding window, integrating short-term temporal information and enhancing feature stability. A correction mechanism based on error compensation between previous and next frame features effectively identifies and suppresses dramatic feature changes caused by single-frame anomalies (such as blur and momentary occlusions), while preserving the scene's true dynamics. This results in a smoother temporal representation of the generated second fused features, significantly improving the reliability and accuracy of visual recognition under continuous flight conditions and in the presence of transient interference.

[0108] Preferably, based on the source domain subset and the target subdomain data, performing phased transfer learning to train a visual recognition model and using the trained visual recognition model to perform visual recognition on the second fused features of the current target domain; referring to Figure 3 ;

[0109] The visual recognition model is a model formed during the staged transfer learning training process and ultimately used to perform a specific recognition task on the second fused feature, specifically comprising: a feature processing and local adaptation unit, a task recognition head unit, and a visual recognition output unit;

[0110] The feature processing and local adaptation unit includes a feature transformation layer and a local adaptation layer;

[0111] The feature transformation layer includes a series of neural network layers for performing nonlinear processing on the input second fusion features;

[0112] The local adaptation layer aligns the source domain subset with each target subdomain based on the minimum mean square error loss;

[0113] The task recognition head unit is connected after the feature processing and local adaptation unit, and its specific structure is determined according to the final visual recognition task; the visual recognition task includes image classification task, target detection task and other recognition tasks;

[0114] The visual recognition output unit is connected after the task recognition head unit and is used to output visual recognition results.

[0115] Specifically, the specific structure of the feature processing and local adaptation unit includes:

[0116] Feature transformation layer: This layer contains three residual blocks, each consisting of two convolutional layers and a short-circuit connection. The convolution kernel size is 3×3, and the number of channels is 512, 512, and 512, respectively. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function to perform nonlinear transformations on the input second fused features, enhancing feature expression.

[0117] Local Adaptation Layer: This layer uses a multi-core maximum average pooling structure, extracting multi-scale features using kernels of different sizes (1×1, 3×3, and 5×5). It then calculates the weighted sum of the maximum and average pooling results. This layer is followed by a fully connected layer with an output dimension of 256, which is used to align features between the source domain subset and each target subdomain.

[0118] The structure of the task recognition head unit is determined according to the specific visual recognition task:

[0119] Image classification task: It contains two fully connected layers with 128 neurons and C categories (C = 9). The last layer uses the Softmax activation function to output the category probability distribution.

[0120] Object detection task: uses an improved FPN structure, including a feature pyramid network and two parallel branches, including a classification branch and a bounding box regression branch;

[0121] Other recognition tasks (such as scene segmentation, anomaly detection, etc.): Customize the corresponding network structure according to specific task requirements.

[0122] Visual Recognition Output Unit: Connected after the task recognition head unit, it outputs the corresponding recognition results based on the task type. For classification tasks, it outputs the category label and confidence level; for detection tasks, it outputs the object location, category, and confidence level.

[0123] The staged transfer learning training process consists of three stages: source domain pre-training, local domain adaptation, and target domain fine-tuning.

[0124] The visual recognition model architecture designed in this embodiment fully considers the needs of feature processing, domain adaptation, and task identification, and realizes the knowledge transfer from the source domain to the target domain through a phased transfer learning strategy. The feature processing and local adaptation units enhance the feature expression ability and realize feature alignment between different subdomains; the task recognition head unit is customized according to the specific task, which improves the task specificity of the model; the phased training and local adaptation loss for each target subdomain ensure that the model can accurately adapt to the unique data distribution of each subdomain caused by environmental changes within the target domain. This overcomes the limitations of traditional global domain adaptation methods, significantly improves the comprehensive recognition performance of the model on the entire heterogeneous target domain, and makes the final visual recognition results more accurate and reliable.

[0125] This invention provides a multimodal visual recognition method for drones using cross-scene transfer learning. By integrating environmental perception, scene prediction, modal dynamic trade-offs, and time series modeling, it significantly improves the robustness and generalization of drone visual recognition in complex and changing environments. The method first utilizes drone sensors to synchronously collect environmental parameters and multimodal image data in real time. It then combines these parameters with preliminary image features for scene prediction, dynamically selecting the source domain subset that best matches the current scene, improving the relevance and effectiveness of the initial transfer starting point. Furthermore, a target domain subdomain partitioning mechanism based on environmental features is employed to construct a more fine-grained subdomain structure. This is combined with local adaptation to optimize the transfer process, effectively mitigating the risk of negative transfer. Regarding feature processing, a dynamic switching mechanism for fusion quality assessment and scene classification fusion strategies is proposed to address the differences and environmental adaptability of multimodal images, achieving optimal multimodal feature combinations in different scenarios. Furthermore, a sliding window accumulation and history state correction mechanism is designed to address temporal disturbances such as image blur and occlusion during drone flight, significantly improving the consistency of feature representation across consecutive frames and recognition stability. Finally, a staged transfer learning approach was implemented to train the visual recognition model using data from both the source and target subdomains, effectively improving the recognition accuracy and generalization capabilities of the model in new scenarios. The overall solution boasts a rational structure, strong adaptability, and high robustness, making it suitable for intelligent UAV visual perception systems in dynamic mission scenarios.

[0126] Example 2

[0127] In Example 1, the method proposed in this invention successfully implemented dynamic feature fusion strategies in combination with environmental perception, even in the absence of labeled data, and achieved efficient cross-scene migration. To further verify the effectiveness of this invention, this example also performed multimodal visual recognition on another drone in a continuous flight environment with dynamic scene changes or interference.

[0128] A multimodal visual recognition method for drones using cross-scene transfer learning, including:

[0129] The sensors carried by the drone can collect environmental parameter information and multimodal image data in real time and synchronously;

[0130] The environmental parameter information includes: at least one of light intensity, visibility, temperature, humidity and wind speed; the multimodal image data includes: visible light image, infrared image and lidar point cloud;

[0131] Preferably, a scene prediction model is used to predict the scene category of the current flight area;

[0132] The scene prediction model includes: an environmental parameter processing unit, a preliminary image feature extraction unit, a multi-source information fusion unit and a scene classification prediction unit;

[0133] The environmental parameter processing unit normalizes and / or standardizes the environmental parameter information, encodes the category parameters, and outputs an environmental feature vector; the preliminary image feature extraction unit extracts a visual feature vector based on the multimodal image data using a lightweight feature extraction network; the visual feature vector contains scene content information; the multi-source information fusion unit fuses the environmental feature vector and the visual feature vector to obtain a scene representation feature; the scene classification prediction unit calculates the probability that the scene representation feature corresponds to each predefined scene category through M fully connected layers and a Softmax activation function, and outputs the scene category with the highest probability as a prediction result.

[0134] Preferably, by calculating the matching degree between the scene category and the source domain subset in the preset source domain data set, the source domain subset with the highest matching degree is selected;

[0135] The process of obtaining the source domain subset includes:

[0136] Pre-label the scene category to which each source domain sample belongs in the source domain dataset;

[0137] According to the current scene category predicted by the scene prediction model, the Mahalanobis distance is used to calculate the matching degree between the current scene category and each scene category in the source domain dataset;

[0138] The matching degree is sorted from high to low, and a set of source domain samples whose matching degree exceeds a set threshold is selected as the source domain subset, while source domain samples whose matching degree is lower than the set threshold are eliminated.

[0139] Preferably, the target domain sample is divided into multiple subdomains based on the environmental feature information of the target domain sample to obtain target subdomain data; the target subdomain data acquisition process includes:

[0140] Collect unlabeled sample data of the current target domain; perform cluster analysis on the target domain samples using a clustering algorithm based on the environmental feature vectors obtained synchronously with the unlabeled sample data to obtain N clusters; the number of clusters is adaptively adjusted according to the scene category;

[0141] The target domain samples belonging to the same cluster are divided into a subdomain to obtain N target subdomain data.

[0142] Preferably, the single modal features of each modality image in the multimodal image data are extracted respectively, and the fusion strategy of the single modal features is dynamically switched based on the quality assessment results of each modality image and the scene category, and the single modal features are fused to generate a first fused feature;

[0143] The process of generating the first fusion feature includes:

[0144] A feature extraction network is used to extract the depth features of visible light images, infrared images and lidar point clouds in multimodal image data as single-modal features;

[0145] By calculating the clarity score, signal-to-noise ratio, contrast, and point cloud density indicators of visible light images, infrared images, and lidar point clouds in multimodal image data, the quality assessment results of each modality image are obtained;

[0146] Based on real-time environmental parameters, predicted scene categories, and modal image quality assessment results, a fusion strategy switching rule table is set, and one and / or multiple feature fusion strategies are dynamically selected and executed to fuse the single modal features to generate the first fusion feature.

[0147] Table 4 shows a partial example of the fusion strategy switching rule table.

[0148] Table 4: Example of a partial table of fusion policy switching rules

[0149] Scene Category Environmental conditions Modal mass condition Fusion Strategy City-Sunny Light>80klux Visible light clarity>0.8 Weighted fusion (visible light 0.7, infrared 0.1, point cloud 0.2) City-Sunny Light 20-80klux Visible light clarity>0.6 Attention Mechanism Suburbs - Night Light <0.1klux Infrared clarity>0.7 Weighted Fusion Mountainous area - foggy day Visibility <0.1km <![CDATA[Point cloud density > 10pts / m 2 > Feature complementary fusion Any scene Any condition All modal masses are < 0.4 Decision-level fusion

[0150] For the first fusion features of multiple frames continuously collected during flight, a sliding window mechanism is used to accumulate time series features and correct them according to the historical feature state at the previous moment to obtain the second fusion feature;

[0151] The process of obtaining the second fusion feature includes:

[0152] Setting a time sliding window to accumulate the first fusion features of K frames collected continuously;

[0153] The historical feature state is introduced to correct the first fusion feature in the current time window to obtain the second fusion feature; the correction is based on an error compensation mechanism between the first fusion feature at the previous moment and the first fusion feature at the current moment.

[0154] Based on the source domain subset and the target subdomain data, staged transfer learning is performed to train a visual recognition model and the trained visual recognition model is used to perform visual recognition on the second fused features of the current target domain.

[0155] The visual recognition model is a model formed during the staged transfer learning training process and ultimately used to perform a specific recognition task on the second fusion feature. It specifically includes: a feature processing and local adaptation unit, a task recognition head unit, and a visual recognition output unit; the feature processing and local adaptation unit includes a feature transformation layer and a local adaptation layer; the feature transformation layer contains a series of neural network layers for performing nonlinear processing on the input second fusion feature; the local adaptation layer aligns the source domain subset with each target subdomain based on the minimum mean square error loss; the task recognition head unit is connected after the feature processing and local adaptation unit, and its specific structure is determined according to the final visual recognition task; the visual recognition task includes image classification tasks, target detection tasks and other recognition tasks; the visual recognition output unit is connected after the task recognition head unit for outputting visual recognition results.

[0156] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal visual recognition method for drones based on cross-scene transfer learning, characterized by: include: The sensors onboard the drone collect environmental parameter information and multimodal image data in real time and synchronously, and use the scene prediction model to predict the scene category of the current flight area; By calculating the matching degree between the scene category and the source domain subset in the preset source domain dataset, selecting the source domain subset with the highest matching degree; Divide the target domain sample into multiple subdomains based on the environmental feature information of the target domain sample to obtain target subdomain data; Extracting unimodal features of each modality image in the multimodal image data respectively, dynamically switching a fusion strategy of the unimodal features based on a quality assessment result of each modality image and a scene category, fusing the unimodal features, and generating a first fused feature; For the first fusion features of multiple frames continuously collected during flight, a sliding window mechanism is used to accumulate time series features and correct them according to the historical feature state at the previous moment to obtain the second fusion feature; Based on the source domain subset and the target subdomain data with the highest matching degree, staged transfer learning is performed to train a visual recognition model and the trained visual recognition model is used to perform visual recognition on the second fused features of the current target domain.

2. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The environmental parameter information includes: at least one of light intensity, visibility, temperature, humidity and wind speed; the multimodal image data includes: visible light image, infrared image and lidar point cloud; The scene prediction model includes: an environmental parameter processing unit, a preliminary image feature extraction unit, a multi-source information fusion unit and a scene classification prediction unit; The environmental parameter processing unit normalizes and / or standardizes the environmental parameter information, encodes the category parameters, and outputs an environmental feature vector; the preliminary image feature extraction unit extracts a visual feature vector based on the multimodal image data using a lightweight feature extraction network; the visual feature vector contains scene content information; the multi-source information fusion unit fuses the environmental feature vector and the visual feature vector to obtain a scene representation feature; the scene classification prediction unit calculates the probability that the scene representation feature corresponds to each predefined scene category through M fully connected layers and a Softmax activation function, and outputs the scene category with the highest probability as a prediction result.

3. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The process of obtaining the source domain subset includes: Pre-label the scene category to which each source domain sample belongs in the source domain dataset; According to the current scene category predicted by the scene prediction model, the Mahalanobis distance is used to calculate the matching degree between the current scene category and each scene category in the source domain dataset; The matching degree is sorted from high to low, and a set of source domain samples whose matching degree exceeds a set threshold is selected as the source domain subset, while source domain samples whose matching degree is lower than the set threshold are eliminated.

4. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The process of acquiring the target subdomain data includes: Collect unlabeled sample data of the current target domain; perform cluster analysis on the target domain samples using a clustering algorithm based on the environmental feature vectors obtained synchronously with the unlabeled sample data to obtain N clusters; the number of clusters is adaptively adjusted according to the scene category; The target domain samples belonging to the same cluster are divided into a subdomain to obtain N target subdomain data.

5. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The process of generating the first fusion feature includes: A feature extraction network is used to extract the depth features of visible light images, infrared images and lidar point clouds in multimodal image data as single-modal features; By calculating the clarity score, signal-to-noise ratio, contrast, and point cloud density indicators of visible light images, infrared images, and lidar point clouds in multimodal image data, the quality assessment results of each modality image are obtained; Based on real-time environmental parameters, predicted scene categories, and modal image quality assessment results, a fusion strategy switching rule table is set, and one and / or multiple feature fusion strategies are dynamically selected and executed to fuse the single modal features to generate the first fusion feature.

6. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The process of acquiring the second fusion feature includes: Setting a time sliding window to accumulate the first fusion features of K frames collected continuously; The historical feature state is introduced to correct the first fusion feature in the current time window to obtain the second fusion feature; the correction is based on an error compensation mechanism between the first fusion feature at the previous moment and the first fusion feature at the current moment.

7. The multimodal visual recognition method for drones based on cross-scene transfer learning according to claim 1 is characterized in that: The visual recognition model is a model formed during the staged transfer learning training process and ultimately used to perform a specific recognition task on the second fused feature, specifically comprising: a feature processing and local adaptation unit, a task recognition head unit, and a visual recognition output unit; The feature processing and local adaptation unit includes a feature transformation layer and a local adaptation layer; the feature transformation layer contains a series of neural network layers for performing nonlinear processing on the input second fusion features; the series of neural network layers contains three residual blocks, each residual block consists of two convolutional layers and a short-circuit connection; the local adaptation layer aligns the source domain subset with each target subdomain based on the minimum mean square error loss; the task recognition head unit is connected after the feature processing and local adaptation unit, and its specific structure is determined according to the final visual recognition task; the visual recognition task includes image classification task, target detection task and other recognition tasks; the other recognition tasks include scene segmentation and anomaly detection; the visual recognition output unit is connected after the task recognition head unit for outputting visual recognition results.

Images