Unmanned aerial vehicle inspection method and device based on missing cross-modal data class incremental learning

Abstract

The application discloses a kind of based on missing cross-modal data class incremental learning unmanned aerial vehicle inspection method and device, method includes: to the missing data that unmanned aerial vehicle collects is completed;The data after completion is preprocessed, obtains the characteristics of each mode hidden in data;The feature channel of different mode in input feature map is mixed;Each group is assigned a set of convolution kernel in the input feature map after mixing, after grouping convolution, adaptive average pooling is applied to feature map, and the final output feature map is obtained;The output feature map is flattened, then it is input into fully connected layer, and activation function is applied to introduce nonlinearity, generate final classification result, subsequently apply loss function to calculate loss, and use gradient descent method to update model parameters;After model training is completed, it is deployed to unmanned aerial vehicle, and unmanned aerial vehicle identifies scene and entity.The device includes: processor and memory.The application improves the identification and analysis capability of unmanned aerial vehicle in complex environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection in intelligent unmanned systems, and more particularly to a UAV inspection method and apparatus based on incremental learning of missing cross-modal data classes. Background Technology

[0002] Unmanned aerial vehicle (UAV) inspection systems primarily utilize drones for automated monitoring and inspection of critical infrastructure, such as power lines, oil and gas pipelines, railways, and bridges. These systems are also widely used in agriculture, forestry monitoring, and urban management. Their main advantages include increased operational efficiency, reduced operating costs, and decreased personnel safety risks. When performing inspection tasks, UAVs collect and analyze data to make automated decisions. During the day, UAVs capture clear images using high-definition cameras for analysis; at night, they typically rely on infrared cameras to process images acquired in low-light conditions.

[0003] Current automatic detection models primarily process images captured by high-definition or infrared cameras. These models train deep neural networks based on collected data during the training phase, and then deploy these networks to analyze newly acquired images to assess the current situation. However, due to the significant differences in information and features between images under daytime and nighttime conditions, such a single model often struggles to adapt to both situations simultaneously. Furthermore, in practice, some sensing modalities may be unable to provide data due to technical malfunctions, environmental interference, or other external factors. Even if a modality is available, limitations in sensor coverage or the sensor itself may prevent the capture of the entire scene, resulting in data loss.

[0004] Cross-modal learning focuses on learning and reasoning from the connections between different modalities, enhancing the model's perception and understanding capabilities through the mutual supplementation and information sharing of data from different modalities. In this learning paradigm, the model integrates information from different perceptual modalities to improve its understanding of the environment and decision-making abilities. Incremental learning, on the other hand, allows the model to maintain its memory of existing data while absorbing new data. However, cross-modal learning relies on the complementarity of information between different modalities to improve model performance. When data for a particular modality is missing, the model cannot fully utilize the features of that modality, resulting in incomplete information and reduced overall model performance. Data gaps can also lead to an imbalanced distribution of data across modalities, causing the model's training to be entirely biased towards one modality. In incremental learning, the model needs to maintain its memory of old data while continuously learning new data. Data gaps exacerbate forgetting because missing data may contain key features, making it difficult for the model to retain these features during incremental learning.

[0005] In cross-modal incremental learning, modality forgetting is a significant challenge. During incremental learning, different modalities may have different feature distributions and learning dynamics, causing the model to forget previously learned knowledge about different modalities when learning a new modality. Furthermore, in cross-modal learning, each modality typically provides different types of information. Ideally, the model should extract and utilize information from each modality in a balanced way. However, in practical applications, one modality may gradually become dominant, leading to competitive modality forgetting. This dominance can cause the model to over-rely on information from one modality while ignoring useful information from other modalities, thus affecting the overall learning performance and decision quality.

[0006] To address the issue of missing data, various strategies have been proposed, including imputation, interpolation, and matrix factorization, to fill in missing data and mitigate its impact. While these strategies can solve the problem to some extent, they often rely on basic statistics to impute data, ignoring correlations between data points, or incurring high computational costs and complexity. To address the problem of competitive modality forgetting, existing methods primarily employ modality balancing mechanisms, data augmentation, and resampling. However, these methods also suffer from high resource costs and the subjective nature of setting modality weights. Therefore, effectively addressing the issues of missing data and competitive modality forgetting in cross-modal incremental learning, while simultaneously considering storage and computational efficiency, is crucial for the large-scale deployment and application of UAV inspection models on UAVs. Summary of the Invention

[0007] This invention provides a UAV inspection method and apparatus based on incremental learning of missing cross-modal data. Considering that in real-world applications, the cross-modal data received by UAVs during inspections may be incomplete, leading to the poor performance of traditional incremental learning methods in these scenarios, this invention designs a method based on incremental learning of incomplete cross-modal data. This method integrates three strategies: completion, interactive fusion, and forgetting prevention. It enhances data integrity, maximizes the utilization of information across different modalities, improves the classification accuracy of the UAV inspection model, strengthens the model's ability to handle missing cross-modal data, and effectively reduces the forgetting problem of the UAV inspection model during incremental learning. It can better process and utilize information from different sensors, improving the UAV's recognition and analysis capabilities in complex environments. See the description below for details:

[0008] Firstly, a drone inspection method based on incremental learning of missing cross-modal data classes, the method comprising:

[0009] Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each modality hidden in the data; mix the feature channels of different modalities in the input feature map;

[0010] A set of convolutional kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map.

[0011] The output feature map is flattened and then fed into a fully connected layer. An activation function is applied to introduce non-linearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using gradient descent. After the model is trained, it is deployed on a drone, which then identifies the scene and entities.

[0012] Specifically, the mixing of feature channels of different modalities in the input feature map involves:

[0013] The features from each modality in the input feature map are divided into different channels. The feature channels from each modality are evenly grouped. Channels are selected from each group and mixed with the corresponding group of another modality.

[0014] The loss function consists of two parts: the true classification loss L. cls And new category reserved loss L vr ,

[0015] The true classification loss L cls Used to measure the difference between the true label and the model's predicted output;

[0016] The new type of reserved loss L vr Virtual categories and virtual samples are used to pre-place new, unlearned categories in the embedding space, adjusting the distribution of the embedding space and helping the inspection and recognition model in the UAV maintain its memory of old categories while learning new categories.

[0017] The process of completing the missing data collected by the UAV involves using a masking autoencoder during training to complete the missing data collected by the sensors.

[0018] The channel blending process assigns channels to different predefined groups and recombines feature channels from different modalities. Channel blending is defined as follows:

[0019]

[0020] Where || represents connection, and This represents feature groups extracted from two different modalities, with feature channels evenly distributed into the different groups. t represents the t-th scene, i represents different modalities, N is the number of groups, and C is the number of channels. and This indicates a channel blending operation performed on feature groups extracted from two different modalities. It represents the j-th element of the feature channel in the i-th modality of the t-th scene.

[0021] The virtual sample is:

[0022] Two samples x1 and x2 and their corresponding labels y1 and y2 are randomly selected from the training set. These two samples are then input into the neural network and propagated forward to the selected hidden layer. Let the outputs of the selected hidden layer for x1 and x2 be z1 and z2, respectively. Interpolation is performed on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient. The corresponding labels are then interpolated accordingly: y = λy1 + (1-λ)y2. The new representation z obtained by interpolation is then propagated forward through the remaining part of the neural network to obtain the final output.

[0023] The virtual loss corresponding to the sample is:

[0024]

[0025] in, y' is a virtual category label, y' is a pseudo-label in an existing category, and L img L represents the new class space preserved by the Mask function. mi and L mc To avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed samples, and the Mask function is defined as:

[0026]

[0027] in, Let represent the Hadamard product, used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class; all other class positions are 0, and the current class position is 1. The operation 1-OneHot(y) sets the known class positions to 0 and the unknown class positions to 1. The objective function is ultimately defined as:

[0028] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0029] Among them, L cls This represents the basic loss function calculated using cosine similarity, where λ1 is a balancing factor used to adjust L. vr The relative importance of x and y is given by x, which represents the original input data of the model, and y, which represents the true class label corresponding to the input x.

[0030] A second aspect is a drone inspection device based on incremental learning of missing cross-modal data classes, characterized in that the device includes: a processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to execute the method described in any one of the first aspects.

[0031] Third aspect, a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0032] The beneficial effects of the technical solution provided by this invention are:

[0033] 1. This invention addresses the issue of missing cross-modal data received by UAVs in real-world applications by employing a masking autoencoder to automatically complete the missing data, thereby enhancing data quality, improving data integrity, and providing more stable and reliable input data.

[0034] 2. This invention develops a cross-modal information fusion module for cross-modal data generated in UAV inspection and identification scenarios. This module enhances the diversity of features by establishing connections between different data modalities, which helps to retain the memory of old knowledge and effectively reduces the problem of catastrophic forgetting. It enables the model to better process and utilize information from different sensors, thereby improving the UAV's identification and analysis capabilities in complex environments.

[0035] 3. This invention effectively solves the data sparsity problem caused by the inability to collect data due to the failure of certain sensors; by integrating richer data from other modalities, this invention can effectively make up for the lack of data in certain modalities, thereby improving the efficiency of data use and the overall performance of the model; this cross-modal interactive fusion strategy enhances the model's ability to process uneven data sources, ensuring good performance even when data is scarce.

[0036] 4. This invention can achieve state-of-the-art performance in incremental learning tasks such as UAV recognition, with average accuracy improvements of up to 4.96%, 6.53%, and 3.86% compared to comparative methods. Attached Figure Description

[0037] Figure 1A flowchart of a drone inspection method based on incremental learning of missing cross-modal data classes;

[0038] Figure 2 This is a schematic diagram illustrating the use of a modal completer to complete missing cross-modal data.

[0039] Figure 3 A schematic diagram of a large cross-modal dataset, LasHeR-CIL;

[0040] Figure 4 A schematic diagram of a large cross-modal dataset, VTDV-CIL;

[0041] Figure 5 For comparison purposes;

[0042] The three sub-figures show the results when the missing rate is 0.3, 0.5, and 0.7, respectively. This figure shows the improvement in accuracy of the cross-modal fusion completion strategy compared to other state-of-the-art methods when applied to the VTDV-CIL dataset. It can be seen that the accuracy of the cross-modal fusion completion strategy is significantly improved after multiple stages with different missing rates.

[0043] Figure 6 For comparison purposes;

[0044] The two sub-figures show the results applied to the LasHeR-CIL dataset and the VTDV-CIL dataset, respectively. This figure represents the performance comparison with and without the forgetting prevention strategy when the missing rate is set to 0.3. "Visible" represents the visible light mode, "Thermal" represents the infrared light mode, "ADD" represents the interactive fusion strategy, and "-F" represents the use of the forgetting prevention strategy. It can be seen that the model with the forgetting prevention strategy performs significantly better than the model without the forgetting prevention strategy.

[0045] Figure 7 This is a schematic diagram of parameter analysis;

[0046] The two subplots show the parameter analysis results of the balance factor λ1 and the number of channel mixing groups γ when the missing data rate is set to 0.5. The left plot is ACC, and the right plot is NPD. This figure shows the performance comparison of different parameter settings on the LasHeR-CIL dataset. The value of λ1 is selected from {0.0001, 0.001, 0.01, 0.1, 1}, and γ is selected from {2, 4, 6, 8, 16, 32}. It can be seen that the model performs best when λ1 is set to 0.01 and γ is set to 8.

[0047] Figure 8 This is a schematic diagram of the structure of a drone inspection system based on incremental learning of incomplete cross-modal data.

[0048] Table 1 compares the accuracy, performance degradation rate, and normalization performance degradation rate of different methods on the LasHeR-CIL dataset;

[0049] Table 2 shows the ablation experiments conducted on the LasHeR-CIL dataset, demonstrating the effectiveness of each component of this method. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0051] Example 1

[0052] During inspection missions, UAVs can adjust and optimize their onboard models in real time based on newly collected data to adapt to dynamically changing environments. However, UAVs may encounter situations where some sensors are damaged or obstructed, resulting in incomplete cross-modal data. Furthermore, modal forgetting is common during the learning process. To address these issues, incremental learning of new categories in UAV inspection missions leads to a severe degradation in the recognition ability of older categories. This invention proposes a UAV inspection method based on incremental learning of incomplete cross-modal data, aiming to mitigate the impact of missing data in target detection while incrementally learning new categories and maintaining the ability to recognize older categories. This achieves rapid and efficient model updates and continuous learning. (See [link to related documentation]). Figure 1 The method includes the following steps:

[0053] 101: The drone performs an inspection mission to acquire missing data;

[0054] Drones are deployed according to predetermined routes and areas to perform inspection missions. During the inspection process, the drones collect environmental information and data in different modalities using various onboard sensors (cameras, infrared cameras, lidar, etc.) and transmit the data back to the ground processing center in real time. However, the sensors may be obstructed or damaged during operation, resulting in incomplete data collection.

[0055] 102: Complete the missing data collected;

[0056] In this process, a masking autoencoder is used to automatically complete missing data. In practical applications, the masking autoencoder first transforms the input data containing missing values ​​into a low-dimensional hidden representation using an encoder. This representation captures the core features of the unmasked portions of the data. Next, the decoder uses these core features to attempt to reconstruct the complete form of the original data, including the missing parts. This step significantly enhances data quality, improves data integrity, and provides more stable and reliable input data for subsequent processing.

[0057] 103: Preprocess the completed data to obtain the features of each hidden modality in the data;

[0058] Techniques such as convolution and pooling are used to extract features from the completed cross-modal data. This extracts data containing key features of different modalities from a large dataset, ignoring noise and compressing the data volume to generate feature maps as input for subsequent processing. The number of convolutional kernels in the convolutional layer determines the number of channels in the feature map. Each channel corresponds to the response of the input data to a specific feature.

[0059] 104: Mix feature channels of different modalities in the input feature map;

[0060] In this process, features from each modality in the input feature map are divided into different channels. The feature channels from each modality are evenly grouped. Before the actual convolution operation, a channel is selected from each group and mixed with the corresponding group of another modality. This operation enhances the connection between data from different modalities.

[0061] 105: Assign a set of convolutional kernels to each group in the mixed input feature map and perform group convolution;

[0062] The input feature maps, after channel mixing, are divided into several independent groups, each containing a subset of the feature maps. Similarly, the convolutional kernels are also divided into a corresponding number of groups, with each group processing only the input feature maps of its corresponding group. After grouping, the convolutional kernels within each group independently perform convolution operations with their corresponding feature maps, generating independent output feature maps for each group. Each group's convolutional kernel interacts only with the feature maps of its corresponding group. Each group performs convolution operations independently, generating a subset of its own output feature map. Finally, the output feature map is formed by concatenating the output feature maps of all groups along the depth dimension, ensuring that each channel of the output feature map contains information from a specific group. This operation ensures that the network can handle complex feature extraction tasks while reducing the number of parameters.

[0063] 106: After grouped convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map;

[0064] The feature map data, after being processed by grouped convolutions, is fed into an adaptive average pooling layer for further processing, enabling the model to adapt to inputs of different sizes. Adaptive average pooling automatically calculates the pooling window size and stride to ensure that a predetermined-size output feature map is generated from an input feature map of any size. Specifically, the steps are as follows: First, the size of the pooling window is determined based on the size ratio of the input to the expected output. Then, the average value of the elements contained in each window of the input feature map is calculated, and this average value is assigned to the corresponding position in the output feature map. This process is repeated across the entire input feature map until a complete output feature map is generated. This step, through adaptive average pooling, improves the model's invariance to input changes and reduces the risk of overfitting.

[0065] 107: Flatten the output feature map, then input it into a fully connected layer, and apply an activation function to introduce non-linearity to generate the final classification result. Then apply a loss function to calculate the loss and use gradient descent to update the model parameters.

[0066] The loss function designed in this embodiment of the invention consists of two parts: the true classification loss L. cls And new category reserved loss L vr True classification loss L cls The new class reserved loss L is responsible for measuring the difference between the true label and the model's predicted output. vr Using virtual categories and virtual samples to pre-place unlearned new classes in the embedding space and adjusting the distribution of the embedding space helps the inspection and recognition model within the UAV maintain memory of old categories while learning new ones. The model pre-assigns a set of hypothetical prototypes in the embedding space, treating them as virtual categories, allowing the model to practice integrating and distinguishing different types of information without needing real data. Masking is used to ensure that the influence of any single modality or category does not dominate the learning process. Two loss functions are weighted and combined to form a total loss function, and the gradient of the model parameters is calculated based on this total loss function. Gradient descent is used to progressively update the model parameters to minimize the total loss function. This process is iterated until a preset number of iterations is reached or the model performance no longer shows significant improvement.

[0067] 108: After the model training is completed, the model is deployed on a drone. During the operation, the drone uses its onboard multiple sensors to collect cross-modal data, extract features, and identify scenes and entities.

[0068] During the inspection process, drones can collect cross-modal data in real time and use this data to incrementally learn the in-flight model, thereby improving classification accuracy.

[0069] In summary, the embodiments of the present invention, through steps 101-108, achieve incremental learning using incomplete cross-modal data, complete missing data collected by sensors, improve data integrity, enhance the model's ability to mine existing incomplete cross-modal data and learn new and old categories, and improve the accuracy of UAV identification and the efficiency of UAV inspection.

[0070] Example 2

[0071] The following is combined Figure 1 The calculation formulas and tables further illustrate the scheme in Example 1, as detailed below:

[0072] I. Model Framework

[0073] The model framework consists of a missing data completion module, an interactive fusion module, an adaptive average pooling layer, a classifier, and a forgetting prevention module. See [link to relevant documentation]. Figure 1 As shown. The following three components form the core of the method:

[0074] (1) Missing data completion module: During the training process, a masked autoencoder is used to complete the missing data collected by the sensor, thereby improving the data quality and completeness and reducing the impact of incomplete data.

[0075] The aforementioned masking autoencoder is well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0076] (2) Interactive fusion module: Previous cross-modal learning strategies would lead to a single modality dominating. Therefore, this embodiment of the invention designs an interactive fusion module, which enables the model to effectively integrate information from different modalities and learn each modality on an average basis.

[0077] (3) Forgetting prevention module: Since the interactive fusion module alone is insufficient to solve the problem of modal forgetting in cross-modal incremental learning, this embodiment of the invention designs a forgetting prevention module, which pre-allocates virtual samples and virtual class labels in the embedding space as placeholders for unseen samples and unseen categories, which helps the model to remain adaptable to new knowledge while retaining old knowledge.

[0078] II. Introduction to the Dataset

[0079] 1. LasHeR-CIL dataset

[0080] The LasHeR dataset contains 1,224 spatially aligned visible light and thermal infrared video pairs, totaling over 730,000 frames, each with manually annotated bounding boxes. Using these bounding boxes, samples were cropped to create the LasHeR-CIL dataset, comprising 18 classes and 2,324 samples. The dataset exhibits high diversity in terms of captured object categories, camera perspective, scene complexity, and environmental factors (e.g., season, weather, day / night). In LasHeR-CIL, all classes are divided into three groups: 6 / 1, 6 / 2, and 6 / 3, where the first number represents the number of classes in the base session, and the second number represents the number of classes added in each incremental session.

[0081] 2. VTDV-CIL dataset

[0082] VTUAV comprises 500 visible-thermal-infrared sequences, totaling 1.7 million high-resolution (1920x1080 pixels) frame pairs. In this embodiment, bounding boxes were used to extract 10 categories from VTUAV, and 5 categories were collected from different scenes such as urban roads and residential areas using a Zenmuse XT 2 camera mounted on a DJI M200 aircraft, to construct a cross-modal CIL dataset containing 6,429 samples. In VTDV-CIL, all categories are similarly divided into three groups: 3 / 1, 3 / 2, and 3 / 3. Similarly, the first number represents the number of categories in the base session, and the second number represents the number of new categories added in each incremental session.

[0083] III. Missing Data Completion Module

[0084] In this embodiment of the invention, a masking autoencoder is used to complete the missing information in practical applications. The masking autoencoder learns an effective representation of the data by masking a part of the input data and then training a model to predict these masked parts.

[0085] Before the data is input into the model, it is first partially masked. A portion of the data is randomly selected from each sample and either set to zero or replaced with noise to simulate missing or corrupted data. Next, the masked data is fed into the encoder, which consists of a series of neural network layers that compress the input data (the partially masked data) into a smaller, denser internal representation called the latent space representation. This latent space representation is then fed into the decoder. The decoder's task is to expand the compressed latent space representation back to the dimensions of the original data, attempting to reconstruct a complete version of the original data. The reconstructed output (the decoder's output) is compared to the original, unmasked input data, and the difference between the two is quantified by a loss function. The model is then updated via backpropagation to minimize the error.

[0086] By completing the missing data, the quality and completeness of the data were improved, and the impact of incomplete data was mitigated.

[0087] IV. Interactive Integration Module

[0088] This invention provides an interactive fusion module, which includes three parts: channel mixing, grouped convolution, and adaptive average pooling.

[0089] Channel blending assigns channels to different predefined groups and recombines feature channels from different modalities. Mathematically, channel blending is defined as:

[0090]

[0091] Where || represents connection, and This represents feature groups extracted from two different modalities, with feature channels evenly distributed into the different groups. t represents the t-th scene, i represents different modalities, N is the number of groups, and C is the number of channels. and This indicates a channel blending operation performed on feature groups extracted from two different modalities. It represents the j-th element of the feature channel in the i-th modality of the t-th scene.

[0092] After channel mixing, the data is input into a grouped convolutional layer. This layer first divides the input feature map into several independent groups, each containing a subset of the feature map. Similarly, the convolutional kernels are also divided into a corresponding number of groups, with each kernel processing only the input feature map of its corresponding group. After grouping, the convolutional kernels within each group independently perform convolution operations with their corresponding feature map, generating an independent output feature map for each group. Each group's kernel interacts only with the feature map of its corresponding group. After each group performs independent convolution operations, the final output feature map is formed by concatenating the output feature maps of all groups along the depth dimension, ensuring that each channel of the output feature map contains information from a specific group. Assume the input feature map X has C... in G channels are divided into G groups by grouped convolution, so each group has G channels. There are G channels, and for each group, an independent convolutional kernel W is assigned. Each convolutional kernel operates only on the channels of its corresponding group. The output feature map Y is divided into G groups, then each group has G channels. There are 1 channel. Given X is H×W×C. in The input feature map is of dimension K, where K is A convolution kernel of dimension m×n, where m×n is the spatial dimension of the convolution kernel, and Y is H'×W'×C. out 3D output feature map, C outHere, H represents the number of channels in the output feature map, H' represents the height of the input feature map, and W' represents the width of the output feature map. The process of grouped convolution can be represented as:

[0093] For each position (i,j) in the output feature map, the following calculation is performed:

[0094]

[0095] Where (i,j) represents the position in the output feature map, (u,v) represents the spatial index inside the convolution kernel, g is the group index, and X g and Y g K represents the g-th group of the input and output graphs. g It is the convolution kernel of the g-th group.

[0096] After grouped convolution, the data is fed into an adaptive average pooling layer for processing. The adaptive average pooling layer first determines the size of the pooling window based on the ratio of the input to the expected output size. Then, it calculates the average value of the elements within each window of the input feature map and assigns this average value to the corresponding position in the output feature map. This process is repeated across the entire input feature map until a complete output feature map is generated.

[0097] Let the input feature map X have dimensions H×W, and the target output dimension be H'×W'. Adaptive average pooling is represented as:

[0098]

[0099] Where Y(i,j) is the element of the output feature map at position (i,j), and R i,j Area(R) is the region in the input feature map corresponding to the output (i,j). i,j ) is region R i,j The number of pixels contained in region R, where (h,w) is the region R. i,j The pixel coordinates within.

[0100] Through the interactive fusion module, data features from different modalities are integrated, enhancing the richness of feature representation. This enables the model to capture more details and information, helping it avoid over-reliance on features from a single data source.

[0101] V. Forgetting Prevention Module

[0102] This invention proposes a forgetting prevention module that pre-allocates a set of hypothetical prototypes in the embedding space and treats them as virtual categories. Then, interpolation is performed between the feature representations of different data samples to create new virtual hybrid features to generate virtual samples.

[0103] The process is as follows: Two samples x1 and x2, along with their corresponding labels y1 and y2, are randomly selected from the training set. These two samples are then input into the neural network and propagated forward to the selected hidden layer. Let the outputs of the selected hidden layer for x1 and x2 be z1 and z2, respectively. Interpolation is performed on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient controlling the degree of interpolation. The corresponding labels are also interpolated accordingly: y = λy1 + (1-λ)y2. The new representation z obtained through interpolation is then propagated forward through the remaining part of the neural network to obtain the final output, i.e., the virtual sample.

[0104] To reserve space for new categories, a virtual loss was constructed:

[0105]

[0106] in, y' is a virtual category label, y' is a pseudo-label in an existing category, and L img L represents the new class space preserved by the Mask function. mi and L mc To avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed samples. The Mask function is defined as:

[0107]

[0108] in, This represents the Hadamard product (element-wise multiplication), used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class; all other class positions are 0, and the current class position is 1. The 1-OneHot(y) operation sets the positions of known classes to 0 and the positions of unknown classes to 1. This clears the components of the feature vector corresponding to the known classes, while retaining the components that might belong to the unknown classes. The objective function is ultimately defined as:

[0109] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0110] Among them, L cls This represents the basic loss function calculated using cosine similarity, where λ1 is a balancing factor used to adjust L. vrThe relative importance of x and y is given by x, which represents the original input data of the model, and y, which represents the true class label corresponding to the input x.

[0111] VI. Cross-modal information fusion module

[0112] The above-mentioned modal completion module, interactive fusion module, and forgetting prevention module are combined to form the cross-modal information fusion module of this invention.

[0113] The working process of the cross-modal information fusion module applied to this invention is as follows:

[0114] First, a masked autoencoder is used to complete the missing cross-modal data, improving its quality. Second, an interactive fusion module is used to perform channel mixing, grouped convolution, and adaptive average pooling on the completed data, integrating the features of the cross-modal data to help the model capture more information. Finally, a forgetting prevention module is used to pre-assign a set of hypothetical prototypes considered as virtual categories in the embedding space and create new virtual fusion features to generate augmented data samples. By calculating the value of the designed total loss function, gradient descent is applied to update the model parameters until the model performance no longer improves significantly or reaches the preset number of iterations. The final trained model is then obtained.

[0115] VII. Classifier

[0116] The classifier's structure includes a fully connected layer whose output layer has the number of neurons corresponding to the total number of all classes encountered so far (current stage and all previous stages). As incremental learning progresses, the number of neurons in the output layer increases accordingly whenever a new class is introduced.

[0117] This design ensures that the classifier can adapt to an increasing number of categories, thereby continuously learning and adapting to new data.

[0118] In summary, the embodiments of the present invention, through the above-described parts, achieve the completion of missing modal data in practical applications, alleviate the problem of a single modality dominating, improve the model's learning ability for new and old classes, reduce the catastrophic forgetting phenomenon of the model, improve the target detection capability of UAVs, and significantly improve inspection efficiency.

[0119] Example 3

[0120] The following is combined Figure 2 , Figure 3 The specific experimental data verifies the feasibility of the schemes in Examples 1 and 2, as detailed below:

[0121] The method was validated using the LasHeR-CIL and VTDV-CIL datasets.

[0122] On the LasHeR-CIL dataset, the base class learning rate and epoch were set to 0.05 and 1000, the new class learning rate and epoch were set to 0.05 and 500, and the number of interactive fusion groups was set to 8. On the VTDV-CIL dataset, the base class learning rate and epoch were set to 0.1 and 1000, the new class learning rate and epoch were set to 0.1 and 300, and the number of interactive fusion groups was 4. The final results are the average of 5 experiments. For fair comparison, all class incremental learning methods used additive feature maps to achieve cross-modal class incremental learning.

[0123] In this embodiment of the invention, the experiments use average accuracy (ACC) and performance degradation rate (PD) as evaluation metrics for incremental learning. ACC is primarily used to evaluate the overall performance of the model and is calculated using the average accuracy across all learning stages.

[0124]

[0125] ACC t ACC represents the Top-1 accuracy after the t-th stage. t A higher value indicates better prediction accuracy. PD primarily assesses the degree of forgetting in the model during continuous learning. PD can be calculated as the difference between the accuracy of the first stage and the accuracy of the last stage:

[0126] PD = ACC0 - ACC T

[0127] Where ACC0 represents the precision of the first stage, ACC T This indicates the precision of the final stage.

[0128] Furthermore, considering that different models may have varying accuracy in the first stage, which could affect the evaluation of PD, a new metric for measuring the degree of forgetting is proposed, called Normalized Performance Degradation Rate (NPD), as follows:

[0129]

[0130] Both PD and NPD are indicators used to assess the degree of forgetting; the lower the value, the less forgetting there is.

[0131] The experimental results were compared with the following five baseline methods: DER, FOSTER, MEMO, PODNet, and MRFA+PDF. For all methods, five experiments were conducted, and the final result was the average of the five experiments.

[0132] Table 1 shows the accuracy, performance degradation rate, and normalized performance degradation rate for different new classes in each incremental stage on the LasHeR-CIL dataset, comprehensively demonstrating the performance of the model provided by the embodiments of the present invention. The embodiments of the present invention take into account potential data incompleteness, single-modality dominance, and forgetting problems in cross-modal incremental learning, and perform data completion and fusion processing, thus improving model performance.

[0133] Table 1 shows the incremental learning results on the LasHeR-CIL dataset (accuracy, performance degradation rate, normalization performance degradation rate).

[0134]

[0135]

[0136] Furthermore, the effectiveness of the components in this embodiment of the invention is verified, as shown in Table 2. Here, Completion represents data completion, Visible represents the visible light mode, and Thermal represents the infrared mode. The experimental results demonstrate the effectiveness and necessity of the design of each component of the model.

[0137] Table 2 Ablation experimental results on the LasHeR-CIL dataset

[0138]

[0139] Example 4

[0140] A drone inspection device based on incremental learning of missing cross-modal data classes. The device includes a processor 1 and a memory 2. The memory 2 stores program instructions. The processor 1 calls the program instructions stored in the memory 2 to cause the device to execute the following method steps in Embodiment 1:

[0141] Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each modality hidden in the data; mix the feature channels of different modalities in the input feature map;

[0142] A set of convolutional kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map.

[0143] The output feature map is flattened and then fed into a fully connected layer. An activation function is applied to introduce non-linearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using gradient descent. After the model is trained, it is deployed on a drone, which then identifies the scene and entities.

[0144] Specifically, the mixing of feature channels of different modalities in the input feature map involves:

[0145] The features from each modality in the input feature map are divided into different channels. The feature channels from each modality are evenly grouped. Channels are selected from each group and mixed with the corresponding group of another modality.

[0146] The loss function consists of two parts: the true classification loss L. cls And new category reserved loss L vr ,

[0147] True classification loss L cls Used to measure the difference between the true label and the model's predicted output;

[0148] New category of reserved loss L vr Virtual categories and virtual samples are used to pre-place new, unlearned categories in the embedding space, adjusting the distribution of the embedding space and helping the inspection and recognition model in the UAV maintain its memory of old categories while learning new categories.

[0149] The missing data collection from the drone is completed by using a masked autoencoder during training to complete the missing data collected by the sensors.

[0150] Channel blending assigns channels to different predefined groups and recombines feature channels from different modalities. Channel blending is defined as follows:

[0151]

[0152] Where || represents connection, and This represents feature groups extracted from two different modalities, with feature channels evenly distributed into the different groups. t represents the t-th scene, i represents different modalities, N is the number of groups, and C is the number of channels. and This indicates a channel blending operation performed on feature groups extracted from two different modalities. It represents the j-th element of the feature channel in the i-th modality of the t-th scene.

[0153] The virtual samples are:

[0154] Two samples x1 and x2 and their corresponding labels y1 and y2 are randomly selected from the training set. These two samples are then input into the neural network and propagated forward to the selected hidden layer. Let the outputs of the selected hidden layer for x1 and x2 be z1 and z2, respectively. Interpolation is performed on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient. The corresponding labels are then interpolated accordingly: y = λy1 + (1-λ)y2. The new representation z obtained by interpolation is then propagated forward through the remaining part of the neural network to obtain the final output.

[0155] The virtual loss corresponding to the sample is:

[0156]

[0157] in, y' is a virtual category label, y' is a pseudo-label in an existing category, and L img L represents the new class space preserved by the Mask function. mi and L mc To avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed samples, and the Mask function is defined as:

[0158]

[0159] in, Let represent the Hadamard product, used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class; all other class positions are 0, and the current class position is 1. The operation 1-OneHot(y) sets the known class positions to 0 and the unknown class positions to 1. The objective function is ultimately defined as:

[0160] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0161] Among them, L cls This represents the basic loss function calculated using cosine similarity, where λ1 is a balancing factor used to adjust L. vr The relative importance of x and y is given by x, which represents the original input data of the model, and y, which represents the true class label corresponding to the input x.

[0162] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0163] The execution entities of the processor 1 and memory 2 mentioned above can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementation, the embodiments of the present invention do not limit the execution entity and can select it according to the needs of actual application.

[0164] Data signals are transmitted between memory 2 and processor 1 via bus 3, which will not be described in detail in this embodiment of the invention.

[0165] Example 5

[0166] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0167] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0168] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0169] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0170] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0171] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0172] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A UAV inspection method based on incremental learning of missing cross-modal data classes, characterized in that, The method includes: Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each modality hidden in the data; mix the feature channels of different modalities in the input feature map; A set of convolutional kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map. The output feature map is flattened and then fed into a fully connected layer. An activation function is applied to introduce non-linearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using gradient descent. After the model is trained, it is deployed on a drone, which identifies scenes and entities. The specific steps of mixing feature channels of different modalities in the input feature map are as follows: The features from each modality in the input feature map are divided into different channels. The feature channels from each modality are evenly grouped. Channels are selected from each group and mixed with the corresponding group of another modality. The channel blending process assigns channels to different predefined groups and recombines feature channels from different modalities. Channel blending is defined as follows: ； ； in, Indicates a connection. and This represents feature groups extracted from two different modalities, with feature channels evenly distributed into the different groups. , Representing the One scenario, Representing different modes, It is the number of groups. It is the number of channels. and This indicates a channel blending operation performed on feature groups extracted from two different modalities. Indicates the first In the first scenario The first feature channel in the modality Each element.

2. The UAV inspection method based on incremental learning of missing cross-modal data classes according to claim 1, characterized in that, The loss function consists of two parts: the true classification loss. and new categories of reserved losses , The true classification loss Used to measure the difference between the true label and the model's predicted output; The new type of reserved loss Virtual categories and virtual samples are used to pre-place new, unlearned categories in the embedding space, adjusting the distribution of the embedding space and helping the inspection and recognition model in the UAV maintain its memory of old categories while learning new categories.

3. The UAV inspection method based on incremental learning of missing cross-modal data classes according to claim 2, characterized in that, The process of completing the missing data collected by the UAV involves using a masked autoencoder during training to complete the missing data collected by the sensors.

4. The UAV inspection method based on incremental learning of missing cross-modal data classes according to claim 2, characterized in that, The virtual sample is: Two samples were randomly selected from the training set. and and their corresponding tags and The two samples are input into the neural network and forward-propagated to the selected hidden layer. Let the selected hidden layer be... and The outputs are respectively and Output in the hidden layer and Interpolation is performed on the data to generate a new internal representation. : ,in It is an interpolation coefficient used to perform corresponding interpolation on the corresponding label: The new representation obtained by interpolation The remaining part of the neural network continues to propagate forward to obtain the final output.

5. The UAV inspection method based on incremental learning of missing cross-modal data classes according to claim 4, characterized in that, The virtual loss corresponding to the sample is: ； in, These are virtual category tags. It is a pseudo-label in the existing category. Indicates passage The new class space reserved by the function and This is used to avoid excessive compression of the old category space. This indicates that the network responds to the input samples. The generated feature representation, This represents the feature representation generated by the network for mixed samples. The function is defined as: ； in, This represents the Hadamard product, used to multiply an eigenvector by the element-wise complement of a one-hot encoded vector. This indicates the label of the currently known category; other categories are at position 0, and the current category is at position 1. The operation sets the positions of known categories to 0 and the positions of unknown categories to 1. The objective function is ultimately defined as follows: ； in, This represents the basic loss function calculated using cosine similarity. It is a balancing factor used to adjust The relative importance of This represents the original input data of the model. Representation and Input The corresponding real category label.

6. A drone inspection device based on incremental learning of missing cross-modal data classes, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-5.

Images