Unmanned aerial vehicle inspection method and device based on missing cross-modal data class incremental learning

Abstract

The invention discloses an unmanned aerial vehicle inspection method and device based on missing cross-modal data class incremental learning. The method comprises the following steps: complementing missing data collected by an unmanned aerial vehicle; preprocessing the complemented data to obtain features of each mode hidden in the data; feature channels of different modes in the input feature map are mixed; allocating a set of convolution kernel to each group in the mixed input feature map, performing grouping convolution, and performing adaptive average pooling on the feature map to obtain a final output feature map; flattening the output feature map, inputting the flattened output feature map into a full connection layer, applying an activation function to introduce nonlinearity, generating a final classification result, applying a loss function to calculate loss, and updating model parameters by using a gradient descent method; after model training is completed, the model is deployed to the unmanned aerial vehicle, and the unmanned aerial vehicle identifies the scene and the entity. The device comprises a processor and a memory. According to the invention, the identification and analysis capability of the unmanned aerial vehicle in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle inspection in intelligent unmanned systems, and in particular to a method and device for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data. Background Art

[0002] Drone inspection systems mainly use drones to implement automated monitoring and inspection of key infrastructure, such as power lines, oil and gas pipelines, railways and bridges. Such systems are also widely used in agriculture, forest monitoring and urban management. Its main advantages include improving work efficiency, reducing operating costs and reducing personnel safety risks. When performing inspection tasks, drones need to collect and analyze data to make automated decisions. During the day, drones use high-definition cameras to capture clear images for analysis; at night, they usually rely on infrared cameras to process images obtained due to insufficient light.

[0003] Current automatic detection models mainly process images captured by high-definition or infrared cameras. These models train deep neural networks based on the collected data during the training phase, and then deploy these networks to analyze newly acquired images to assess the current situation. However, due to the large differences in information and features of images under daytime and nighttime conditions, this single model often has difficulty adapting to both situations at the same time; in addition, in actual operation, some perception modalities may not be able to provide data due to technical failures, environmental interference or other external factors. Even if the modality is available, it may be due to the limitations of the sensor coverage or the limitations of the sensor itself that make it impossible to capture the full picture of the scene, resulting in data loss.

[0004] Cross-modal learning focuses on learning and reasoning from the associations between different modalities, and enhances the perception and understanding capabilities of the model by complementing and sharing information between data from different modalities. In this learning category, the model integrates information from different perceptual modalities to improve its understanding of the environment and decision-making capabilities. Incremental learning enables the model to maintain its memory of existing data while absorbing new data. However, cross-modal learning relies on the complementarity of information between different modalities to improve the performance of the model. When data from a certain modality is missing, the model cannot fully utilize the features of that modality, resulting in incomplete information and reduced overall performance of the model. Missing data may also lead to an uneven distribution of data between modalities, causing the model training to completely favor one modality. In incremental learning, the model needs to keep its memory of old data while constantly learning new data. Missing data exacerbates the phenomenon of forgetting because the missing data may contain key features, making it difficult for the model to keep its memory of these features during the incremental learning process.

[0005] In cross-modal incremental learning, the problem of modality forgetting is an important challenge. In the incremental learning process, due to the different feature distributions and learning dynamics between different modalities, the model may forget the previously learned knowledge about different modalities when learning a new modality. In addition, in cross-modal learning, each modality usually provides different types of information. Ideally, the model should extract and utilize information from each modality in a balanced manner. However, in practical applications, a certain modality may gradually become dominant, resulting in the problem of modality competitive forgetting. This advantage may cause the model to rely too much on the information of a certain modality and ignore the useful information provided by other modalities, which in turn affects the overall learning effect and decision quality.

[0006] To solve the problem of missing data, a variety of strategies have been proposed, including: difference filling method, difference method, matrix decomposition method and other methods to fill in the data and reduce the impact of missing data. Although these strategies can solve the problem to a certain extent, they often use basic statistics to fill in the data, ignore the correlation between data, or have high computational overhead and high computational complexity; to solve the problem of competitive forgetting of modalities, existing methods mainly adopt the strategies of introducing modal balance mechanism, data enhancement and resampling. However, there are also problems of high resource overhead and subjective setting of weights of each modality. Therefore, how to effectively solve the problem of missing data and competitive forgetting of modalities in cross-modal incremental learning, while taking into account storage and computing efficiency, has become the key to whether the drone inspection model can be installed on drones on a large scale for application. Summary of the invention

[0007] The present invention provides a method and device for drone inspection based on incremental learning of missing cross-modal data. The present invention takes into account that in practical applications, the cross-modal data received by the drone during the inspection process may be incomplete, which leads to the problem that the traditional incremental learning method is not effective in these scenarios. A method based on incremental learning of incomplete cross-modal data is designed. The method integrates three strategies of completion, interactive fusion and forgetting prevention, enhances the integrity of the data, realizes the maximum utilization of information between different modalities, improves the classification accuracy of the drone inspection model, strengthens the model's processing ability for missing cross-modal data, and effectively reduces the forgetting problem of the drone inspection model in the incremental learning process, can better process and utilize information from different sensors, and improves the recognition and analysis capabilities of drones in complex environments. See the following description for details:

[0008] In the first aspect, a drone inspection method based on incremental learning of missing cross-modal data is provided, the method comprising:

[0009] Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each mode hidden in the data; mix the feature channels of different modes in the input feature map;

[0010] A set of convolution kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map.

[0011] The output feature map is flattened and then input into the fully connected layer. The activation function is applied to introduce nonlinearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using the gradient descent method. After the model training is completed, it is deployed on the drone, which recognizes the scene and entities.

[0012] The mixing of feature channels of different modes in the input feature map is specifically as follows:

[0013] The features from each modality in the input feature map are divided into different channels, the feature channels from each modality are evenly grouped, and channels are selected from each group and mixed with the corresponding group of the other modality.

[0014] Among them, the loss function consists of two parts, the real classification loss L cls and the new class reservation loss L vr ,

[0015] The true classification loss L cls Used to measure the difference between the true label and the model's predicted output;

[0016] The new class reserves loss L vr Virtual categories and virtual samples are used to pre-occupy places for new categories that have not been learned in the embedding space, and the distribution of the embedding space is adjusted to help the inspection and recognition model in the drone maintain its memory of old categories when learning new categories.

[0017] The method of completing the missing data collected by the drone includes: using a masked autoencoder to complete the missing data collected by the sensor during the training process.

[0018] The channel mixing allocates channels to different predefined groups and recombines feature channels from different modalities. The channel mixing is defined as:

[0019]

[0020] Among them, || represents connection, and Represents feature groups extracted from two different modalities, and feature channels are evenly divided into different groups t represents the tth scene, i represents different modes, N is the number of groups, C is the number of channels, and Indicates that channel mixing operation is performed on feature groups extracted from two different modalities; represents the jth element of the feature channel in the i-th modality in the t-th scene.

[0021] Wherein, the virtual sample is:

[0022] Randomly select two samples x1 and x2 and their corresponding labels y1 and y2 in the training set, input the two samples into the neural network respectively, and propagate forward to the selected hidden layer. Suppose the outputs of the selected hidden layer for x1 and x2 are z1 and z2 respectively; interpolate on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient, and interpolate the corresponding labels accordingly: y = λy1 + (1-λ)y2, and continue to propagate the new representation z obtained by interpolation forward through the rest of the neural network to obtain the final output.

[0023] Among them, the virtual loss corresponding to the sample is:

[0024]

[0025] in, is the virtual category label, y' is the pseudo label in the existing category, L img Represents the new category space retained by the Mask function, L mi and L mc Used to avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed sample, and the Mask function is defined as:

[0026]

[0027] in, Represents the Hadamard product, which is used to multiply the feature vector by the element-by-element complement of a one-hot encoding vector. The one-hot encoding vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1. Through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1. The objective function is finally defined as:

[0028] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0029] Among them, L cls represents the basic loss function calculated by cosine similarity, λ1 is a balancing factor used to adjust L vr The relative importance of , x represents the original input data of the model, and y represents the true category label corresponding to the input x.

[0030] In the second aspect, a drone inspection device based on incremental learning of missing cross-modal data is provided, characterized in that the device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0031] A third aspect, a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0032] The beneficial effects of the technical solution provided by the present invention are:

[0033] 1. The present invention takes into account the problem of missing cross-modal data received by drones in real applications, and uses a masked autoencoder to automatically complete the missing data, enhance the quality of the data, improve the integrity of the data, and provide more stable and reliable input data;

[0034] 2. The present invention develops a cross-modal information fusion module for the cross-modal data generated in the UAV inspection and recognition scenario. This module enhances the diversity of features by establishing connections between different data modalities, helps retain the memory of old knowledge, and effectively reduces the problem of catastrophic forgetting; it enables the model to better process and utilize information from different sensors, and improves the recognition and analysis capabilities of UAVs in complex environments;

[0035] 3. The present invention effectively solves the data sparsity problem caused by the inability to collect data due to the failure of some sensors; by integrating richer data from other modalities, the present invention can effectively make up for the lack of data in some modalities, thereby improving the efficiency of data use and the overall performance of the model; this cross-modal interactive fusion strategy enhances the model's ability to handle uneven data sources, ensuring good performance even in the absence of data;

[0036] 4. The present invention can achieve state-of-the-art performance in incremental learning tasks such as drone identification, with average accuracy improvements of up to 4.96%, 6.53%, and 3.86% compared to the comparison methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1This is a flowchart of a drone inspection method based on incremental learning of missing cross-modal data;

[0038] Figure 2 A schematic diagram of using a modality completer to complete missing cross-modal data;

[0039] Figure 3 A schematic diagram of a large cross-modal dataset LasHeR-CIL;

[0040] Figure 4 A schematic diagram of a large cross-modal dataset VTDV-CIL;

[0041] Figure 5 Schematic diagram for comparison;

[0042] Among them, the three sub-figures are the situations when the missing rates are 0.3, 0.5, and 0.7 respectively. The figure shows the improvement in accuracy of the cross-modal fusion completion strategy compared with other advanced methods when applied to the VTDV-CIL dataset. It can be seen that after multiple stages of different missing rates, the accuracy of the cross-modal fusion completion strategy is significantly improved.

[0043] Figure 6 Schematic diagram for comparison;

[0044] Among them, the two sub-figures are respectively applied on the LasHeR-CIL dataset and the VTDV-CIL dataset. This figure shows the performance comparison of whether the forgetting prevention strategy is included when the missing rate is set to 0.3. "Visible" represents the visible light modality, "Thermal" represents the infrared light modality, "ADD" represents the interactive fusion strategy, and "-F" represents the use of the forgetting prevention strategy. It can be seen that the model performance when the forgetting prevention strategy is used is significantly better than the model without the forgetting prevention strategy.

[0045] Figure 7 It is a schematic diagram of parameter analysis;

[0046] Among them, the two sub-figures are the parameter analysis results of the balance factor λ1 and the number of channel mixing groups γ when the missing rate is set to 0.5. The left figure is ACC and the right figure is NPD. This figure shows the performance comparison under different parameter settings applied to the LasHeR-CIL dataset. The value of λ1 is selected from {0.0001, 0.001, 0.01, 0.1, 1}, and γ is selected from {2, 4, 6, 8, 16, 32}. It can be seen that the model performance is best when λ1 is set to 0.01 and γ is set to 8.

[0047] Figure 8 The schematic diagram of the structure of a drone inspection system based on incremental learning of incomplete cross-modal data.

[0048] Table 1 shows the comparison of accuracy, performance degradation rate, and normalized performance degradation rate of different methods on the LasHeR-CIL dataset;

[0049] Table 2 shows the ablation experiments conducted on the LasHeR-CIL dataset, demonstrating the effectiveness of each component of the proposed method. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0051] Example 1

[0052] In the process of performing inspection tasks, the drone can adjust and optimize the internal model in real time according to the newly collected data to adapt to the dynamically changing environment. However, when performing tasks, the drone may encounter situations where some sensors are damaged or blocked, resulting in incomplete cross-modal data received, and modal forgetting is very common during the learning process. In order to solve the problems existing in the background technology, incremental learning of new categories in drone inspection tasks will lead to serious degradation of the recognition ability of old categories. The embodiment of the present invention proposes a drone inspection method based on incremental learning of incomplete cross-modal data, which aims to reduce the impact of missing data in target detection, and at the same time can incrementally learn new categories and maintain the recognition ability of old categories, thereby through fast and efficient model updates and continuous learning, see Figure 1 , the method comprises the following steps:

[0053] 101: The drone performs inspection tasks and obtains missing data;

[0054] According to the predetermined routes and areas, drones are deployed to perform inspection tasks. During the inspection process, drones collect environmental information and data of different modes through the various sensors they carry (cameras, infrared cameras, lidar, etc.), and transmit them back to the ground processing center in real time. However, the sensors may be blocked or damaged during operation, resulting in incomplete data collected.

[0055] 102: Complete the missing data collected;

[0056] Among them, a masked autoencoder is used to automatically complete the missing data. In practical applications, the masked autoencoder first converts the input data with missing values ​​into a low-dimensional hidden representation through the encoder. This representation captures the core features of the unmasked part of the data. Then, the decoder part uses these core features to try to reconstruct the complete form of the original data, including the missing parts. This step significantly enhances the data quality and improves the data integrity, providing more stable and reliable input data for subsequent processing.

[0057] 103: Preprocess the completed data to obtain the features of each modality hidden in the data;

[0058] Use convolution, pooling and other technologies to extract features from the completed cross-modal data, extract data containing key features of different modalities from a large amount of data, ignore the noise contained in the data, compress the data volume, and generate feature maps as input for subsequent processing. Among them, the number of convolution kernels in the convolution layer determines the number of channels in the feature map. Each channel corresponds to the response of the input data under a specific feature.

[0059] 104: Mix feature channels of different modes in the input feature map;

[0060] Among them, the features from each modality in the input feature map are divided into different channels, and the feature channels from each modality are evenly grouped. Before the actual convolution operation, channels are selected from each group and mixed with the corresponding group of the other modality. This operation enhances the connection between the data of different modalities.

[0061] 105: Assign a set of convolution kernels to each group in the mixed input feature map for group convolution;

[0062] Among them, the input feature maps after channel mixing are divided into several independent groups, each group contains a part of the feature map. Similarly, the convolution kernels are also divided into a corresponding number of groups, and each group of convolution kernels only processes the input feature map of the corresponding group. After the grouping is completed, the convolution kernels in each group will independently perform convolution operations with their corresponding feature maps, and each group generates an independent output feature map. The convolution kernels of each group only interact with the feature map of its corresponding group. Each group performs convolution operations independently and generates a part of the output feature map independently. Finally, the output feature map is formed by splicing the output feature maps of all groups in the depth dimension, so that each channel of the output feature map can contain information from a specific group. Such operations ensure that the network can handle complex feature extraction tasks while reducing parameters.

[0063] 106: After the group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map;

[0064] The feature map data after group convolution is sent to the adaptive average pooling layer for processing, so that the model can adapt to inputs of different sizes. Adaptive average pooling can automatically calculate the size and step size of the pooling window to ensure that an output feature map of a predetermined size is generated from an input feature map of any size. The specific operation is as follows: First, the size of the pooling window is determined based on the size ratio of the input and the expected output. Then, for each window in the input feature map, the average value of the contained elements is calculated and assigned to the corresponding position in the output feature map. This process is repeated on the entire input feature map until a complete output feature map is generated. This step improves the model's invariance to input changes through adaptive average pooling and reduces the risk of overfitting of the model.

[0065] 107: Flatten the output feature map, then input it into the fully connected layer, apply the activation function to introduce nonlinearity, generate the final classification result, then apply the loss function to calculate the loss, and use the gradient descent method to update the model parameters;

[0066] The loss function designed in the embodiment of the present invention consists of two parts: the real classification loss L cls and the new class reservation loss L vr The true classification loss L cls Responsible for measuring the difference between the true label and the model prediction output, the new class reservation loss L vr Using virtual categories and virtual samples to pre-occupy new categories that have not been learned in the embedding space and adjusting the distribution of the embedding space can help the inspection and recognition model in the drone maintain its memory of old categories when learning new categories. The model pre-assigns a set of hypothetical prototypes in the embedding space and treats them as virtual categories, allowing the model to practice integrating and distinguishing different types of information without the need for real data. Masks are used to ensure that the influence of any single modality or category does not dominate the learning process. The two loss functions are weightedly combined to form a total loss function, and the gradient of the model parameters is calculated based on the total loss function. The model parameters are gradually updated using the gradient descent method to minimize the total loss function. This process is iterated until the preset number of iterations is reached or the model performance is not significantly improved.

[0067] 108: After the model training is completed, the model is deployed on the drone. During the operation, the drone uses the various sensors it carries to collect cross-modal data, extract features, and identify scenes and entities.

[0068] Among them, drones can collect cross-modal data in real time during the inspection process, and use these data to perform incremental learning on the in-machine model to improve classification accuracy.

[0069] In summary, the embodiment of the present invention realizes the use of incomplete cross-modal data for incremental learning through the above steps 101 to 108, completes the missing data collected by the sensor, improves data integrity, enhances the model's ability to mine existing incomplete cross-modal data and learn new and old categories, and improves the accuracy of drone recognition and the efficiency of drone inspection.

[0070] Example 2

[0071] Combine the following Figure 1 , calculation formulas, and tables are used to further introduce the scheme in Example 1, as described below:

[0072] 1. Model Framework

[0073] The model framework consists of a missing data completion module, an interactive fusion module, an adaptive average pooling layer, a classifier, and a forgetting prevention module, see Figure 1 The following three components are the core of the method:

[0074] (1) Missing data completion module: During the training process, a masked autoencoder is used to complete the missing data collected by the sensor, improve the data quality and completeness, and reduce the impact of incomplete data.

[0075] Among them, the above-mentioned masked autoencoder is well known to those skilled in the art, and will not be described in detail in the embodiment of the present invention.

[0076] (2) Interactive fusion module: Previous cross-modal learning strategies would result in a single modality occupying an absolute dominant position. Therefore, an interactive fusion module is designed in the embodiment of the present invention, so that the model can effectively integrate information from different modalities and learn each modality on average.

[0077] (3) Forgetting prevention module: Since using only the interactive fusion module is not sufficient to solve the problem of modal forgetting in cross-modal incremental learning, an embodiment of the present invention designs a forgetting prevention module, which pre-assigns virtual samples and virtual class labels in the embedding space as placeholders for unseen samples and unseen categories, helping the model to remain adaptable to new knowledge while retaining old knowledge.

[0078] 2. Dataset Introduction

[0079] 1. LasHeR-CIL dataset

[0080] The LasHeR dataset contains 1,224 pairs of spatially aligned visible and thermal infrared video pairs, totaling more than 730K frame pairs, each of which has been manually annotated with bounding boxes. Using these bounding boxes, samples of different categories were cropped to create a dataset called LasHeR-CIL, which includes a total of 18 categories and 2,324 samples. The dataset has extremely high diversity in capturing object categories, camera viewpoints, scene complexity, and environmental factors (such as season, weather, day and night, etc.). In LasHeR-CIL, all categories are divided into three groups: 6 / 1, 6 / 2, and 6 / 3, where the first number represents the number of categories in the base session and the second number represents the number of new categories added in each incremental session.

[0081] 2. VTDV-CIL dataset

[0082] VTUAV includes 500 visible-thermal infrared sequences, a total of 1.7 million high-resolution (1920x 1080 pixel) frame pairs. The embodiment of the present invention uses bounding boxes to extract 10 categories from VTUAV, and uses the Zenmuse XT 2 camera equipped with the DJI M200 aircraft to collect 5 categories from different scenes such as urban roads and residential areas to construct a cross-modal CIL dataset, containing 6,429 samples. In VTDV-CIL, all categories are also divided into three groups: 3 / 1, 3 / 2, and 3 / 3. Similarly, the first number represents the number of categories in the base session, and the second number represents the number of new categories added in each incremental session.

[0083] 3. Missing Data Completion Module

[0084] The embodiment of the present invention uses a masked autoencoder to complete the missing information in practical applications. The masked autoencoder learns the effective representation of data by masking part of the input data and then training the model to predict the masked parts.

[0085] Before the data is fed into the model, it is first partially masked. A portion of the data is randomly selected from each data sample and set to zero or replaced with noise to simulate missing or corrupted data. The masked data is then fed into the encoder, which consists of a series of neural network layers that compress the input data (i.e., the partially masked data) into a smaller, dense internal representation, called the latent space representation. The latent space representation is then fed into the decoder. The decoder's task is to expand the compressed representation of the latent space back to the dimensions of the original data, trying to reconstruct a complete version of the original data. The reconstructed output (i.e., the output of the decoder) is compared to the original unmasked input data, and the difference between the two is quantified by a loss function. The model is updated through backpropagation to minimize the error.

[0086] By completing missing data, the data quality and completeness are improved and the impact of incomplete data is reduced.

[0087] 4. Interactive Fusion Module

[0088] The embodiment of the present invention designs an interactive fusion module, which includes three parts: channel mixing, group convolution, and adaptive average pooling.

[0089] Channel mixing assigns channels into different predefined groups and recombine feature channels from different modalities. Mathematically, channel mixing is defined as:

[0090]

[0091] Among them, || represents connection, and Represents feature groups extracted from two different modalities, and feature channels are evenly divided into different groups t represents the tth scene, i represents different modes, N is the number of groups, C is the number of channels, and Indicates that channel mixing operation is performed on feature groups extracted from two different modalities; represents the jth element of the feature channel in the i-th modality in the t-th scene.

[0092] After the channel mixing is completed, the data is input into the grouped convolution layer, which first divides the input feature map into several independent groups, each group containing a part of the feature map. Similarly, the convolution kernels are also divided into a corresponding number of groups, and each group of convolution kernels only processes the input feature map of the corresponding group. After the grouping is completed, the convolution kernels in each group will independently perform convolution operations with their corresponding feature maps, and each group generates an independent output feature map. The convolution kernels of each group only interact with the feature maps of their corresponding group. After each group performs the convolution operation independently, the final output feature map is formed by splicing the output feature maps of all groups in the depth dimension, so that each channel of the output feature map can contain information from a specific group. Suppose the input feature map X has C in channels, and grouped convolution divides these channels into G groups, so each group has channels, and for each group, an independent convolution kernel W is assigned, and each convolution kernel operates only on the channels of its corresponding group. Divide the output feature map Y into G groups, then each group has channels. Given X is H×W×C in dimensional input feature map, K is dimensional convolution kernel, where m×n is the spatial dimension of the convolution kernel, and Y is H'×W'×C out The output feature map of dimension C outRepresents the number of channels of the output feature map, H represents the height of the input feature map, H' represents the height of the output feature map, and W' represents the width of the output feature map. The process of grouped convolution can be expressed as:

[0093] For each position (i, j) in the output feature map, the following calculation is performed:

[0094]

[0095] Among them, (i, j) represents the position in the output feature map, (u, v) represents the spatial index inside the convolution kernel, g is the group index, and X g and Y g represents the g-th group of input and output graphs, K g is the convolution kernel of the g-th group.

[0096] After the group convolution is completed, the data is sent to the adaptive average pooling layer for processing. The adaptive average pooling layer first determines the size of the pooling window based on the ratio of the input and expected output sizes. Then, the average value of the elements in each window of the input feature map is calculated and assigned to the corresponding position in the output feature map. This process is repeated on the entire input feature map until a complete output feature map is generated.

[0097] Assuming the dimension of the input feature map X is H×W, the target output dimension is H'×W', and the adaptive average pooling is expressed as:

[0098]

[0099] Among them, Y(i,j) is the element of the output feature map at position (i,j), R i,j Area(R i,j ) is the region R i,j The number of pixels contained in (h,w) is the area R i,j The pixel coordinates within .

[0100] Through the interactive fusion module, data features of different modalities are integrated, enhancing the richness of feature expression, enabling the model to capture more details and information, and helping the model avoid over-reliance on the features of a single data source.

[0101] 5. Forgetting Prevention Module

[0102] The embodiment of the present invention designs a forgetting prevention module, which pre-assigns a set of hypothetical prototypes in the embedding space and regards them as virtual categories. Then, interpolation is performed between the feature representations of different data samples to create new virtual mixed features to generate virtual samples.

[0103] The process is as follows: Randomly select two samples x1 and x2 and their corresponding labels y1 and y2 in the training set, input the two samples into the neural network respectively, and propagate forward to the selected hidden layer. Suppose the outputs of the selected hidden layer for x1 and x2 are z1 and z2 respectively. Interpolate the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ) z2, where λ is an interpolation coefficient that controls the degree of interpolation. The corresponding labels are also interpolated accordingly: y = λy1 + (1-λ) y2. The new representation z obtained by interpolation continues to be forward propagated through the rest of the neural network to obtain the final output, that is, the virtual sample.

[0104] In order to reserve space for new categories, a virtual loss is constructed:

[0105]

[0106] in, is the virtual category label, y' is the pseudo label in the existing category, L img Represents the new category space retained by the Mask function, L mi and L mc Used to avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed sample. The Mask function is defined as:

[0107]

[0108] in, Represents the Hadamard product (element-wise multiplication), which is used to multiply the feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the current known category, the other categories are 0, and the current category is 1. Through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1. In this way, the components corresponding to the known categories in the feature vector are cleared, while the components that may belong to the unknown categories are retained. The objective function is finally defined as:

[0109] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0110] Among them, L cls represents the basic loss function calculated by cosine similarity, λ1 is a balancing factor used to adjust L vrThe relative importance of , x represents the original input data of the model, and y represents the true category label corresponding to the input x.

[0111] 6. Cross-modal Information Fusion Module

[0112] The combination of the above-mentioned modality completion module, interactive fusion module, and forgetting prevention module constitutes the cross-modal information fusion module of the embodiment of the present invention.

[0113] The working process of the cross-modal information fusion module applied to the invention is as follows:

[0114] First, the masked autoencoder is used to complete the missing cross-modal data and improve the data; secondly, the interactive fusion module is used to perform channel mixing, group convolution, and adaptive average pooling on the completed data to integrate the features of cross-modal data and help the model capture more information; finally, the forgetting prevention module is used to pre-allocate a set of imaginary prototypes regarded as virtual categories in the embedding space, and create new virtual mixed features to generate enhanced data samples. By calculating the value of the designed total loss function, the gradient descent method is applied to update the model parameters until the model performance is no longer significantly improved or the preset number of iterations is reached. Finally, a trained model is obtained.

[0115] 7. Classifier

[0116] The structure of the classifier consists of a fully connected layer, whose output layer has a number of neurons corresponding to the total number of all categories encountered so far (the current stage and all previous stages). As incremental learning progresses, the number of neurons in the output layer will increase accordingly whenever a new category is introduced.

[0117] This design ensures that the classifier can adapt to the increasing number of categories and thus continue to learn and adapt to new data.

[0118] In summary, the embodiments of the present invention realize the completion of missing modal data in practical applications through the above parts, alleviate the problem of absolute dominance of a single modality, improve the model's learning ability for new and old classes, reduce the catastrophic forgetting phenomenon of the model, improve the target detection capability of drones, and greatly improve inspection efficiency.

[0119] Example 3

[0120] Combine the following Figure 2 , Figure 3 , the specific experimental data are used to verify the feasibility of the schemes in Examples 1 and 2, as described below:

[0121] The LasHeR-CIL dataset and VTDV-CIL dataset were used to validate the method.

[0122] The base class learning rate and epoch on the LasHeR-CIL dataset are set to 0.05 and 1000, the new class learning rate and epoch are set to 0.05 and 500, and the number of interactive fusion groups is set to 8. On the VTDV-CIL dataset, the base class learning rate and epoch are set to 0.1 and 1000, the new class learning rate and epoch are set to 0.1 and 300, and the number of interactive fusion groups is 4. The final result is the average result of 5 experiments. For fair comparison, all class incremental learning methods use the method of adding feature maps to achieve cross-modal class incremental learning.

[0123] The experiment in the embodiment of the present invention uses the average accuracy ACC and the performance degradation rate PD as evaluation indicators of class incremental learning. ACC is mainly used to evaluate the overall performance of the model, and is calculated using the average accuracy of all learning stages:

[0124]

[0125] ACC t Indicates the Top-1 accuracy after the tth stage, ACC t The higher the value, the better the prediction accuracy. PD mainly evaluates the degree of forgetting of the model during continuous learning. PD can be calculated by the difference between the accuracy of the first stage and the last stage:

[0126] PD=ACC0-ACC T

[0127] Among them, ACC0 represents the accuracy of the first stage, ACC T Indicates the accuracy of the last stage.

[0128] In addition, considering that different models may have different accuracies in the first stage, which may affect the evaluation of PD, a new indicator to measure the degree of forgetting is proposed, called the normalized performance degradation rate NPD, as follows:

[0129]

[0130] Both PD and NPD are indicators used to assess the degree of forgetting. The lower the value, the less forgetting.

[0131] The experimental results are compared with the following five baseline methods: DER, FOSTER, MEMO, PODNet, and MRFA+PDF. For all methods, five experiments are conducted and the final result is the average result of the five experiments.

[0132] Table 1 shows the accuracy, performance degradation rate and normalized performance degradation rate of different new categories in each incremental stage on the LasHeR-CIL dataset, which comprehensively reflects the performance of the model provided by the embodiment of the present invention. The embodiment of the present invention takes into account the problems of incomplete data, single modality dominance and forgetting that may exist in cross-modal incremental learning, and completes and fuses the data, thereby improving the model performance.

[0133] Table 1 Incremental learning results on the LasHeR-CIL dataset (accuracy, performance degradation rate, normalized performance degradation rate)

[0134]

[0135]

[0136] In addition, the embodiment of the present invention verifies the effectiveness of its own components, as shown in Table 2. Completion indicates data completion, Visible indicates visible light mode, and Thermal indicates infrared mode. The experimental results reflect the effectiveness and necessity of the design of each component of the model.

[0137] Table 2 Ablation experiment results on LasHeR-CIL dataset

[0138]

[0139] Example 4

[0140] A drone inspection device based on incremental learning of missing cross-modal data class, the device comprising: a processor 1 and a memory 2, the memory 2 storing program instructions, the processor 1 calling the program instructions stored in the memory 2 to enable the device to execute the following method steps in embodiment 1:

[0141] Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each mode hidden in the data; mix the feature channels of different modes in the input feature map;

[0142] A set of convolution kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map.

[0143] The output feature map is flattened and then input into the fully connected layer. The activation function is applied to introduce nonlinearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using the gradient descent method. After the model training is completed, it is deployed on the drone, which recognizes the scene and entities.

[0144] Among them, the feature channels of different modes in the input feature map are mixed specifically as follows:

[0145] The features from each modality in the input feature map are divided into different channels, the feature channels from each modality are evenly grouped, and channels are selected from each group and mixed with the corresponding group of the other modality.

[0146] Among them, the loss function consists of two parts, the real classification loss L cls and the new class reservation loss L vr ,

[0147] True classification loss L cls Used to measure the difference between the true label and the model's predicted output;

[0148] New class reservation loss L vr Virtual categories and virtual samples are used to pre-occupy places for new categories that have not been learned in the embedding space, and the distribution of the embedding space is adjusted to help the inspection and recognition model in the drone maintain its memory of old categories when learning new categories.

[0149] Among them, the missing data collected by the drone is completed by using a masked autoencoder to complete the missing data collected by the sensor during the training process.

[0150] Among them, channel mixing assigns channels to different predefined groups and recombine feature channels from different modalities. Channel mixing is defined as:

[0151]

[0152] Among them, || represents connection, and Represents feature groups extracted from two different modalities, and feature channels are evenly divided into different groups t represents the tth scene, i represents different modes, N is the number of groups, C is the number of channels, and Indicates that channel mixing operation is performed on feature groups extracted from two different modalities; represents the jth element of the feature channel in the i-th modality in the t-th scene.

[0153] Among them, the virtual sample is:

[0154] Randomly select two samples x1 and x2 and their corresponding labels y1 and y2 in the training set, input the two samples into the neural network respectively, and propagate forward to the selected hidden layer. Suppose the outputs of the selected hidden layer for x1 and x2 are z1 and z2 respectively; interpolate on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient, and interpolate the corresponding labels accordingly: y = λy1 + (1-λ)y2, and continue to propagate the new representation z obtained by interpolation forward through the rest of the neural network to obtain the final output.

[0155] Among them, the virtual loss corresponding to the sample is:

[0156]

[0157] in, is the virtual category label, y' is the pseudo label in the existing category, L img Represents the new category space retained by the Mask function, L mi and L mc Used to avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed sample, and the Mask function is defined as:

[0158]

[0159] in, Represents the Hadamard product, which is used to multiply the feature vector by the element-by-element complement of a one-hot encoding vector. The one-hot encoding vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1. Through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1. The objective function is finally defined as:

[0160] L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y)

[0161] Among them, L cls represents the basic loss function calculated by cosine similarity, λ1 is a balancing factor used to adjust L vr The relative importance of , x represents the original input data of the model, and y represents the true category label corresponding to the input x.

[0162] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0163] The execution subjects of the above-mentioned processor 1 and memory 2 can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0164] The data signal is transmitted between the memory 2 and the processor 1 via the bus 3, which will not be described in detail in the embodiment of the present invention.

[0165] Example 5

[0166] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0167] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0168] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0169] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0170] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0171] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0172] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone inspection method based on incremental learning of missing cross-modal data, characterized in that: The method comprises: Complete the missing data collected by the drone; preprocess the completed data to obtain the features of each mode hidden in the data; mix the feature channels of different modes in the input feature map; A set of convolution kernels is assigned to each group in the mixed input feature map. After group convolution, adaptive average pooling is applied to the feature map to obtain the final output feature map. The output feature map is flattened and then input into the fully connected layer. The activation function is applied to introduce nonlinearity to generate the final classification result. The loss function is then applied to calculate the loss, and the model parameters are updated using the gradient descent method. After the model training is completed, it is deployed on the drone, which recognizes the scene and entities.

2. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 1 is characterized in that: The mixing of feature channels of different modes in the input feature map is specifically as follows: The features from each modality in the input feature map are divided into different channels, the feature channels from each modality are evenly grouped, and channels are selected from each group and mixed with the corresponding group of the other modality.

3. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 1 is characterized in that: The loss function consists of two parts: the true classification loss L cls and the new class reservation loss L vr , The true classification loss L cls Used to measure the difference between the true label and the model's predicted output; The new class reserves loss L vr Virtual categories and virtual samples are used to pre-occupy places for new categories that have not been learned in the embedding space, and the distribution of the embedding space is adjusted to help the inspection and recognition model in the drone maintain its memory of old categories when learning new categories.

4. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 1 is characterized in that: The method of completing the missing data collected by the drone includes: using a masked autoencoder to complete the missing data collected by the sensor during the training process.

5. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 2 is characterized in that: The channel mixing assigns channels to different predefined groups and recombine feature channels from different modalities. The channel mixing is defined as: Among them, || represents connection, and Represents feature groups extracted from two different modalities, and feature channels are evenly divided into different groups t represents the tth scene, i represents different modes, N is the number of groups, C is the number of channels, and Indicates that channel mixing operation is performed on feature groups extracted from two different modalities; represents the jth element of the feature channel in the i-th modality in the t-th scene.

6. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 4 is characterized in that: The virtual sample is: Randomly select two samples x1 and x2 and their corresponding labels y1 and y2 in the training set, input the two samples into the neural network respectively, and propagate forward to the selected hidden layer. Suppose the outputs of the selected hidden layer for x1 and x2 are z1 and z2 respectively; interpolate on the outputs z1 and z2 of the hidden layer to generate a new internal representation z: z = λz1 + (1-λ)z2, where λ is an interpolation coefficient, and interpolate the corresponding labels accordingly: y = λy1 + (1-λ)y2, and continue to propagate the new representation z obtained by interpolation forward through the rest of the neural network to obtain the final output.

7. The method for unmanned aerial vehicle inspection based on incremental learning of missing cross-modal data according to claim 4 is characterized in that: The virtual loss corresponding to the sample is: in, is the virtual category label, y' is the pseudo label in the existing category, L img Represents the new category space retained by the Mask function, L mi and L mc Used to avoid over-compression of the old category space, f i (x) represents the feature representation generated by the network for the input sample x, f i (m) represents the feature representation generated by the network for the mixed sample, and the Mask function is defined as: in, Represents the Hadamard product, which is used to multiply the feature vector by the element-by-element complement of a one-hot encoding vector. The one-hot encoding vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1. Through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1. The objective function is finally defined as: L(x,m,y)=L cls (f i (x),y)+λ1L vr (x,m,y) Among them, L cls represents the basic loss function calculated by cosine similarity, λ1 is a balancing factor used to adjust L vr The relative importance of , x represents the original input data of the model, and y represents the true category label corresponding to the input x.

8. A drone inspection device based on incremental learning of missing cross-modal data, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

Images