Unmanned aerial vehicle inspection method and system based on cross-modal data class incremental learning

Abstract

The application discloses a kind of based on cross-modal data class increment learning's unmanned plane inspection method and system, comprising: by cross-modal feature mixed fusion module, the features of each mode are spliced and reorganized;Each group of processed input feature map is assigned a set of independent convolution kernel, and grouping convolution is carried out, and finally complete feature map is generated;For each pooling window in complete feature map, the average value of all elements in the window is calculated according to adaptive average pooling, the average value is used as the element of the corresponding position of the output feature map, the process is repeated for all regions, and the final output feature map is obtained;The output feature map obtained after being processed by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features, generate the final classification output, then update the model parameters according to the gradient of the loss function;The trained model is deployed on the unmanned plane to collect visible light and thermal imaging images in real time, extract features and identify and predict through the model.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection in intelligent unmanned systems, and particularly to a UAV inspection method and system based on cross-modal data class incremental learning. Background Technology

[0002] Unmanned aerial vehicle (UAV) inspection systems primarily utilize drones for automated monitoring and inspection of facilities or areas. These systems are commonly used for the maintenance of critical infrastructure such as power lines, oil and gas pipelines, railways, and bridges, as well as in agriculture, forestry monitoring, and urban management. The core advantages of UAV inspection systems include increased efficiency, reduced costs, and reduced safety risks to personnel. UAV inspection requires the collection and analysis of input data for automated decision-making. Visible light and infrared images are two commonly used and complementary types of data. During the day, visible light images provide richer information; at night, due to insufficient lighting, infrared images provide richer information. Therefore, designing UAV inspection systems oriented towards cross-modal data has become an urgent problem to be solved.

[0003] Existing drone inspection models process high-definition or infrared images captured by high-definition or infrared cameras, train deep neural network models on cross-modal image datasets, and use the models to infer and predict the category of newly collected cross-modal images. While this method achieves good performance, drone inspection models cannot be updated once deployed. Typically, the types of inspection targets change gradually with societal development, with numerous new categories constantly emerging in various modalities. Existing methods, because they can only identify learned, older types of cross-modal targets, are difficult to use in practice.

[0004] Cross-modal incremental learning is a method that enables models to learn new tasks based on existing cross-modal tasks, thus possessing the ability to handle both new and old cross-modal tasks simultaneously. In cross-modal incremental learning, the model continuously integrates new information from different perceptual modalities to improve its understanding and generalization abilities. However, because the cross-modal data distribution of new tasks differs significantly from that of old tasks, deep neural networks may alter important parameters related to old tasks when learning new tasks, leading to a significant decline in the model's ability to recognize old tasks. This phenomenon is known as catastrophic forgetting, where the model's parameters become biased towards new tasks, thereby affecting its ability to recognize and process old tasks.

[0005] To address this problem, existing methods have devised several approaches to mitigate knowledge forgetting, such as regularization, replay mechanisms, and dynamic architecture expansion. However, these methods largely focus on preserving unimodal knowledge. In practical applications, while these strategies help alleviate the problem of catastrophic forgetting of unimodal data, they typically ignore the unique challenges of cross-modal data. Therefore, how to address the problem of model forgetting of old tasks in cross-modal incremental learning is a key factor in whether UAV inspection models can be effectively deployed on a large scale. Summary of the Invention

[0006] This invention provides a UAV inspection method and system based on cross-modal data class incremental learning. Considering the problem of modal forgetting in cross-modal incremental learning for UAV inspection tasks, this invention designs a method based on cross-modal data class incremental learning. By using a cross-modal feature information exchange mechanism based on the block-diagonal structure sparsity at the channel scale and the construction of virtual categories, the model is constrained to incrementally introduce new categories without losing previously learned knowledge. Enhanced features of known categories are used to reserve learning space for new cross-modal data, effectively mitigating interference between new and old cross-modal tasks. This improves the cross-modal incremental learning capability of the UAV inspection model and significantly increases the robustness and generalization of UAV inspection. See the description below for details:

[0007] A UAV inspection method based on incremental learning of cross-modal data classes, the method comprising:

[0008] The cross-modal feature fusion module concatenates and recombines features from various modalities; each set of processed input feature maps is assigned an independent convolution kernel, and group convolution is performed to finally generate a complete feature map;

[0009] For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to adaptive average pooling. This average value is used as the element at the corresponding position in the output feature map. This process is repeated for all regions to obtain the final output feature map.

[0010] The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on a drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model.

[0011] Specifically, the step of splicing and recombining the features of various modalities through the cross-modal feature hybrid fusion module involves:

[0012] The features from each modality in the feature map are divided into multiple groups, each containing a certain number of channels. A portion of the channels from a specific group of one modality are selected and concatenated with the channels from the corresponding group of another modality.

[0013] The loss function includes: calculating the classification loss L based on the true labels. cls And the unknown new category reserved loss L ukn ,

[0014] Calculate the classification loss L using the real labels cls Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data;

[0015] Unknown new category reserved loss L ukn A portion of the learning space is reserved in the output space of the model. During training, the input data is passed through the model to calculate the predicted output for each category.

[0016] The two types of losses are weighted and combined to form the total loss.

[0017] The cross-modal feature fusion module is as follows:

[0018] In the batch normalization layer, the features of each channel are standardized and adjusted by scaling and translation. A larger scaling factor γ value indicates that the channel has high activity and importance in the model learning process, while a scaling factor γ value close to zero indicates that the channel has redundant information and has little impact on the output.

[0019] The input features of visible light and infrared light modes are subdivided into multiple predefined groups according to channels and processed specifically. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one mode and combine these channels with the corresponding channels in another mode. During training, the channel recombination strategy is automatically adjusted according to data features and task requirements.

[0020] The cross-modal feature fusion module is defined as follows:

[0021]

[0022]

[0023] Where || represents connection, and This represents feature sets extracted from two different modalities. and Their features are mutually inclusive.

[0024] The loss function is:

[0025] Loss function L cls The model learns known categories from labels, and the loss function L is used. ukn To reserve learning space for unknown new categories and make the already learned known category representations more compact, the objective function is defined as:

[0026]

[0027] in, It is the enhanced embedding obtained from the cross-modal feature fusion module, where y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient that balances the two losses;

[0028] The mask function is defined as:

[0029]

[0030] The Mask function is applied to the feature vector. This feature vector represents the embedding representation obtained from incremental learning. and These represent embeddings of two different modalities. The Hadamard product is used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class, with the other class positions set to 0 and the current class position set to 1. The operation 1-OneHot(y) sets the known class position to 0 and the unknown class position to 1.

[0031] In a second aspect, a drone inspection system based on cross-modal data class incremental learning, the system comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to execute the method described in any one of the first aspects.

[0032] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0033] The beneficial effects of the technical solution provided by this invention are:

[0034] 1. This invention simultaneously considers cross-modal balanced modeling and the correlation between new and old categories, further improving the learning ability of cross-modal new and old categories, reducing the modal forgetting phenomenon of cross-modal models, improving the incremental learning ability of cross-modal data, significantly increasing the robustness and generalization of UAV inspection, and improving the recognition accuracy of UAV inspection.

[0035] 2. This invention proposes a novel cross-modal data class incremental learning method for cross-modal data recognition tasks in UAV inspection scenarios. This method improves the modal forgetting problem in incremental learning of UAV inspection models by combining feature grouping fusion and learning space reservation. It can help UAVs retain the memory of old categories while accepting new category information, reducing the need for retraining.

[0036] 3. This invention designs a module that combines hybrid fusion with learning space reservation. It mines cross-modal knowledge by grouping and interacting according to channel features, and makes the representation of known categories more compact by constructing virtual classes, reserving learning space for unknown new categories.

[0037] 4. By optimizing the feature fusion and information exchange mechanism, this invention can effectively integrate data from different sensors, thereby improving the perception capabilities of UAVs in complex environments. This is especially important in environments with insufficient light or obstructed visibility.

[0038] 5. This invention allows drones to predict and adapt to unknown new categories without accessing actual data through a pre-configured virtual space; this ability to quickly adapt to new environments can significantly improve the operational efficiency and response speed of drones, especially when dealing with emergencies or making rapid deployments.

[0039] 6. This invention effectively solves the forgetting problem in the model learning process, reduces the consumption of computing resources caused by frequent model updates and the amount of data that needs to be manually labeled, improves the adaptability of the model, and helps to reduce long-term operating costs. Attached Figure Description

[0040] Figure 1 A flowchart of a drone inspection method based on incremental learning of cross-modal data classes;

[0041] Figure 2 A schematic diagram of a cross-modal incremental learning task;

[0042] Figure 3 This is a schematic diagram of the structure of the LasHeR-CIL cross-modal dataset for UAV inspection;

[0043] Figure 4 This is a schematic diagram of the structure of the VTDV-CIL cross-modal dataset for UAV inspection;

[0044] Figure 5 For comparison purposes;

[0045] The figure was applied to the LasHeR-CIL dataset. Compared with other state-of-the-art methods, the hybrid fusion forgetting prevention strategy showed a significant improvement in accuracy after multiple stages.

[0046] Figure 6 For comparison purposes;

[0047] The figure was applied to the VTDV-CIL dataset. Compared with other state-of-the-art methods, the hybrid fusion forgetting prevention strategy showed a significant improvement in accuracy after multiple stages.

[0048] Figure 7 This is a schematic diagram of the structure of a drone inspection system based on incremental learning of cross-modal data classes.

[0049] Table 1 compares the accuracy, forgetting rate, and normalized forgetting rate of different methods on the LasHeR-CIL and VTDV-CIL datasets.

[0050] Table 2 shows the ablation experiments conducted on the LasHeR-CIL and VTDV-CIL datasets, demonstrating the effectiveness of each component of this method. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0052] Example 1

[0053] Because drones encounter new categories during inspections, they continuously perceive the surrounding environment using cross-modal data, thereby constantly updating the onboard model. To address the problems in the background technology, incremental learning of new categories in drone inspection tasks leads to a severe degradation of the recognition ability of old categories. This invention proposes a drone inspection method based on incremental learning of cross-modal data categories. This method incrementally learns new cross-modal categories in target recognition while maintaining the recognition ability of previous categories. Cross-modal incremental learning is achieved through effective model updates. (See [link to relevant documentation]). Figure 1 The method includes the following steps:

[0054] 101: The drone acquires images and transmits cross-modal data during the inspection process;

[0055] The drone is launched and performs its inspection mission according to the predetermined flight path and plan. During the inspection, the drone uses its onboard sensors (high-definition cameras, infrared cameras, lidar, etc.) to collect images and data in different modalities in real time. The raw data collected by the drone during flight is transmitted in real time to the ground control center or data processing platform.

[0056] 102: Preprocess the cross-modal data collected by the UAV to obtain the features of each preprocessed mode;

[0057] Convolution and pooling techniques are used to extract features from cross-modal data collected by UAVs, compress the amount of data in the original image, and obtain feature maps. The number of convolution kernels in the convolutional layer determines the number of channels in the feature map, and each channel is a response to a specific feature of the input image.

[0058] 103: Divide the features of each modality into multiple groups, select a portion of channels from a specific group of one modality, and splice and recombine them with the channels of the corresponding group in another modality;

[0059] This step involves dividing the features from various modalities in the feature map into multiple groups, each containing a certain number of channels. The next step is channel blending, where a subset of channels from a specific group of one modality is selected and concatenated with channels from the corresponding group of another modality. Through this channel exchange, features from different modalities are blended.

[0060] Among them, the cross-modal feature hybrid fusion module combines information from different modalities by exchanging fusion channel features between different modalities, and explores the correlation between different modalities to better maintain robustness.

[0061] 104: Assign an independent set of convolution kernels to each group of processed input feature maps, perform group convolution, and finally generate a complete feature map;

[0062] After channel swapping and feature mixing, the processed input feature map is divided into multiple groups, each containing a certain number of channels. An independent set of convolutional kernels is assigned to each group, and these kernels perform convolution operations only within their assigned group. The convolutional kernels slide within their corresponding feature map group. At each location, the convolutional kernel performs element-wise multiplication with the corresponding local region of the feature map, and then the results are summed to form a single output value. This process is repeated at every possible location of each convolutional kernel to generate a complete feature map and output it.

[0063] In this process, the cross-fused feature maps are input into grouped convolutional layers. These layers divide the input feature maps into multiple groups, and each group undergoes convolution operations independently. This reduces the number of parameters while maintaining the network's expressive power.

[0064] 105: For each pooling window in the complete feature map, calculate the average value of all elements in the window according to adaptive average pooling, and use this average value as the element at the corresponding position in the output feature map. Repeat this process for all regions to obtain the final output feature map.

[0065] The dimensions of the input feature map after grouped convolution and the desired dimensions of the output feature map are determined. Then, the size and stride of the pooling window are automatically calculated based on the input and output dimensions. For each pooling window in the input feature map, the average value of all elements within the window is calculated, and this average value is used as the element at the corresponding position in the output feature map. This process is repeated for all regions until the entire input feature map is covered, resulting in the final output feature map.

[0066] Among them, adaptive average pooling transforms the input feature map into a fixed-size output, enhancing the model's invariance to different input changes, especially to translation and size changes; the pooling operation reduces the spatial resolution of features, which can reduce the risk of overfitting to some extent.

[0067] 106: The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to further extract and combine features to generate the final classification output. Then, the loss function is applied to calculate the loss, and the model parameters are updated according to the gradient of the loss function.

[0068] This invention proposes two loss functions: a classification loss L calculated from the true labels. cls And the unknown new category reserved loss L ukn L cls Its primary responsibility is to correctly classify the currently known categories, measuring the difference between the model's output and the true labels, and ensuring that the model can accurately identify and classify the categories contained in the training data; L ukn A portion of the model's output space is reserved for learning, allowing the model sufficient "elasticity" to learn new categories without excessively interfering with already learned categories when they appear. During training, input data is passed through the model, and the predicted output for each category is calculated. Two losses are calculated based on the predicted output and the true label, and these two losses are weighted and combined to form the total loss. The gradient of the model parameters is calculated based on the total loss, and the model parameters are updated using gradient descent to minimize the total loss. In each iteration, the model's weights and biases are updated based on the calculated gradient until the maximum number of iterations is reached or the model performance no longer improves significantly.

[0069] Two loss functions L are used. cls and L ukn Each ensures that the model can effectively learn known categories from the labels, and reserves learning space for unknown new categories.

[0070] 107: Deploy the trained model onto the drone. During the inspection process, the drone collects visible light and thermal imaging images in real time, extracts features, and uses the model for identification and prediction.

[0071] During model training, the two loss functions mentioned above are used to jointly optimize the model parameters. The classification loss function calculated for the true labels ensures the model can recognize known categories, while the reserved loss function for unknown new categories reserves space for future new categories. After model training is complete, the trained model is deployed on a drone. During inspections, the drone acquires visible light and thermal imaging images in real time, extracts features, and uses the model for identification and prediction. As new environments and equipment are added to the drone inspections, the model can use newly acquired data for incremental learning, continuously updating the model and retaining old knowledge while learning new knowledge.

[0072] In summary, the embodiments of the present invention, through steps 101-107, simultaneously consider cross-modal balanced modeling and the correlation between new and old categories, further improving the learning ability of cross-modal new and old categories, reducing the model's modal forgetting phenomenon, improving the UAV's ability to identify targets of different categories, and significantly improving the efficiency of UAV inspection.

[0073] Example 2

[0074] The following is combined Figure 1 The calculation formulas and tables further illustrate the scheme in Example 1, as detailed below:

[0075] I. Model Framework

[0076] After preprocessing the cross-modal data collected by the UAV sensors, the data is input into the model for training. The model framework consists of a cross-modal feature fusion module, grouped convolutional layers, an adaptive average pooling module, and a learning space reservation module. See [link to relevant documentation]. Figure 1 As shown. The following two components form the core of the method:

[0077] (1) Cross-modal feature fusion module: During training, a cross-modal feature fusion strategy is used to group features and exchange information between different modalities, so that the model can integrate and utilize information from different modalities and better maintain the robustness of the model;

[0078] (2) Learning space reservation module: Since it is difficult to solve the modal forgetting in cross-modal incremental learning by using only cross-modal feature hybrid fusion strategy, this embodiment of the invention designs a learning space reservation module based on cross-modal feature hybrid fusion, so that the model considers the balance of new and old cross-modal knowledge.

[0079] II. Introduction to the Dataset

[0080] 1. LasHeR-CIL dataset

[0081] The LasHeR dataset contains 1,224 spatially aligned visible light and thermal infrared video pairs, totaling over 730,000 frames, each with manually annotated bounding boxes. Using these bounding boxes, samples were cropped to create the LasHeR-CIL dataset, comprising 18 classes and 2,324 samples. The dataset exhibits high diversity in terms of captured object categories, camera perspective, scene complexity, and environmental factors (e.g., season, weather, day / night). In LasHeR-CIL, all classes are divided into three groups: 6 / 1, 6 / 2, and 6 / 3. The first number represents the number of classes in the base stage, and the second number represents the number of new classes added in each incremental stage.

[0082] 2. VTDV-CIL dataset

[0083] The VTUAV dataset comprises 500 visible-thermal-infrared sequences, totaling 1.7 million high-resolution (1920x1080 pixels) frame pairs. Ten categories were extracted from VTUAV using bounding boxes, and five categories were collected from different scenes such as urban roads and residential areas using a Zenmuse XT 2 camera mounted on a DJI M200 aircraft to construct a cross-modal CIL dataset containing 6,429 samples. In VTDV-CIL, all categories are similarly divided into three groups: 3 / 1, 3 / 2, and 3 / 3. Similarly, the first number represents the number of categories in the base stage, and the second number represents the number of new categories added in each incremental stage.

[0084] III. Cross-modal feature hybrid fusion module

[0085] This invention presents a cross-modal feature fusion module. This fusion module employs a carefully designed channel allocation strategy before the grouped convolutional layers, using scaling factors in the batch normalization layers to evaluate the importance of each channel, thereby effectively organizing and scheduling input features from different sensors.

[0086] In the batch normalization layer, the features of each channel are first normalized and then adjusted by scaling (represented by parameter γ) and translation (represented by parameter β). The scaling factor γ directly adjusts the magnitude of the channel features, affecting the activation level of that channel in the network. A larger γ value indicates that the channel has higher activity and importance during model learning, while a γ value close to zero indicates that the channel is informationally redundant and has little impact on the output. This allows the model to dynamically identify and prioritize information-rich channels while suppressing or replacing those that contribute less.

[0087] Through this channel allocation strategy, the module subdivides the input features of visible and infrared light modes into multiple predefined groups according to channels and processes them specifically. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one mode, and then combines these channels with the corresponding channels from another mode to improve the accuracy and efficiency of detection and monitoring.

[0088] Furthermore, the fusion module's design supports automatic adjustment of channel recombination strategies based on data characteristics and task requirements during training. This adaptive mechanism enables the module to continuously optimize its performance and maintain efficient and accurate processing capabilities when facing dynamically changing data streams, such as real-time monitoring or inspection of complex environments. In this way, the fusion module can effectively integrate visible light and infrared light modal information, greatly improving the performance of UAVs in image recognition, target tracking, or other complex scenarios.

[0089] In mathematics, a cross-modal feature fusion module is defined as:

[0090]

[0091]

[0092] Where || represents connection, and This represents a set of features extracted from two different modalities. The feature fusion module fuses the data from these different modalities. and The features are mutually inclusive, which alleviates the problem of unbalanced modal fusion.

[0093] IV. Grouped Convolution

[0094] Before the convolution operation, the input feature maps are divided into multiple groups, each containing a specific number of channels. Each group is assigned a dedicated convolution kernel, which performs the convolution operation only within its own group. The convolution kernel slides across the corresponding feature map group, performing a dot product operation between the feature map at each location and the kernel. These multiplications are then summed to produce an output value. This process is repeated at each location covered by the convolution kernel, thereby constructing a complete output feature map.

[0095] Suppose the input feature map X has C in G channels are divided into G groups by grouped convolution, so each group has G channels. There are G channels, and for each group, there is an independent convolutional kernel W. Each convolutional kernel operates only on the channels of its corresponding group. The output feature map Y is also divided into G groups, each group having G channels. There are 1 channel. Given X is H×W×C. inThe input feature map is of dimension K, where K is A convolution kernel of dimension m×n, where m×n is the spatial dimension of the convolution kernel. Y is H'×W'×C. out The output feature map is dimensional. The process of grouped convolution can be represented as follows: for each position (i,j) in the output feature map, the following calculation is performed:

[0096]

[0097] Where (i,j) represents the position in the output feature map, (u,v) represents the spatial index inside the convolution kernel, g is the group index, and X g and Y g K represents the g-th group of the input and output graphs. g It is the convolution kernel of the g-th group.

[0098] V. Adaptive Average Pooling

[0099] This invention employs adaptive average pooling, which automatically adjusts the spatial dimension of the feature map so that the output size is independent of the input size. Adaptive average pooling works by calculating the average value of each region in the input feature map, and the size of the region is automatically adjusted according to the desired output dimension.

[0100] Let the input feature map X have dimensions H×W, and the target output dimension be H'×W'. Adaptive average pooling is represented as:

[0101]

[0102] Where Y(i,j) is the element of the output feature map at position (i,j), and R i,j Area(R) is the region in the input feature map corresponding to the output (i,j). i,j ) is region R i,j The number of pixels contained in region R, where (h,w) is the region R. i,j The pixel coordinates within.

[0103] VI. Classifier

[0104] The classifier consists of a single fully connected network, with the number of neurons in its output layer equal to the total number of all classes encountered so far (including the current stage and all previous stages). As incremental learning progresses, the addition of new classes will lead to an increase in the number of neurons in the corresponding output layer.

[0105] VII. Learning Space Reserved Module

[0106] This invention constructs two loss functions L. cls and L ukn To address the problem of preventing modal forgetting during training. clsThe aim is to enable the model to learn known categories well from the labels, L ukn This allows for the reservation of learning space for unknown new categories, while simultaneously making the representations of already learned known categories more compact. The objective function is defined as:

[0107]

[0108] in, It is the enhanced embedding obtained from the cross-modal feature fusion module, where y is the label, y′ is the pseudo-label in the virtual category, and λ is the coefficient that balances the two losses.

[0109] The mask function is defined as follows:

[0110]

[0111] The Mask function is applied to the feature vector. This feature vector represents the embedding representation obtained from incremental learning. and These represent embeddings of two different modalities. This represents the Hadamard product (element-wise multiplication), used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class; all other class positions are 0, and the current class position is 1. The operation 1-OneHot(y) sets the positions of known classes to 0 and the positions of unknown classes to 1. This clears the components of the feature vector corresponding to the known classes, while retaining the components that might belong to the unknown classes.

[0112] In summary, the embodiments of the present invention alleviate the problem of single modality dominance in cross-modal incremental learning by the above-mentioned parts, improve the model's learning ability across new and old modal classes, reduce the model's modality forgetting phenomenon, improve the ability of UAV inspection to identify targets, and significantly improve the efficiency of UAV inspection.

[0113] Example 3

[0114] The following is combined Figure 2 , Figure 3 The specific experimental data verifies the feasibility of the schemes in Examples 1 and 2, as detailed below:

[0115] The method was validated using the LasHeR-CIL and VTDV-CIL datasets.

[0116] For the LasHeR-CIL dataset, the learning rate and periodicity for the base class were set to 0.005 and 400, respectively, while the learning rate and periodicity for the new class were set to 0.1 and 100, respectively. The number of groups for channel blending was set to 32. For VTDV-CIL, the learning rate and periodicity for the base class were set to 0.005 and 400, respectively, while the learning rate and periodicity for the new class were set to 0.1 and 100, respectively. The number of groups for channel blending was set to 8. For fair comparison, all class incremental learning methods employed an additive feature map approach to achieve cross-modal class incremental learning.

[0117] In this embodiment of the invention, the experiments use average accuracy (ACC) and forgetting rate (PD) as evaluation metrics for incremental learning. ACC is primarily used to evaluate the overall performance of the model and is calculated using the average accuracy across all learning stages.

[0118]

[0119] Among them, ACC t ACC represents the Top-1 accuracy after the t-th stage. t A higher value indicates better prediction accuracy. PD primarily assesses the degree of forgetting in the model during continuous learning. PD can be calculated as the difference between the accuracy of the first stage and the accuracy of the last stage:

[0120] PD = ACC0 - ACC T

[0121] Where ACC0 represents the precision of the first stage, ACC T This indicates the precision of the final stage.

[0122] Furthermore, considering that different models may have varying accuracy in the first stage, which could affect the evaluation of PD, a new metric for measuring the degree of forgetting is proposed, called Normalized Forgetting Rate (NPD), as follows:

[0123]

[0124] PD and NPD are both indicators used to assess the degree of forgetting; the lower the value, the less forgetting.

[0125] The experimental results were compared with seven existing methods: CEC, DER, FOSTER, FACT, MEMO, PODNet, and WA. For all methods, five experiments were run, and the average results were used.

[0126] Table 1 shows the accuracy, forgetting rate, and normalized forgetting rate for different new classes in each incremental stage on the LasHeR-CIL and VTDV-CIL datasets, comprehensively demonstrating the performance of the model provided by the embodiments of the present invention. The embodiments of the present invention take into account the potential single-modality dominance and modality forgetting problems in cross-modal incremental learning, and perform cross-modal hybrid fusion processing on the data, thus improving model performance.

[0127] Table 1 shows the incremental learning results (accuracy, forgetting rate, normalized forgetting rate) on the two datasets.

[0128]

[0129]

[0130] Furthermore, the effectiveness of the self-modal is verified in the embodiments of the present invention, as shown in Table 2.

[0131] Table 2 Ablation experimental results on the two datasets

[0132]

[0133] Wherein, Visible represents the visible light mode, Thermal represents the infrared mode, FP represents the forgetting prevention strategy through reserved learning space, and HF represents the cross-modal feature hybrid fusion module.

[0134] Example 4

[0135] A drone inspection system based on incremental learning of cross-modal data classes, see [link to relevant documentation]. Figure 7 The system includes a processor 1 and a memory 2, wherein the memory 2 stores program instructions, and the processor 1 calls the program instructions stored in the memory 2 to cause the device to perform the following method steps in Embodiment 1:

[0136] The cross-modal feature fusion module concatenates and recombines features from various modalities; each set of processed input feature maps is assigned an independent convolution kernel, and group convolution is performed to finally generate a complete feature map;

[0137] For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to adaptive average pooling. This average value is used as the element at the corresponding position in the output feature map. This process is repeated for all regions to obtain the final output feature map.

[0138] The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on a drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model.

[0139] Specifically, the cross-modal feature fusion module concatenates and recombines features from various modalities as follows:

[0140] The features from each modality in the feature map are divided into multiple groups, each containing a certain number of channels. A portion of the channels from a specific group of one modality are selected and concatenated with the channels from the corresponding group of another modality.

[0141] The loss function includes: the classification loss L calculated from the true labels. cls And the unknown new category reserved loss L ukn ,

[0142] Calculate classification loss L using real labels cls Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data;

[0143] Unknown new category reserved loss L ukn A portion of the learning space is reserved in the output space of the model. During training, the input data is passed through the model to calculate the predicted output for each category.

[0144] The two types of losses are weighted and combined to form the total loss.

[0145] The cross-modal feature fusion module is as follows:

[0146] In the batch normalization layer, the features of each channel are standardized and adjusted by scaling and translation. A larger scaling factor γ value indicates that the channel has high activity and importance in the model learning process, while a scaling factor γ value close to zero indicates that the channel has redundant information and has little impact on the output.

[0147] The input features of visible light and infrared light modes are subdivided into multiple predefined groups according to channels and processed specifically. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one mode and combine these channels with the corresponding channels in another mode. During training, the channel recombination strategy is automatically adjusted according to data features and task requirements.

[0148] The cross-modal feature fusion module is defined as follows:

[0149]

[0150]

[0151] Where || represents connection, and This represents feature sets extracted from two different modalities. and Their features are mutually inclusive.

[0152] The loss function is:

[0153] Loss function L cls The model learns known categories from labels, and the loss function L is used. ukn To reserve learning space for unknown new categories and make the already learned known category representations more compact, the objective function is defined as:

[0154]

[0155] in, It is the enhanced embedding obtained from the cross-modal feature fusion module, where y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient that balances the two losses;

[0156] The mask function is defined as follows:

[0157]

[0158] The Mask function is applied to the feature vector. This feature vector represents the embedding representation obtained from incremental learning. and These represent embeddings of two different modalities. The Hadamard product is used to multiply a feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the currently known class, with the other class positions set to 0 and the current class position set to 1. The operation 1-OneHot(y) sets the known class position to 0 and the unknown class position to 1.

[0159] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0160] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0161] Data signals are transmitted between the memory and the processor via bus 3, which will not be described in detail in this embodiment of the invention.

[0162] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0163] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0164] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0165] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0166] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0167] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0168] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A UAV inspection method based on incremental learning of cross-modal data classes, characterized in that, The method includes: The cross-modal feature fusion module concatenates and recombines features from various modalities; each set of processed input feature maps is assigned an independent convolution kernel, and group convolution is performed to finally generate a complete feature map; For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to adaptive average pooling. This average value is used as the element at the corresponding position in the output feature map. This process is repeated for all regions to obtain the final output feature map. The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on a drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model. The loss function includes: calculating the classification loss based on the true labels. and unforeseen new categories of reserved losses , Calculate classification loss using real labels Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data; Unknown new category reserved loss A portion of the learning space is reserved in the output space of the model. During training, the input data is passed through the model to calculate the predicted output for each category. The two types of losses are weighted and combined to form the total loss; The loss function is: loss function The model learns known categories from labels; loss function. To reserve learning space for unknown new categories and make the already learned known category representations more compact, the objective function is defined as: ； in, It is an enhanced embedding obtained from the cross-modal feature fusion module. It's a tag. These are pseudo-tags within virtual categories. It is a coefficient that balances the two types of losses; The mask function is defined as follows: ； in, Function applied to feature vectors This feature vector represents the embedding representation obtained from incremental learning. and These represent embeddings of two different modalities. The Hadamard product represents the multiplication of an eigenvector with the element-wise complement of a one-hot encoded vector; the one-hot encoded vector... This indicates the label for the currently known category; all other categories are at position 0, and the current category is at position 1. The operation sets the positions of known categories to 0, and the positions of unknown categories to 0. Set the other positions to 1.

2. The UAV inspection method based on cross-modal data class incremental learning according to claim 1, characterized in that, The specific steps of splicing and recombining features from various modalities through the cross-modal feature hybrid fusion module are as follows: The features from each modality in the feature map are divided into multiple groups, each containing a certain number of channels. A portion of the channels from a specific group of one modality are selected and concatenated with the channels from the corresponding group of another modality.

3. The UAV inspection method based on cross-modal data class incremental learning according to claim 1, characterized in that, The cross-modal feature fusion module is as follows: In the batch normalization layer, the features of each channel are normalized and adjusted by scaling and translation, with a larger scaling factor. The value indicates that the channel has high activity and importance during the model learning process, and is a scaling factor close to zero. The value indicates that the channel has redundant information and has little impact on the output; The input features of visible light and infrared light modes are subdivided into multiple predefined groups according to channels and processed specifically. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one mode and combine these channels with the corresponding channels in another mode. During training, the channel recombination strategy is automatically adjusted according to data features and task requirements.

4. The UAV inspection method based on cross-modal data class incremental learning according to claim 3, characterized in that, The cross-modal feature fusion module is defined as follows: ； ； in, Indicates a connection. and This represents feature sets extracted from two different modalities. and Their features are mutually inclusive.

5. A drone inspection system based on incremental learning of cross-modal data classes, the system comprising: A processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the apparatus to perform the method described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-4.

Images