Unmanned aerial vehicle inspection method and system based on cross-modal data class incremental learning

Abstract

The invention discloses an unmanned aerial vehicle inspection method and system based on cross-modal data class incremental learning. The method comprises the steps that features of all modals are spliced and recombined through a cross-modal feature mixing and fusing module; a set of independent convolution kernels is distributed to each group of processed input feature maps, grouping convolution is carried out, and finally a complete feature map is generated; for each pooling window in the complete feature map, calculating an average value of all elements in the window according to adaptive average pooling, taking the average value as an element of a corresponding position of the output feature map, and repeating the process on all regions to obtain a final output feature map; flattening an output feature map obtained after processing of the self-adaptive average pooling module, inputting the output feature map into a full-connection layer to extract and combine features, generating final classification output, and updating model parameters according to the gradient of a loss function; and deploying the trained model to an unmanned aerial vehicle to collect visible light and thermal imaging images in real time, extracting features, and performing identification and prediction through the model.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle inspection in intelligent unmanned systems, and in particular to a method and system for unmanned aerial vehicle inspection based on incremental learning of cross-modal data. Background Art

[0002] The drone inspection system mainly uses drones to perform automated monitoring and inspection of facilities or areas. This system is usually used in the maintenance of key infrastructure such as power lines, oil and gas pipelines, railways, bridges, as well as in agriculture, forest monitoring and urban management. The core advantages of drone inspection systems include: improving efficiency, reducing costs and reducing personnel safety risks. Drone inspections require the collection and analysis of input data for automatic decision-making. Visible light images and infrared images are two commonly used data with certain complementarity. During the day, visible light images can provide richer information; at night, due to insufficient lighting conditions, infrared images can provide richer information. Therefore, how to design drone inspection research for cross-modal data has become an urgent problem to be solved.

[0003] The existing drone inspection model collects high-definition images or infrared images captured by high-definition cameras or infrared cameras for processing, trains a deep neural network model based on a cross-modal image dataset, and uses the model to infer and predict the newly collected cross-modal images to determine their category. Although this method can achieve good performance, the drone inspection model cannot be updated once deployed. Generally, the types of inspection targets will gradually change with the development of society, and a large number of new categories will continue to emerge in the form of various modalities. Existing methods are difficult to use in practice because they can only recognize old types of cross-modal targets that have been learned.

[0004] Cross-modal incremental learning is a method that enables a model to learn new tasks based on existing cross-modal tasks, thereby being able to handle new and old cross-modal tasks simultaneously. In cross-modal incremental learning, the model continuously integrates new information from different perceptual modalities to improve its understanding and generalization capabilities. However, due to the large difference between the cross-modal data distribution of new tasks and that of old tasks, deep neural networks may change important parameters related to old tasks when learning new tasks, resulting in a significant decrease in the model's ability to recognize old tasks. This phenomenon is called catastrophic forgetting, where the model's parameters tend to favor new tasks, affecting its ability to recognize and process old tasks.

[0005] To address this problem, existing methods have designed some solutions to solve knowledge forgetting, such as regularization, adding replay mechanisms, dynamic architecture expansion, etc., but they largely focus on maintaining unimodal knowledge. In practical applications, although these strategies help alleviate the problem of unimodal catastrophic forgetting, they usually ignore the unique challenges of cross-modal data. Therefore, how to solve the problem of model forgetting the modality of old tasks in cross-modal incremental learning is a key factor in whether the drone inspection model can be effectively deployed on a large scale. Summary of the invention

[0006] The present invention provides a method and system for UAV inspection based on incremental learning of cross-modal data. The present invention takes into account the problem of modal forgetting in the incremental learning of cross-modal UAV inspection tasks, and designs a method based on incremental learning of cross-modal data. By using the cross-modal feature information exchange mechanism of block diagonal structure sparsity on the channel scale and the construction of virtual categories, the constraint model incrementally introduces new categories without losing previous learning knowledge, and uses enhanced features of known categories to reserve learning space for new cross-modal data, so that the interference between new and old cross-modal tasks is effectively alleviated, thereby improving the cross-modal incremental learning ability of the UAV inspection model, and greatly increasing the robustness and generalization of UAV inspection. See the following description for details:

[0007] A drone inspection method based on cross-modal data incremental learning, the method comprising:

[0008] The features of each modality are spliced ​​and reorganized through the cross-modal feature hybrid fusion module; a set of independent convolution kernels are assigned to each group of processed input feature maps, and group convolution is performed to finally generate a complete feature map;

[0009] For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to the adaptive average pooling, and the average value is used as the element at the corresponding position of the output feature map. This process is repeated for all regions to obtain the final output feature map;

[0010] The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on the drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model.

[0011] The method of combining and recombining the features of each modality through the cross-modal feature hybrid fusion module is as follows:

[0012] The features from each modality in the feature map are divided into multiple groups, each group contains a certain number of channels, and a part of the channels are selected from a specific group of one modality and spliced ​​and reorganized with the channels of the corresponding group in another modality.

[0013] The loss function includes: the real label calculation classification loss L cls and unknown new class reservation loss L ukn ,

[0014] The actual label is used to calculate the classification loss L cls Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data;

[0015] Unknown new class reservation loss L ukn A portion of the learning space is reserved in the model's output space. During training, the input data passes through the model to calculate the predicted output for each category.

[0016] The two losses are weighted together to form the total loss.

[0017] Among them, the cross-modal feature hybrid fusion module is:

[0018] In the batch normalization layer, the features of each channel are standardized and adjusted by scaling and translation. A larger scaling factor γ value indicates that the channel has higher activity and importance in the model learning process, while a scaling factor γ value close to zero indicates that the channel information is redundant and has little impact on the output.

[0019] The input features of the visible light and infrared light modalities are subdivided into multiple predefined groups by channel and specially processed. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one modality and combine these channels with the channels of the corresponding group in the other modality. During the training process, the channel reorganization strategy is automatically adjusted according to the data characteristics and task requirements.

[0020] Among them, the cross-modal feature hybrid fusion module is defined as:

[0021]

[0022]

[0023] Among them, || represents connection, and represents the feature set extracted from two different modalities, and The features contain each other.

[0024] Wherein, the loss function is:

[0025] Loss function L cls The model learns known categories from labels, and the loss function L ukn Reserve learning space for unknown new categories to make the learned known categories more compact. The objective function is defined as:

[0026]

[0027] in, is the enhanced embedding obtained from the cross-modal feature hybrid fusion module, y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient to balance the two losses;

[0028] The mask function is defined as:

[0029]

[0030] Among them, the Mask function is applied to the feature vector This feature vector represents the embedding representation obtained from incremental learning. and Represent the embedding of two modalities respectively. Indicates that the Hadamard product is used to multiply the feature vector by the element-by-element complement of a one-hot encoded vector; the one-hot encoded vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1; through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1.

[0031] In the second aspect, a drone inspection system based on incremental learning of cross-modal data classes is provided, the system comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0032] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0033] The beneficial effects of the technical solution provided by the present invention are:

[0034] 1. The present invention simultaneously considers cross-modal balanced modeling and the correlation between new and old categories, further improves the learning ability of cross-modal new and old categories, alleviates the modal forgetting phenomenon of cross-modal models, improves the incremental learning ability of cross-modal data, greatly increases the robustness and generalization of drone inspections, and improves the recognition accuracy of drone inspections;

[0035] 2. This paper proposes a new incremental learning method for cross-modal data in the cross-modal data recognition task in the UAV inspection scene. This method improves the modal forgetting problem of incremental learning of the UAV inspection model through a strategy combining feature grouping fusion and learning space reservation. It can help the UAV to keep the memory of the old categories while accepting new category information, reducing the need for retraining;

[0036] 3. The present invention designs a module that combines hybrid fusion with learning space reservation, mines cross-modal knowledge according to the interaction of channel feature grouping, and makes the representation of known categories more compact by constructing virtual classes, reserving learning space for unknown new categories;

[0037] 4. The present invention can effectively integrate data from different sensors by optimizing feature fusion and information exchange mechanisms, and improve the perception ability of drones in complex environments, which is especially important in environments with insufficient light or obstructed vision;

[0038] 5. The present invention allows drones to predict and adapt to unknown new categories without touching actual data through pre-configured virtual spaces; this ability to quickly adapt to new environments can significantly improve the operational efficiency and response speed of drones, especially when responding to emergencies or performing rapid deployments;

[0039] 6. The present invention effectively solves the forgetting problem in the model learning process, reduces the computing resource consumption caused by frequent model updates and the amount of data that needs to be manually labeled, improves the adaptability of the model, and helps to reduce long-term operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a drone inspection method based on incremental learning of cross-modal data;

[0041] Figure 2 Schematic diagram of the cross-modal incremental learning task;

[0042] Figure 3 This is a schematic diagram of the structure of LasHeR-CIL, a cross-modal dataset for drone inspections;

[0043] Figure 4 This is a schematic diagram of the structure of a UAV inspection cross-modal dataset VTDV-CIL;

[0044] Figure 5 A schematic diagram for comparison;

[0045] Among them, this figure is applied to the LasHeR-CIL dataset. Compared with other advanced methods, the accuracy of the hybrid fusion forgetting prevention strategy has been significantly improved after multiple stages.

[0046] Figure 6 A schematic diagram for comparison;

[0047] Among them, this figure is applied to the VTDV-CIL dataset. Compared with other advanced methods, the accuracy of the hybrid fusion forgetting prevention strategy has been significantly improved after multiple stages.

[0048] Figure 7 This is a structural diagram of a drone inspection system based on incremental learning of cross-modal data.

[0049] Table 1 compares the accuracy, forgetting rate, and normalized forgetting rate of different methods on the LasHeR-CIL dataset and the VTDV-CIL dataset.

[0050] Table 2 shows the ablation experiments conducted on the LasHeR-CIL dataset and the VTDV-CIL dataset, demonstrating the effectiveness of each component of the method. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0052] Example 1

[0053] Since the drone encounters new categories during the inspection process, it uses cross-modal data to continuously perceive the surrounding environment, thereby continuously updating the model carried in the drone. In order to solve the problems existing in the background technology, incremental learning of new categories in drone inspection tasks will lead to serious degradation of the recognition ability of old categories. The embodiment of the present invention proposes a drone inspection method based on incremental learning of cross-modal data, which incrementally learns new cross-modal categories in target recognition while maintaining the recognition ability of previous old categories, and realizes cross-modal incremental learning through effective model update, see Figure 1 , the method comprises the following steps:

[0054] 101: UAVs acquire images and transmit cross-modal data during inspections;

[0055] Start the drone and perform the inspection task according to the predetermined flight path and plan. During the inspection process, the drone collects images and data of different modes in real time through its own sensors (HD camera, infrared camera, laser radar, etc.). During the flight, the raw data collected by the drone will be transmitted to the ground control center or data processing platform in real time.

[0056] 102: Preprocess the cross-modal data collected by the drone to obtain the features of each preprocessed mode;

[0057] Convolution, pooling and other methods are used to extract features from the cross-modal data collected by the drone, compress the data volume of the original image, and obtain the feature map; the number of convolution kernels in the convolution layer determines the number of channels in the feature map, and each channel is a response to a certain specific feature of the input image.

[0058] 103: Divide the features of each modality into multiple groups, select a part of the channels from a specific group of one modality, and splice and reorganize them with the channels of the corresponding group in another modality;

[0059] That is, this step divides the features from each modality in the feature map into multiple groups, each of which contains a certain number of channels. The next step is channel mixing, which selects a part of the channels from a specific group of one modality and splices and recombines them with the channels of the corresponding group in the other modality. Through this channel exchange, the features of different modalities are mixed.

[0060] Among them, the cross-modal feature hybrid fusion module combines the information of different modalities by exchanging fusion channel features between different modalities, and mines the correlation relationship between different modalities to better maintain robustness.

[0061] 104: Assign a set of independent convolution kernels to each group of processed input feature maps, perform group convolution, and finally generate a complete feature map;

[0062] After channel exchange and feature mixing, the processed input feature map is divided into multiple groups, each containing a certain number of channels. For each group, a set of independent convolution kernels are assigned, and these convolution kernels only perform convolution operations within their designated groups. The convolution kernel slides within its corresponding feature map group. For each position, the convolution kernel is element-wise multiplied with the corresponding local area of ​​the feature map, and the results are added to form a single output value. This process is repeated at each possible position of each convolution kernel to generate a complete feature map and output it.

[0063] The cross-fused feature maps are input into the grouped convolutional layer, which divides the input feature maps into multiple groups, and performs convolution operations on each group independently, which can reduce parameters while maintaining the network's expressiveness.

[0064] 105: For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to the adaptive average pooling, and the average value is used as the element at the corresponding position of the output feature map. The process is repeated for all regions to obtain the final output feature map;

[0065] Determine the size of the input feature map after the grouped convolution and the size of the required output feature map, and then automatically calculate the size and step size of the pooling window based on the input and output sizes. For each pooling window in the input feature map, calculate the average value of all elements in the window and use the average value as the element at the corresponding position in the output feature map. Repeat this process for all regions until the entire input feature map is covered, and obtain the final output feature map.

[0066] Among them, adaptive average pooling converts the input feature map into an output of a fixed size, enhancing the model's invariance to different input changes, especially to translation and size changes; the spatial resolution of the features is reduced through the pooling operation, which can reduce the risk of overfitting of the model to a certain extent.

[0067] 106: Flatten the output feature map obtained after processing by the adaptive average pooling module and input it into the fully connected layer to further extract and combine features to generate the final classification output, then apply the loss function to calculate the loss, and update the model parameters according to the gradient of the loss function;

[0068] The embodiment of the present invention designs two loss functions: the real label calculation classification loss L cls and unknown new class reservation loss L ukn , L cls Mainly responsible for the correct classification of currently known categories, used to measure the difference between the model output and the true label, to ensure that the model can accurately identify and classify the categories contained in the training data; L ukn A portion of learning space is reserved in the output space of the model so that when new categories that the model has not encountered appear, it can have enough "elasticity" to learn these new categories without excessively interfering with the learned categories. During training, the input data passes through the model to calculate the predicted output of each category; two losses are calculated based on the predicted output and the true label, and the two losses are weighted and combined to form a total loss; the gradient of the model parameters is calculated based on the total loss, and the model parameters are updated using the gradient descent method to minimize the total loss; in each iteration, the weights and biases of the model are updated according to the calculated gradient until the maximum number of iterations is reached or the model performance is no longer significantly improved.

[0069] Among them, two loss functions L are used cls and L ukn Respectively ensure that the model can effectively learn known categories from labels and reserve learning space for unknown new categories.

[0070] 107: Deploy the trained model to the drone. During the inspection process, the drone collects visible light and thermal imaging images in real time, extracts features, and uses the model for identification and prediction.

[0071] When training the model, the two loss functions mentioned above are used to jointly optimize the model parameters. The real label calculation classification loss function ensures that the model can recognize known categories, and the unknown new category reservation loss function reserves space for new categories in the future. After the model training is completed, the trained model is deployed on the drone. During the inspection process, the drone collects visible light and thermal imaging images in real time, extracts features, and uses the model for identification and prediction. With the addition of new environments and equipment for drone inspections, the model can use the newly collected data for incremental learning, continuously update the model, learn new knowledge while retaining old knowledge.

[0072] In summary, the embodiment of the present invention realizes the simultaneous consideration of cross-modal balanced modeling and the correlation between new and old categories through the above steps 101 to 107, further improves the learning ability of cross-modal new and old categories, reduces the modal forgetting phenomenon of the model, improves the drone's ability to recognize targets of different categories, and greatly improves the efficiency of drone inspections.

[0073] Example 2

[0074] Combine the following Figure 1 , calculation formulas, and tables are used to further introduce the scheme in Example 1, as described below:

[0075] 1. Model Framework

[0076] After preprocessing the cross-modal data collected by the drone sensor, the data is input into the model for training. The model framework consists of a cross-modal feature hybrid fusion module, a grouped convolutional layer, an adaptive average pooling module, and a learning space reservation module. Figure 1 The following two components are the core of the method:

[0077] (1) Cross-modal feature hybrid fusion module: During the training process, a cross-modal feature hybrid fusion strategy is used to perform feature grouping and information exchange between different modalities, so that the model can integrate and utilize information from different modalities and better maintain the robustness of the model.

[0078] (2) Learning space reservation module: Since it is difficult to solve the modal forgetting in cross-modal incremental learning by only using a cross-modal feature hybrid fusion strategy, an embodiment of the present invention designs a learning space reservation module based on cross-modal feature hybrid fusion, so that the model considers the balance of new and old class knowledge across modalities.

[0079] 2. Dataset Introduction

[0080] 1. LasHeR-CIL dataset

[0081] The LasHeR dataset contains 1,224 pairs of spatially aligned visible and thermal infrared video pairs, totaling more than 730K frame pairs, each of which has been manually annotated with bounding boxes. Using these bounding boxes, samples of different categories were cropped to create a dataset called LasHeR-CIL, which includes a total of 18 categories and 2,324 samples. The dataset has extremely high diversity in capturing object categories, camera viewpoints, scene complexity, and environmental factors (such as seasons, weather, day and night, etc.). In LasHeR-CIL, all categories are divided into three groups: 6 / 1, 6 / 2, and 6 / 3, where the first number represents the number of categories in the base stage, and the second number represents the number of new categories added in each incremental stage.

[0082] 2. VTDV-CIL dataset

[0083] Among them, VTUAV includes 500 visible-thermal infrared sequences, a total of 1.7 million high-resolution (1920x 1080 pixel) frame pairs. 10 categories were extracted from VTUAV using bounding boxes, and 5 categories were collected from different scenes such as urban roads and residential areas using the ZenmuseXT 2 camera equipped with the DJI M200 aircraft to construct a cross-modal CIL dataset containing 6,429 samples. In VTDV-CIL, all categories are also divided into three groups: 3 / 1, 3 / 2, and 3 / 3. Similarly, the first number indicates the number of categories in the basic stage, and the second number indicates the number of new categories added in each incremental stage.

[0084] 3. Cross-modal feature hybrid fusion module

[0085] The embodiment of the present invention designs a cross-modal feature hybrid fusion module. The fusion module adopts a carefully designed channel allocation strategy before the grouped convolution layer, and evaluates the importance of each channel through the scaling factor in the batch normalization layer, so as to effectively organize and schedule input features from different sensors.

[0086] In the batch normalization layer, the features of each channel are first standardized and then adjusted by scaling (represented by the parameter γ) and translation (represented by the parameter β). The scaling factor γ directly adjusts the amplitude of the channel features, affecting the activation of the channel in the network. Larger values ​​of γ indicate that the channel has higher activity and importance in the model learning process, while values ​​close to zero indicate that the channel information is redundant and has little impact on the output. This allows the model to dynamically identify and prioritize channels that are rich in information, while suppressing or replacing channels that contribute less.

[0087] Through this channel allocation strategy, the module subdivides the input features of visible and infrared light modalities into multiple predefined groups by channel and processes them specifically. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one modality, and then combines these channels with the channels of the corresponding group in the other modality to improve the accuracy and efficiency of detection and monitoring.

[0088] In addition, the design of the fusion module also supports automatic adjustment of channel reorganization strategies according to data characteristics and task requirements during training. This adaptive mechanism enables the module to continuously optimize its performance and maintain efficient and accurate processing capabilities when facing dynamically changing data streams, such as real-time monitoring or complex environment inspections. In this way, the fusion module can effectively integrate visible light and infrared light modal information, greatly improving the performance of drones in image recognition, target tracking or other complex scenarios.

[0089] Mathematically, the cross-modal feature hybrid fusion module is defined as:

[0090]

[0091]

[0092] Among them, || represents connection, and Represents a feature group extracted from two different modalities. Through the feature fusion module, data from different modalities are fused. and The features of the models include each other, alleviating the problem of unbalanced modal fusion.

[0093] 4. Grouped Convolution

[0094] Before the convolution operation, the input feature map is divided into multiple groups, each containing a specific number of channels. Each group is assigned a dedicated convolution kernel, which performs convolution operations only within their respective groups. The convolution kernel slides over the corresponding feature map group, performs a dot product operation on the feature map at each position and the convolution kernel, and then adds up these products to produce an output value. This process is repeated at each position covered by the convolution kernel to construct a complete output feature map.

[0095] Assume that the input feature map X has C in channels, grouped convolution divides these channels into G groups, then each group has channels, and for each group, there is an independent convolution kernel W, and each convolution kernel operates only on the channels of its corresponding group. The output feature map Y is also divided into G groups, each group has channels. Given X is H×W×C indimensional input feature map, K is dimensional convolution kernel, where m×n is the spatial dimension of the convolution kernel. Y is H'×W'×C out The process of grouped convolution can be expressed as follows: for each position (i, j) in the output feature map, the following calculation is performed:

[0096]

[0097] Among them, (i, j) represents the position in the output feature map, (u, v) represents the spatial index inside the convolution kernel, g is the group index, and X g and Y g represents the g-th group of input and output graphs, K g is the convolution kernel of the g-th group.

[0098] 5. Adaptive Average Pooling

[0099] The embodiment of the present invention uses adaptive average pooling technology to automatically adjust the spatial dimension of the feature map so that the size of the output is independent of the size of the input. Adaptive average pooling works by calculating the average value of each region in the input feature map, and the size of the region is automatically adjusted according to the desired output dimension.

[0100] Assuming the dimension of the input feature map X is H×W, the target output dimension is H'×W', and the adaptive average pooling is expressed as:

[0101]

[0102] Among them, Y(i,j) is the element of the output feature map at position (i,j), R i,j Area(R i,j ) is the region R i,j The number of pixels contained in (h,w) is the area R i,j The pixel coordinates within .

[0103] 6. Classifier

[0104] The classifier consists of a fully connected network with an output layer whose number of neurons is equal to the total number of all categories encountered so far (including the current stage and all previous stages). As incremental learning proceeds, the addition of new categories will lead to an increase in the number of corresponding output layer neurons.

[0105] 7. Learning Space Reserve Module

[0106] The embodiment of the present invention constructs two loss functions L cls and L ukn To prevent modality forgetting during training. clsIt aims to enable the model to learn known categories well from labels, L ukn Reserve learning space for unknown new categories, while making the learned known categories more compact. The objective function is defined as:

[0107]

[0108] in, is the enhanced embedding obtained from the cross-modal feature hybrid fusion module, y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient to balance the two losses.

[0109] The mask function is defined as:

[0110]

[0111] Among them, the Mask function is applied to the feature vector This feature vector represents the embedding representation obtained from incremental learning. and Represent the embedding of two modalities respectively. Represents the Hadamard product (element-wise multiplication), which is used to multiply the feature vector by the element-wise complement of a one-hot encoded vector. The one-hot encoded vector OneHot(y) represents the label of the current known category, the other categories are 0, and the current category is 1. Through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1. In this way, the components corresponding to the known categories in the feature vector are cleared, while the components that may belong to the unknown categories are retained.

[0112] In summary, the embodiments of the present invention alleviate the problem of a single modality dominating in incremental learning of cross-modal classes through the above-mentioned parts, improve the model's learning ability for new and old classes across modalities, reduce the model's modal forgetting phenomenon, improve the ability to perform target recognition during drone inspections, and greatly improve the efficiency of drone inspections.

[0113] Example 3

[0114] Combine the following Figure 2 , Figure 3 , the specific experimental data are used to verify the feasibility of the schemes in Examples 1 and 2, as described below:

[0115] The LasHeR-CIL dataset and VTDV-CIL dataset were used to validate the method.

[0116] For the LasHeR-CIL dataset, the base class learning rate and period are set to 0.005 and 400, the new class learning rate and period are set to 0.1 and 100, and the number of channel mixing groups is set to 32. On VTDV-CIL, the base class learning rate and period are set to 0.005 and 400, the new class learning rate and period are set to 0.1 and 100, and the number of channel mixing groups is set to 8. For fair comparison, all class incremental learning methods use the method of adding feature maps to achieve cross-modal class incremental learning.

[0117] The experiment in the embodiment of the present invention uses the average accuracy ACC and the forgetting rate PD as evaluation indicators of class incremental learning. ACC is mainly used to evaluate the overall performance of the model, and is calculated using the average accuracy of all learning stages:

[0118]

[0119] Among them, ACC t Indicates the Top-1 accuracy after the tth stage, ACC t The higher the value, the better the prediction accuracy. PD mainly evaluates the degree of forgetting of the model during continuous learning. PD can be calculated by the difference between the accuracy of the first stage and the last stage:

[0120] PD=ACC 0 -ACC T

[0121] Among them, ACC 0 Indicates the accuracy of the first stage, ACC T Indicates the accuracy of the last stage.

[0122] In addition, considering that different models may have different accuracies in the first stage, which may affect the evaluation of PD, a new metric to measure the degree of forgetting is proposed, called normalized forgetting rate NPD, as follows:

[0123]

[0124] Among them, PD and NPD are indicators used to evaluate the degree of forgetting. The lower the value, the lower the degree of forgetting.

[0125] The experimental results are compared with seven existing methods, including: CEC, DER, FOSTER, FACT, MEMO, PODNet and WA. For all methods, 5 experiments are run and the average results are adopted.

[0126] Table 1 shows the accuracy, forgetting rate and normalized forgetting rate of different new categories in each incremental stage on the LasHeR-CIL dataset and the VTDV-CIL dataset, which comprehensively reflects the performance of the model provided by the embodiment of the present invention. The embodiment of the present invention takes into account the single modality dominance problem and modality forgetting problem that may exist in cross-modality incremental learning, and performs cross-modal hybrid fusion processing on the data, thereby improving the model performance.

[0127] Table 1 Incremental learning results on two datasets (accuracy, forgetting rate, normalized forgetting rate)

[0128]

[0129]

[0130] In addition, the embodiment of the present invention verifies the effectiveness of its own modality, as shown in Table 2.

[0131] Table 2 Ablation experiment results on two datasets

[0132]

[0133] Among them, Visible represents the visible light modality, Thermal represents the infrared modality, FP represents the forgetting prevention strategy by reserving learning space, and HF represents the cross-modal feature hybrid fusion module.

[0134] Example 4

[0135] A drone inspection system based on incremental learning of cross-modal data, see Figure 7 The system includes: a processor 1 and a memory 2, wherein the memory 2 stores program instructions, and the processor 1 calls the program instructions stored in the memory 2 to enable the device to execute the following method steps in Example 1:

[0136] The features of each modality are spliced ​​and reorganized through the cross-modal feature hybrid fusion module; a set of independent convolution kernels are assigned to each group of processed input feature maps, and group convolution is performed to finally generate a complete feature map;

[0137] For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to the adaptive average pooling, and the average value is used as the element at the corresponding position of the output feature map. This process is repeated for all regions to obtain the final output feature map;

[0138] The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on the drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model.

[0139] Among them, the features of each modality are spliced ​​and reorganized through the cross-modal feature hybrid fusion module as follows:

[0140] The features from each modality in the feature map are divided into multiple groups, each group contains a certain number of channels, and a part of the channels are selected from a specific group of one modality and spliced ​​and reorganized with the channels of the corresponding group in another modality.

[0141] Among them, the loss function includes: the real label calculation classification loss L cls and unknown new class reservation loss L ukn ,

[0142] The actual label is used to calculate the classification loss L cls Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data;

[0143] Unknown new class reservation loss L ukn A portion of the learning space is reserved in the model's output space. During training, the input data passes through the model to calculate the predicted output for each category.

[0144] The two losses are weighted together to form the total loss.

[0145] Among them, the cross-modal feature hybrid fusion module is:

[0146] In the batch normalization layer, the features of each channel are standardized and adjusted by scaling and translation. A larger scaling factor γ value indicates that the channel has higher activity and importance in the model learning process, while a scaling factor γ value close to zero indicates that the channel information is redundant and has little impact on the output.

[0147] The input features of the visible light and infrared light modalities are subdivided into multiple predefined groups by channel and specially processed. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one modality and combine these channels with the channels of the corresponding group in the other modality. During the training process, the channel reorganization strategy is automatically adjusted according to the data characteristics and task requirements.

[0148] Among them, the cross-modal feature hybrid fusion module is defined as:

[0149]

[0150]

[0151] Among them, || represents connection, and represents the feature set extracted from two different modalities, and The features contain each other.

[0152] Among them, the loss function is:

[0153] Loss function L cls The model learns known categories from labels, and the loss function L ukn Reserve learning space for unknown new categories to make the learned known categories more compact. The objective function is defined as:

[0154]

[0155] in, is the enhanced embedding obtained from the cross-modal feature hybrid fusion module, y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient to balance the two losses;

[0156] The mask function is defined as:

[0157]

[0158] Among them, the Mask function is applied to the feature vector This feature vector represents the embedding representation obtained from incremental learning. and Represent the embedding of two modalities respectively. Indicates that the Hadamard product is used to multiply the feature vector by the element-by-element complement of a one-hot encoded vector; the one-hot encoded vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1; through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1.

[0159] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0160] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0161] Data signals are transmitted between the memory and the processor via bus 3, which will not be described in detail in the embodiment of the present invention.

[0162] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0163] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0164] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0165] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0166] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0167] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0168] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A drone inspection method based on incremental learning of cross-modal data, characterized in that: The method comprises: The features of each modality are spliced ​​and reorganized through the cross-modal feature hybrid fusion module; a set of independent convolution kernels are assigned to each group of processed input feature maps, and group convolution is performed to finally generate a complete feature map; For each pooling window in the complete feature map, the average value of all elements in the window is calculated according to the adaptive average pooling, and the average value is used as the element at the corresponding position of the output feature map. This process is repeated for all regions to obtain the final output feature map; The output feature map obtained after processing by the adaptive average pooling module is flattened and input into the fully connected layer to extract and combine features to generate the final classification output. Then, the model parameters are updated according to the gradient of the loss function. The trained model is deployed on the drone to collect visible light and thermal imaging images in real time, extract features, and perform recognition and prediction through the model.

2. The method for unmanned aerial vehicle inspection based on incremental learning of cross-modal data according to claim 1 is characterized in that: The cross-modal feature hybrid fusion module is used to combine and reorganize the features of each modality as follows: The features from each modality in the feature map are divided into multiple groups, each group contains a certain number of channels, and a part of the channels are selected from a specific group of one modality and spliced ​​and reorganized with the channels of the corresponding group in another modality.

3. The method for unmanned aerial vehicle inspection based on incremental learning of cross-modal data according to claim 1 is characterized in that: The loss function includes: the real label calculation classification loss L cls and unknown new class reservation loss L ukn , The actual label is used to calculate the classification loss L cls Used to correctly classify currently known categories, measure the difference between the model output and the true label, and ensure that the model can accurately identify and classify the categories contained in the training data; Unknown new class reservation loss L ukn A portion of the learning space is reserved in the model's output space. During training, the input data passes through the model to calculate the predicted output for each category. The two losses are weighted together to form the total loss.

4. The method for unmanned aerial vehicle inspection based on incremental learning of cross-modal data according to claim 1 is characterized in that: The cross-modal feature hybrid fusion module is: In the batch normalization layer, the features of each channel are standardized and adjusted by scaling and translation. A larger scaling factor γ value indicates that the channel has higher activity and importance in the model learning process, while a scaling factor γ value close to zero indicates that the channel information is redundant and has little impact on the output. The input features of the visible light and infrared light modalities are subdivided into multiple predefined groups by channel and specially processed. In the hybrid fusion stage, the model uses a dynamic selection mechanism to select the most important or representative channels from a specific group of one modality and combine these channels with the channels of the corresponding group in the other modality. During the training process, the channel reorganization strategy is automatically adjusted according to the data characteristics and task requirements.

5. The method for unmanned aerial vehicle inspection based on incremental learning of cross-modal data according to claim 4 is characterized in that: The cross-modal feature hybrid fusion module is defined as: Among them, || represents connection, and represents the feature set extracted from two different modalities, and The features contain each other.

6. The method for unmanned aerial vehicle inspection based on incremental learning of cross-modal data according to claim 3 is characterized in that: The loss function is: Loss function L cls The model learns known categories from labels, and the loss function L ukn Reserve learning space for unknown new categories to make the learned known categories more compact. The objective function is defined as: in, is the enhanced embedding obtained from the cross-modal feature hybrid fusion module, y is the label, y′ is the pseudo label in the virtual category, and λ is the coefficient to balance the two losses; The mask function is defined as: Among them, the Mask function is applied to the feature vector This feature vector represents the embedding representation obtained from incremental learning. and Represent the embedding of two modalities, Indicates that the Hadamard product is used to multiply the feature vector by the element-by-element complement of a one-hot encoded vector; the one-hot encoded vector OneHot(y) represents the label of the current known category, the positions of the other categories are 0, and the current category position is 1; through the 1-OneHot(y) operation, the positions of the known categories are set to 0 and the positions of the unknown categories are set to 1.

7. A drone inspection system based on incremental learning of cross-modal data, the system comprising: A processor and a memory, wherein the memory stores program instructions, and the processor calls the program instructions stored in the memory to enable the device to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 6.

Images