Multi-modal data fusion method and device for electric power inspection unmanned aerial vehicle, and medium

Abstract

The invention discloses a multi-modal data fusion method and device for an electric power inspection unmanned aerial vehicle, and a medium. Comprising the steps that visible light, thermal infrared, radar point cloud, IMU / GNSS and other multi-modal data collected by an unmanned aerial vehicle are acquired; taking a multi-modal large model as a teacher model and a lightweight network as a student model, and performing unsupervised distillation training by utilizing InfoNCE comparison loss; and deploying the trained student model at an airborne edge end of the unmanned aerial vehicle, extracting multi-scale features of each modal in real time, mapping the multi-scale features to a public potential space, generating a refined token through learnable importance scoring and sparse sampling, and outputting fusion features through linear self-attention and gating expert routing fusion.

Description

Technical Field

[0001] This invention belongs to the field of intelligent inspection technology for unmanned aerial vehicles (UAVs), and particularly relates to a multimodal data fusion method, electronic equipment, and medium for power line inspection UAVs. Background Technology

[0002] With the expansion of power grid scale and the improvement of intelligence level, traditional manual inspection methods can no longer meet the needs of efficient, safe, and all-weather operation and maintenance. Power inspection drones (UAVs) are shifting from "single visible light inspection" to "multimodal collaborative perception". Equipped with various types of sensors such as visible light RGB, thermal infrared TIR, depth / LiDAR, and airborne IMU / GNSS attitude and trajectory information, they have become the core means of power inspection.

[0003] In recent years, significant progress has been made in the development of multimodal large-scale models based on deep learning on the server side. By fusing multi-source data such as RGB, thermal infrared (TIR), depth / LiDAR, and IMU / GNSS, the detection and segmentation accuracy of targets such as towers, insulators, fittings, and conductors can be significantly improved. However, the field inspection environment is complex and variable, presenting the following challenges: 1. Field operations suffer from modal loss or degradation: such as thermal infrared overexposure, LiDAR penetration failure, IMU mutation, etc., resulting in information loss.

[0004] 2. Drastic domain shift: such as changes in light at dawn and dusk, specular reflection from high altitudes, and shadow interference.

[0005] 3. The target scale is large and structural details are easily lost, such as small targets at long distances, such as insulators and conductor clamps. Summary of the Invention

[0006] To address the shortcomings of existing technologies, embodiments of the present invention provide a multimodal data fusion method, device, and medium for power line inspection drones.

[0007] In a first aspect, embodiments of the present invention provide a multimodal feature fusion method for power line inspection drones, the method comprising the following steps: Acquire multimodal data collected by the UAV during power line inspection, including visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose; Construct and train a feature fusion model; the training process of the feature fusion model includes: using a multimodal large model as a teacher model and the feature fusion model as a student model, and performing unsupervised training of the teacher-student model using InfoNCE contrastive loss; Multimodal data is input into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; performing importance scoring and sparse sampling on the latent feature representation for each modality to obtain a refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gated expert routing to obtain the fused features. The trained feature fusion model is deployed on the drone to perform power line inspection tasks.

[0008] Secondly, embodiments of the present invention provide a multimodal large-scale feature fusion system for power line inspection drones. The system is used to implement the aforementioned multimodal data fusion method for power line inspection drones, and the system includes: The multi-module data acquisition module is used to acquire multimodal data collected by the UAV during power line inspection. The multimodal data includes visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose. The feature fusion model training module constructs and trains the feature fusion model. The training process of the feature fusion model includes: using the multimodal large model as the teacher model and the feature fusion model as the student model, and performing unsupervised training of the teacher-student model using InfoNCE contrastive loss. The feature fusion processing module inputs multimodal data into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; performing importance scoring and sparse sampling on the latent feature representation for each modality to obtain the refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gating expert routing to obtain the fused features.

[0009] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described multimodal data fusion method for power line inspection drones.

[0010] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described multimodal data fusion method for power line inspection drones.

[0011] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned multimodal data fusion method for power line inspection drones.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a multimodal feature fusion method for power line inspection drones. The method uses a large multimodal model as the teacher model and a feature fusion model as the student model. Unsupervised distillation training is performed using InfoNCE contrastive loss, enabling consistent and aligned semantic representations of multimodal data, including visible light, thermal infrared, depth radar point clouds, lidar point clouds, and IMU / GNSS pose data, within a common latent space. Compared to existing methods that rely solely on single-modal features or simple concatenation, this invention significantly improves the consistency and discriminative power among multimodal features, providing a more robust feature foundation for subsequent tasks such as defect detection and target recognition. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart of a feature fusion method for power line inspection drones provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0017] like Figure 1 As shown in the figure, this embodiment of the invention provides a multimodal data fusion method for power line inspection drones, the method comprising the following steps: Step S1: Acquire multimodal data collected by the UAV during power line inspection. The multimodal data includes visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose.

[0018] Specifically, let the original observation of each mode in the t-th frame be... , where m represents the modality index and M represents the set of all modalities.

[0019] Specifically, step S1 further includes preprocessing the multimodal data to obtain registration data, denoted as... ,include: The visible light camera, thermal infrared camera and lidar are jointly calibrated to obtain the intrinsic and extrinsic parameters of each mode; Time synchronization and trajectory smoothing are performed based on IMU and GNSS timestamps, and visible light images, thermal infrared images, depth radar point clouds, and lidar point clouds are uniformly registered to the same coordinate system. Among these steps, voxel downsampling and ground point removal are performed on the lidar point cloud, and radiometric calibration is performed on the thermal infrared image to obtain spatiotemporally aligned and preprocessed multimodal input data.

[0020] Step S2: Construct and train the feature fusion model; wherein, the training process of the feature fusion model includes: using the multimodal large model as the teacher model and the feature fusion model as the student model, and performing unsupervised training of the teacher-student model through InfoNCE contrastive loss.

[0021] In this example, to take edge deployment into account, the feature fusion model sets a corresponding lightweight feature encoder for each modality. The lightweight feature encoder adopts a backbone network composed of MobileConv, RepVGG and / or lightweight Transformer modules, and supports NPU acceleration operators.

[0022] For example, for the RGB modality, the RepVGG-A0 structure is used, combined with the SE attention module, to output three-scale feature maps; for the TIR modality, MobileNetV3-small is used, introducing channel attention and Hard-Swish activation; for the LiDAR modality, a lightweight version of PointPillars is used to output BEV pseudo-image features; and for the IMU modality, a 1D-CNN + GRU structure is used to extract pose temporal features.

[0023] Furthermore, the training process of the feature fusion model specifically includes: The multimodal large model is used as the teacher model and deployed on the server side; the feature fusion model is used as the student model. Construct a joint loss function, which includes at least classification Focal loss, bounding box regression GIoU loss, InfoNCE contrast loss, temperature distillation loss, sparse regularization term for the number of refined tokens, and mask reconstruction loss; The InfoNCE contrastive loss constructs positive sample pairs using different modal representations of the same spatiotemporal segment, and negative sample pairs using modal representations of different spatiotemporal segments or different inspection task segments. The InfoNCE contrastive learning loss aims to shorten the distance between positive sample pairs and widen the distance between negative sample pairs; the expression is as follows: In the formula, This indicates the number of samples within a batch. Let represent the embedding vector of the token corresponding to the i-th sample in the first mode a. Let represent the embedding vector of the token corresponding to the i-th sample in the second modality b. The embedding vector of the token corresponding to the j-th sample in the second modality b. This indicates the temperature hyperparameter.

[0024] The temperature distillation loss includes: applying temperature distillation to the task logits and L2 alignment to intermediate features, as expressed below: In the formula, This indicates the distillation temperature, and KL(.) represents the KL divergence. This represents the logits (unnormalized class scores) output by the teacher model. This represents the ogits output by the student model; σ (⋅ / T This indicates that a soft label is obtained based on the Softmax value of the distillation temperature. The weights of the feature alignment loss are represented (empirical values ​​of 0.5 to 2). This represents the token matrix of the teacher model in the common latent space. Let represent the token matrix of the student model in the common latent space.

[0025] It should be noted that the distillation loss consists of logits distillation and feature distillation. Logits distillation ensures that the soft labels output by the student model are as consistent as possible with those of the teacher model, thus learning the knowledge from the teacher model. Feature distillation directly "aligns" the intermediate representations, giving the student model a representation space similar to that of the teacher model, improving generalization and robustness. Therefore, the distillation loss first uses soft labels to allow the student model to imitate the predictions of the teacher model, and then uses L2 feature loss to align the intermediate layers, achieving a dual distillation of "output + representation".

[0026] Specifically, for inspection tasks (detection / segmentation / key points / alarms), a combined supervised loss is constructed using the classification Focal loss and regression GloU loss of the detection head, expressed as follows: In the formula, and Indicates weight To encourage sparsity and efficiency, a token sparsity regularization is introduced. .

[0027] In the formula, This represents the importance probability vector of the token selector output for modality m (each token corresponds to a weight or probability of 0–1). This represents the L1 norm, which is the sum of the absolute values ​​of all elements.

[0028] It should be noted that the token sparsity regularization loss uses L1 regularization to force the importance of most tokens to approach 0, thereby encouraging sparse selection and reducing the number of tokens actually involved in the computation.

[0029] To improve the robustness of mode reconstruction, a masked auto-encoder loss can be added. .

[0030] The expression for the joint loss function is as follows: In the formula, , , , This represents the weighting coefficient.

[0031] It should be noted that this invention significantly reduces the number of model parameters and computational complexity through lightweight network design, refined token mechanism, and sparse regularization constraints, enabling the feature fusion model to be deployed on embedded CPUs / GPUs / NPUs on the UAV side. Under typical power line inspection flight conditions, the method of this invention can improve single-frame inference speed and overall frame rate while ensuring the accuracy of target detection and equipment defect identification, achieving near real-time or real-time multimodal perception and alarm, and adapting to complex environments such as high towers, long corridors, strong light, and smog.

[0032] Step S3 involves inputting multimodal data into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; assigning importance scores and performing sparse sampling on the latent feature representation for each modality to obtain the refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gating expert routing to obtain the fused features.

[0033] Specifically, step S3 includes the following sub-steps: Step S301: Extract features from the multimodal data to obtain multi-scale features. ,in, Represents the spatial height of the feature map of mode m. This represents the spatial width of the feature map of mode m. This represents the number of channels in the feature map of mode m.

[0034] Step S302, map the multi-scale features By accessing the public latent space, we obtain the latent feature representation of each modality. ;in, This represents the two-dimensional token matrix corresponding to mode m in frame t; The flattening function, consisting of 1×1 convolutions, flattening, or linear layers, is used to transform a three-dimensional feature map into a two-dimensional "token" matrix. This represents the number of tokens obtained after flattening modality m. d represents the feature dimension of each token, i.e., the dimension of the common latent space.

[0035] Step S303: The importance score and sparse sampling of the latent feature representation of each modality are performed to obtain the refined token corresponding to each modality.

[0036] Furthermore, to reduce latency and storage, a learnable token selector is configured for each modality. The learnable token selector includes a multilayer perceptron (MLP) and a sigmoid activation function, used to output the importance scores of all tokens corresponding to each modality. .

[0037] A differentiable sparse sampling strategy based on Gumbel-Softmax is adopted. Given a temperature parameter, the importance scores are randomly perturbed and normalized. The top-K tokens with the highest scores are selected as the refined tokens for this mode. The expression is as follows: In the formula, Indicates the first The unnormalized importance score (logit) of each candidate token; Gumbel(0,1) noise (which can be represented by...) Uniform(0,1) is sampled and taken get); This represents the temperature parameter and controls the softmax function. The degree of "discrete / continuous"; Indicates the first The sampling probability of each candidate token; p represents the sampling probability vector obtained by Gumbel-Softmax, that is, the sampling probability corresponding to all candidate tokens; Representing modes The number of Top-K selections (token retention budget).

[0038] Step S304: Concatenate the refined tokens corresponding to each mode, as shown in the following expression: In the formula, This indicates the refined token corresponding to the RGB modality. This indicates the refined token corresponding to the TIR mode. This represents the refined token corresponding to the depth radar point cloud mode. This represents the refinement token corresponding to the IMU pose mode.

[0039] Step S305: The concatenated refined tokens are processed by a self-attention mechanism and lightweight gating expert routing (Mixture-of-Experts) to obtain the fused features.

[0040] Specifically, this example concatenates refined tokens from different modalities to form a unified token sequence; a multi-head self-attention mechanism is used to model this token sequence to capture long-range dependencies between different modalities, spatial locations, and temporal segments, thereby obtaining a preliminary fused feature representation.

[0041] Based on the self-attention output, several expert subnetworks are set up. Each expert can focus on feature modeling of a specific modality, a specific scale, or a specific task subspace. The weights of each expert are calculated on the input token through a gating network, and only a few experts with high scores are activated. Their outputs are then weighted and combined to obtain the final fused features.

[0042] It should be noted that this example, through its gated hybrid expert structure, can adaptively adjust the dependence on visible light, thermal infrared, point cloud, and pose modalities in different scenarios, thereby improving the expressive power of fused features and the computational efficiency of the model. This invention performs self-attention modeling at the refined token level and introduces a lightweight gated hybrid expert structure to perform expert routing and weighted combination of tokens for different modalities and scenarios. On the one hand, it utilizes a self-attention mechanism to capture long-range dependencies between space, time, and modality; on the other hand, it dynamically selects the sub-network most suitable for the current scenario through gated experts, achieving refined modeling and on-demand computation of multimodal information. Compared to fixed-structure fusion networks, this invention can adaptively adjust the dependence on visible light, thermal infrared, and point cloud modalities in complex power scenarios, improving the ability to identify abnormal targets and hidden defects.

[0043] Step S4: Deploy the trained feature fusion model on the UAV terminal to perform power line inspection tasks.

[0044] Furthermore, the trained feature fusion model is deployed on an embedded computing platform on the drone's edge, such as an ARM architecture processor, embedded GPU, or NPU.

[0045] To ensure real-time performance and energy consumption control at the edge, the following can be further implemented: Based on the target frame rate and chip computing power, set the maximum number of tokens and the maximum number of expert activations, and impose parameter constraints on the token refining module and the gating expert routing module. Based on the current quality of multimodal data (e.g., image sharpness, point cloud density, GNSS signal quality, etc.), the token retention ratio of different modes is dynamically adjusted. When the quality of a certain mode decreases or is missing, its weight in the fusion process is reduced. Optionally, the feature fusion model can be quantized and pruned to reduce the number of model parameters and computational complexity without significantly reducing recognition accuracy.

[0046] It should be noted that this invention achieves more accurate identification and defect analysis of targets such as transmission lines and tower equipment through multi-source information fusion and end-side intelligent processing. This can reduce the frequency of manual inspections and reduce the exposure time of staff in dangerous environments such as high altitudes and high voltages. At the same time, the high detection rate and low false alarm rate brought about by multi-modal fusion help improve the automation level and decision-making efficiency of power operation and maintenance, thereby reducing long-term operation and maintenance costs and improving the safety and reliability of power grid operation.

[0047] On the other hand, embodiments of the present invention provide a multimodal large-scale feature fusion system for power line inspection drones. The system is used to implement the aforementioned multimodal data fusion method for power line inspection drones, and the system includes: The multi-module data acquisition module is used to acquire multimodal data collected by the UAV during power line inspection. The multimodal data includes visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose. The feature fusion model training module constructs and trains the feature fusion model. The training process of the feature fusion model includes: using the multimodal large model as the teacher model and the feature fusion model as the student model, and performing unsupervised training of the teacher-student model using InfoNCE contrastive loss. The feature fusion processing module inputs multimodal data into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; performing importance scoring and sparse sampling on the latent feature representation for each modality to obtain the refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gating expert routing to obtain the fused features.

[0048] In summary, this invention provides a multimodal feature fusion method for power line inspection drones. This invention uses a large multimodal model as the teacher model and the feature fusion model as the student model. Unsupervised distillation training is performed using InfoNCE contrastive loss, enabling consistent and aligned semantic representations of multimodal data such as visible light, thermal infrared, depth radar point clouds, lidar point clouds, and IMU / GNSS pose data within a common latent space. Compared to existing methods that rely solely on single-modal features or simple concatenation, this invention significantly improves the consistency and discriminative power among multimodal features, providing a more robust feature foundation for subsequent tasks such as defect detection and target recognition.

[0049] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0050] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0051] This specification also provides a computer-readable storage medium storing a computer program that can be used to perform the above-described data synchronization method.

[0052] This instruction manual also provides Figure 2 The diagram shows a schematic structural representation of the electronic device. Figure 2 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to achieve the aforementioned data synchronization method.

[0053] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0054] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0055] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0056] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0057] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0058] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0069] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A multimodal feature fusion method for power line inspection drones, characterized in that, The method includes the following steps: Acquire multimodal data collected by the UAV during power line inspection, including visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose; Construct and train a feature fusion model; the training process of the feature fusion model includes: using a multimodal large model as a teacher model and the feature fusion model as a student model, and performing unsupervised training of the teacher-student model using InfoNCE contrastive loss; Multimodal data is input into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; performing importance scoring and sparse sampling on the latent feature representation for each modality to obtain a refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gated expert routing to obtain the fused features. The trained feature fusion model is deployed on the drone to perform power line inspection tasks.

2. The multimodal data fusion method for power line inspection drones according to claim 1, characterized in that, After acquiring the multimodal data collected by the drone during power line inspection, the process also includes: The visible light camera, thermal infrared camera and lidar are jointly calibrated to obtain the intrinsic and extrinsic parameters of each mode; Time synchronization and trajectory smoothing are performed based on IMU and GNSS timestamps, and visible light images, thermal infrared images, depth radar point clouds, and lidar point clouds are uniformly registered to the same coordinate system. Among these steps, voxel downsampling and ground point removal are performed on the lidar point cloud, and radiometric calibration is performed on the thermal infrared image to obtain spatiotemporally aligned and preprocessed multimodal input data.

3. The multimodal data fusion method for power line inspection drones according to claim 1, characterized in that, The feature fusion model sets up a corresponding lightweight feature encoder for each modality. The lightweight feature encoder adopts a backbone network composed of MobileConv, RepVGG and / or lightweight Transformer modules.

4. The multimodal data fusion method for power line inspection drones according to claim 1, characterized in that, The process of mapping multi-scale features to a common latent space to obtain the latent feature representation of each modality includes: The multi-scale features of each modality are dimensionally aligned and normalized through a linear transformation layer and / or a multilayer perceptron, so that the features of each modality are unified into a common latent space of a preset dimension. The features of different scales in the same spatiotemporal segment are spliced ​​or weighted to form the latent feature representation of the modality in the common latent space.

5. A multimodal data fusion method for power line inspection drones according to claim 1, characterized in that, The process of scoring the importance of the latent feature representations of each modality and performing sparse sampling to obtain the refined tokens corresponding to each modality includes: Configure a learnable token selector for each modality to assign cloud computing importance scores for each location or point in the modality’s latent features; A differentiable sparse sampling strategy based on Gumbel-Softmax is adopted. Under a given temperature parameter, the importance scores are randomly perturbed and normalized, and the top-K tokens with the highest scores are selected as the refined tokens for this mode.

6. A multimodal data fusion method for power line inspection drones according to claim 1, characterized in that, The training process of the feature fusion model specifically includes: The multimodal large model is used as the teacher model, and the feature fusion model is used as the student model. Construct a joint loss function, which includes at least classification Focal loss, bounding box regression GIoU loss, InfoNCE contrast loss, temperature distillation loss, sparse regularization term for the number of refined tokens, and mask reconstruction loss; The InfoNCE contrastive loss constructs positive sample pairs using different modal representations of the same spatiotemporal segment and negative sample pairs using modal representations of different spatiotemporal segments or different inspection task segments. The InfoNCE contrastive learning loss narrows the distance between positive sample pairs and widens the distance between negative sample pairs.

7. A multimodal large-scale feature fusion system for power line inspection drones, characterized in that, The system is used to implement the multimodal data fusion method for power line inspection drones as described in any one of claims 1-6, and the system includes: The multi-module data acquisition module is used to acquire multimodal data collected by the UAV during power line inspection. The multimodal data includes visible light images, thermal infrared images, depth radar point clouds, lidar point clouds, IMU pose and / or GNSS pose. The feature fusion model training module constructs and trains the feature fusion model. The training process of the feature fusion model includes: using the multimodal large model as the teacher model and the feature fusion model as the student model, and performing unsupervised training of the teacher-student model using InfoNCE contrastive loss. The feature fusion processing module inputs multimodal data into a trained feature fusion model to obtain fused features. This includes: extracting features from the multimodal data to obtain multi-scale features; mapping the multi-scale features to a common latent space to obtain the latent feature representation for each modality; performing importance scoring and sparse sampling on the latent feature representation for each modality to obtain the refined token corresponding to each modality; concatenating the refined tokens corresponding to each modality; and processing the concatenated refined tokens through a self-attention mechanism and gating expert routing to obtain the fused features.

8. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multimodal data fusion method for power line inspection UAVs as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multimodal data fusion method for power line inspection drones as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the multimodal data fusion method for power line inspection drones as described in any one of claims 1-6.

Images