Method and apparatus for determining a fusion representation vector, and electronic device

Abstract

This invention discloses a method, apparatus, and electronic device for determining a fused representation vector. The method includes: acquiring multimodal sensor data; obtaining multimodal embedding vectors based on the multimodal sensor data; inputting the multimodal embedding vectors into a modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space, wherein the modality alignment model is trained on an initial alignment model using sample data and includes a cross-modal contrastive alignment loss function, which maximizes the similarity between different modality embedding vectors at the same acquisition time and minimizes the similarity between different modality embedding vectors at different acquisition times; obtaining a fused representation vector based on the aligned embedding vectors; and performing a target task based on the fused representation vectors. This invention solves the technical problems of poor unified representation of heterogeneous data and poor cross-modal semantic alignment in UAV scenarios for multimodal sensing data.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method, apparatus, and electronic device for determining a fusion representation vector. Background Technology

[0002] As a crucial component of the power distribution network, 10kV overhead lines operate in a complex and variable environment, constantly exposed to external conditions such as wind, rain, icing, and vegetation growth. This makes them prone to hazards such as insulator damage, foreign object entrapment on conductors, and overheating of clamps. Drone inspection, with its advantages of mobility, flexibility, and wide coverage, has become an important tool for line maintenance. Currently, drone inspection technology often utilizes multiple sensors, including visible light cameras, infrared thermal imagers, lidar, acoustic imagers, inertial measurement units, and GPS receivers, to acquire multi-dimensional perception data of the line corridor. However, the data from these multimodal sensors differ significantly in data type, feature distribution, and semantic representation, posing a typical challenge for heterogeneous data fusion.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a method, apparatus, and electronic device for determining fusion representation vectors, in order to at least solve the technical problems of poor unified representation of heterogeneous data and poor cross-modal semantic alignment of multimodal perception data in UAV scenarios.

[0005] According to one aspect of the present invention, a method for determining a fused representation vector is provided, comprising: acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on a UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; performing a multimodal encoding operation based on the multimodal sensor data to obtain a multimodal embedding vector, wherein the multimodal embedding vector includes a visual imaging embedding vector, an acoustic embedding vector, a point cloud embedding vector, an inertial embedding vector, and a position embedding vector; and inputting the multimodal embedding vector into a modality alignment algorithm. The model obtains aligned embedding vectors in the semantic coreference embedding space. The modal alignment model is trained on an initial alignment model using sample data and includes a cross-modal contrastive alignment loss function. This loss function maximizes the similarity between different modal embedding vectors at the same acquisition time and minimizes the similarity between different modal embedding vectors at different acquisition times. Based on the aligned embedding vectors, a feature fusion operation is performed to obtain a fused representation vector. Based on the fused representation vector, a target task is executed, wherein the target task includes at least one of the following: constructing a route space topology map, or determining UAV flight command tasks.

[0006] Optionally, based on the aligned embedding vector, a feature fusion operation is performed to obtain a fused representation vector, including: when the target task is a fine-grained perception task, acquiring multiple feature sequences, and performing interactive fusion between the multiple feature sequences through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector. The multiple feature sequences include visual feature sequences, point cloud feature sequences, and acoustic feature sequences. The visual feature sequences include spatial grid feature sequences obtained after feature extraction from the visual imaging data. The point cloud feature sequences include local geometric feature sequences of key points in the spatial depth point cloud data. The acoustic feature sequences include those obtained after time-frequency transformation of the acoustic signal data. The time-frequency block feature sequence, wherein the fine-grained perception task includes at least one of the following: wire tracking task, flight obstacle avoidance task, and defect detection and localization task; and / or, when the target task is a semantic reasoning task, multiple global embedding vectors are obtained, and the multiple global embedding vectors are weighted and summed through a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector, wherein the multiple global embedding vectors include the global embedding vectors corresponding to the visual imaging embedding vector, the acoustic embedding vector, the point cloud embedding vector, the inertial embedding vector, and the position embedding vector, respectively, wherein the semantic reasoning task includes at least one of the following: equipment status diagnosis task, flight path planning task.

[0007] Optionally, acquiring multiple feature sequences and performing interactive fusion among the multiple feature sequences through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector includes: acquiring the multiple feature sequences; determining attention weights corresponding to the multiple feature sequences respectively, wherein the corresponding attention weights represent the degree of correlation between the target feature position in the corresponding feature sequence and the predetermined feature position in other feature sequences; and performing an interactive fusion operation through a cross-modal attention mechanism based on the attention weights corresponding to the multiple feature sequences to obtain the fused feature sequence, wherein the interactive fusion operation includes weighted fusion of feature information at each feature position in the multiple feature sequences.

[0008] Optionally, multiple global embedding vectors are obtained, and a weighted sum of the multiple global embedding vectors is performed using a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector. This includes: obtaining the multiple global embedding vectors; performing a vector concatenation operation on the multiple global embedding vectors to obtain a joint embedding vector; inputting the joint embedding vector into a gated network, performing a linear transformation on the joint embedding vector through the learning parameter matrix in the gated network, and then processing it with a normalized exponential function to obtain the gate weights corresponding to the multiple global embedding vectors respectively; and obtaining the fused embedding vector based on the multiple global embedding vectors and the gate weights corresponding to the multiple global embedding vectors respectively.

[0009] Optionally, based on the multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, including: encoding the visual imaging data using a depth-separable convolutional neural network to obtain the visual imaging embedding vector; converting the acoustic signal data into a Mel spectrogram, and then encoding the Mel spectrogram using a convolutional neural network with progressively reduced channel counts to obtain the acoustic embedding vector; downsampling the spatial depth point cloud data to obtain downsampled point cloud data, and then encoding the downsampled point cloud data using a multilayer perceptron-based point cloud processing network to obtain the point cloud embedding vector; encoding the attitude inertial sequence data using a target neural network to obtain the inertial embedding vector, wherein the target neural network includes at least one of the following: a gated recurrent unit network, a dilated convolutional temporal network; and encoding the position coordinate data using a fully connected network to obtain the position embedding vector.

[0010] Optionally, before inputting the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space, the method further includes: acquiring sample data and a cross-modal contrastive alignment loss function, wherein the sample data includes positive sample data and negative sample data, the positive sample data includes any combination of sample embedding vectors of two different modalities at the same acquisition time, and the negative sample data includes any combination of sample embedding vectors of two different modalities at different acquisition times; training the initial alignment model, and using the cross-modal contrastive alignment loss function to maximize the similarity between positive sample data and minimize the similarity between negative sample data, until a target condition is reached to obtain the modality alignment model, wherein the target condition includes at least one of the following: reaching a predetermined number of iterations, and the value of the cross-modal contrastive alignment loss function being lower than a preset loss threshold.

[0011] Optionally, after inputting the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space, the method further includes: obtaining the abnormal embedding vector of the target modality, wherein the target modality is the modality in the multimodal embedding vector that detects an anomaly, and the abnormal embedding vector is the feature vector in the target modality corresponding to the abnormal event; determining the similarity between the abnormal embedding vector and the embedding vectors of other modalities in the semantic coreference embedding space; determining the associated embedding vector that matches the abnormal embedding vector from the embedding vectors corresponding to the other modalities based on the similarity; and outputting fault-related data corresponding to the other modalities based on the data acquisition time and spatial location corresponding to the associated embedding vector.

[0012] According to one aspect of the present invention, a device for determining a fused representation vector is provided, comprising: an acquisition module for acquiring multimodal sensor data, wherein the multimodal sensor data is collected by a sensor system mounted on a UAV, and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; a first determination module for performing a multimodal encoding operation based on the multimodal sensor data to obtain a multimodal embedding vector, wherein the multimodal embedding vector includes a visual imaging embedding vector, an acoustic embedding vector, a point cloud embedding vector, an inertial embedding vector, and a position embedding vector; and a second determination module for inputting the multimodal embedding vector into... The modality alignment model is used to obtain aligned embedding vectors in the semantic coreference embedding space. The modality alignment model is trained on the initial alignment model using sample data and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times. A third determination module is used to perform feature fusion operation based on the aligned embedding vectors to obtain a fused representation vector. An execution module is used to execute a target task based on the fused representation vectors. The target task includes at least one of the following: constructing a route space topology map task and determining UAV flight command task.

[0013] According to one aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method for determining the fusion representation vector as described in any of the preceding claims.

[0014] According to one aspect of the present invention, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method for determining the fusion representation vector as described above.

[0015] In this embodiment of the invention, multimodal sensor data is acquired, which is collected by a sensor system mounted on a UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, a multimodal encoding operation is performed to obtain multimodal embedding vectors, which include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors. The multimodal embedding vectors are input into a modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space. The modality alignment model is trained on an initial alignment model using sample data and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times. Based on the aligned embedding vectors, a feature fusion operation is performed to obtain a fused representation vector. Based on the fused representation vectors, a target task is performed, which includes at least one of the following: constructing a route space topology map task, or determining UAV flight command task. By employing a lightweight multimodal embedding and cross-modal semantic alignment approach, multimodal sensor data is synchronously collected by a UAV-mounted sensor system. The data is then lightweightly multimodal encoded, a modal alignment model is trained using a cross-modal contrastive alignment loss function to achieve semantic alignment, and feature fusion is performed on the aligned embedding vectors. This achieves the goal of unified representation and cross-modal semantic alignment of multimodal heterogeneous data in UAV scenarios, thereby improving the unified representation and cross-modal semantic alignment effect of multimodal perception data. It is also adapted to UAV scenarios with limited edge computing resources, supporting the construction of route spatial topology maps and the determination of UAV flight commands. Ultimately, it solves the technical problems of poor unified representation and poor cross-modal semantic alignment of heterogeneous multimodal perception data in UAV scenarios. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of a method for determining a fusion representation vector according to an embodiment of the present invention;

[0018] Figure 2 This is a diagram illustrating the overall system architecture and workflow provided by an optional embodiment of the present invention;

[0019] Figure 3 This is a structural block diagram of a device for determining fusion representation vectors according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1

[0023] According to an embodiment of the present invention, an embodiment of a method for determining a fusion representation vector is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0024] Figure 1 This is a flowchart of a method for determining a fusion representation vector according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0025] Step S102: Acquire multimodal sensor data, which is collected by the sensor system carried by the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data and position coordinate data.

[0026] Multimodal sensor data refers to the multimodal perception data collected by the sensor system carried by the UAV. It can comprehensively collect multi-dimensional information of the scene and avoid the limitations of single data perception, such as visual imaging data and acoustic signal data.

[0027] Among them, drones refer to flying equipment equipped with sensor systems to perform power grid inspections, which can collect multi-dimensional data of power distribution network lines from the air.

[0028] Among them, the sensor system refers to a combination of multiple sensors mounted on the drone, which can simultaneously collect perception data of different modalities, such as a system composed of visible light cameras, acoustic imagers, and lidar.

[0029] Visual imaging data, including visual image data and visible light imaging data, refers to the visible light image sequence captured by a high-definition visible light camera, which can clearly present the appearance, color, markings and visible defects of power equipment, such as visible anomalies such as broken insulators and foreign objects hanging on the conductors.

[0030] Acoustic signal data refers to audio data such as equipment operating noise and discharge sounds collected by acoustic devices in the sensor system, which can identify abnormal acoustic characteristics of the equipment, such as the acoustic signal of corona discharge.

[0031] Among them, spatial depth point cloud data refers to the three-dimensional geometric data of power equipment and lines collected by lidar in the sensor system, which can accurately present the spatial location and structure of the equipment, such as the three-dimensional point cloud data of towers and conductors.

[0032] Among them, attitude inertial sequence data refers to the time-series data such as the flight attitude and angular velocity of the UAV collected by the inertial measurement unit in the sensor system, which can ensure the attitude stability of the data acquisition, such as the pitch and roll attitude data of the UAV during flight.

[0033] Among them, location coordinate data refers to the spatiotemporal location data of the UAV collected by the positioning module in the sensor system, which can accurately locate the geographic coordinates of the collected data, such as centimeter-level latitude and longitude data of RTK positioning.

[0034] This step involves acquiring multimodal sensor data collected by the sensor system onboard the UAV. The data types include visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. This step simultaneously collects multi-dimensional perception data covering vision, acoustics, space, attitude, and position, providing complete and comprehensive raw data support for subsequent multimodal encoding, alignment, and fusion. This avoids subsequent processing deviations caused by missing information at the data source level.

[0035] Step S104: Based on the multimodal sensor data, perform multimodal encoding operation to obtain multimodal embedding vectors, wherein the multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors.

[0036] Multimodal coding refers to using multiple independent deep neural network encoders to extract features and reduce dimensionality of the original high-dimensional data of different modalities, mapping heterogeneous data into low-dimensional dense vectors of a unified dimension. This can eliminate the heterogeneity of the original data and provide a unified data form for cross-modal semantic alignment. For example, deep separable convolutional neural networks are used to encode visual imaging data, and point cloud processing networks based on multilayer perceptrons are used to encode spatial depth point cloud data.

[0037] Among them, multimodal embedding vectors refer to the set of feature vectors with a unified dimension obtained after the data of each modality is mapped by the encoder. They can compress the original sensor data with different dimensions and distributions into a standardized vector representation. They are the unified input format for subsequent cross-modal semantic alignment and feature fusion, including visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors and position embedding vectors.

[0038] Among them, the visual imaging embedding vector refers to the fixed-dimensional feature vector obtained after encoding the visual imaging data, which can compress and retain key visual feature information such as the appearance and texture of power equipment in visible light images.

[0039] Among them, the acoustic embedding vector refers to the fixed-dimensional feature vector obtained by encoding the acoustic signal data after time-frequency transformation, which can compress and retain the frequency domain features of the audio signal related to partial discharge, corona discharge, and mechanical anomalies.

[0040] Among them, point cloud embedding vector refers to a fixed-dimensional feature vector obtained after encoding spatial depth point cloud data, which can compress and retain three-dimensional spatial geometric information such as conductor suspension shape, tower spatial configuration and safety distance.

[0041] Among them, the inertial embedding vector refers to the fixed-dimensional feature vector obtained by encoding the attitude inertial sequence data after sequence modeling, which can compress and preserve the flight attitude stability and trajectory change characteristics of the UAV.

[0042] Among them, the location embedding vector refers to the fixed-dimensional feature vector obtained after encoding the location coordinate data, which can transform latitude, longitude and elevation coordinates into an embedding representation that can be used for subsequent reasoning.

[0043] In this step, the raw data of various modalities obtained in step S102 are input into the corresponding encoder networks. Each encoder independently extracts features and unifies the output to the same vector dimension, forming a multimodal embedding vector set composed of multiple embedding vectors. Through this step, the first layer of mapping from heterogeneous raw data to a unified embedding space is realized, paving the data path for cross-modal semantic alignment.

[0044] Step S106: Input the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space. The modality alignment model is trained by the sample data on the initial alignment model and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times.

[0045] The modality alignment model refers to a pre-trained neural network model that can map embedding vectors from different modalities to a semantic coreference embedding space, so that different modal vectors describing the same power scenario are adjacent to each other in space. It is trained on the initial alignment model by synchronously collected multimodal sample data and cross-modal contrastive alignment loss function, and can directly perform forward inference in the application stage without retraining to achieve cross-modal semantic alignment.

[0046] Among them, the semantic co-reference embedding space refers to the unified semantic representation space to which different modal data are mapped after alignment. It can realize the semantic association of cross-modal data, such as the semantic space shared by various modal data describing the same power equipment.

[0047] Among them, aligned embedding vectors refer to the embedding vectors that are semantically aligned with each other after each modality embedding vector is mapped to the semantic co-reference embedding space through the modality alignment model. This enables data from different modalities to be directly compared and fused in a unified semantic space, eliminating the semantic gap between modalities.

[0048] The cross-modal contrastive alignment loss function refers to the loss function used when training the modal alignment model. This loss function takes different modal embedding vectors at the same synchronous acquisition time as positive sample pairs and different modal embedding vectors at different synchronous acquisition times as negative sample pairs. By maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the model parameters are optimized. This enables the model to learn cross-modal semantic associations without explicit category labels, and is a key technical means to realize semantic coreference embedding space.

[0049] Among them, sample data refers to the synchronously collected multimodal dataset used to train the modality alignment model. It can include positive sample data and negative sample data, which can provide supervision signals for contrastive learning and enable the model to establish cross-modal semantic associations.

[0050] The initial alignment model refers to the untrained modal alignment model, whose parameters are randomly initialized. After training and optimization using sample data and the cross-modal comparative alignment loss function, a deployable modal alignment model is obtained.

[0051] In this step, the multimodal embedding vectors generated in step S104 are input as a whole into a pre-trained modality alignment model. This model is trained using a cross-modal contrastive alignment loss function and a large amount of synchronously collected multimodal sample data. During the inference phase, the model maps each modality embedding vector to the semantic coreference embedding space in one go, outputting semantically aligned alignment embedding vectors. Through this step, the system obtains a semantic foundation for a unified understanding of data from different sensors, solving the semantic gap problem of multimodal data, and enabling subsequent feature fusion and task execution to be performed in a unified semantic space.

[0052] Step S108: Based on the aligned embedding vector, perform a feature fusion operation to obtain a fused representation vector;

[0053] Feature fusion refers to the process of combining semantically aligned modal embedding vectors into a unified scene representation vector using different fusion strategies according to task requirements. It can include two modes: feature sequence interaction fusion based on cross-modal attention mechanism and global vector weighted fusion based on gated multimodal fusion strategy. It can adaptively generate the most suitable joint representation according to the different needs of different tasks for spatial details or high-level semantics.

[0054] Among them, the fusion representation vector refers to the final joint representation generated after feature fusion operation, which integrates all multi-modal data information. It contains a complete semantic description of the current power scenario of the overhead line corridor and can directly support the reasoning and execution of downstream tasks.

[0055] This step integrates multimodal semantic alignment information into a unified scene-level representation, providing high-quality perceptual input for downstream multi-task execution.

[0056] Step S110: Based on the fused representation vector, execute the target task, wherein the target task includes at least one of the following: constructing a route space topology map task, determining UAV flight command task.

[0057] Among them, the target task refers to the specific power inspection application task executed based on the fusion representation vector, including at least one of the tasks of constructing line spatial topology map and determining UAV flight command task, which can transform the unified multimodal semantic representation into actual inspection operation and result output, and realize the end-to-end closed loop from perception to execution.

[0058] Among them, the task of constructing a spatial topology map of the line refers to the task of generating a spatial topology map describing the connection relationship between overhead line equipment based on the fusion representation vector. It can solve the problems of data lag and inaccurate topology relationship in traditional GIS systems, and provide an accurate topology data foundation for the digital operation and maintenance of the power distribution network.

[0059] Among them, the task of determining UAV flight commands refers to the task of generating attitude control commands to control the UAV to autonomously track and fly along the guide line based on the fused representation vector. This can drive the UAV to achieve fully autonomous line tracking flight without the need for preset flight paths or manual operation.

[0060] Through this step, the semantic information of the fused representation vector is transformed into concrete engineering results, realizing a complete application loop from perception to control.

[0061] Through steps S102-S110 above, multimodal sensor data is acquired. This multimodal sensor data is collected by the sensor system mounted on the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. Based on the multimodal sensor data, a multimodal encoding operation is performed to obtain multimodal embedding vectors. These multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors. The multimodal embedding vectors are then input into a modality alignment model to obtain semantic consensus. The term refers to the aligned embedding vector in the embedding space. The modal alignment model is trained on the initial alignment model using sample data. It includes a cross-modal contrastive alignment loss function, which is used to maximize the similarity between different modal embedding vectors at the same acquisition time and minimize the similarity between different modal embedding vectors at different acquisition times. Based on the aligned embedding vector, a feature fusion operation is performed to obtain a fused representation vector. Based on the fused representation vector, a target task is performed, where the target task includes at least one of the following: constructing a route space topology map task, or determining UAV flight command task. By employing a lightweight multimodal embedding and cross-modal semantic alignment approach, multimodal sensor data is synchronously collected by a UAV-mounted sensor system. The data is then lightweightly multimodal encoded, a modal alignment model is trained using a cross-modal contrastive alignment loss function to achieve semantic alignment, and feature fusion is performed on the aligned embedding vectors. This achieves the goal of unified representation and cross-modal semantic alignment of multimodal heterogeneous data in UAV scenarios, thereby improving the unified representation and cross-modal semantic alignment effect of multimodal perception data. It is also adapted to UAV scenarios with limited edge computing resources, supporting the construction of route spatial topology maps and the determination of UAV flight commands. Ultimately, it solves the technical problems of poor unified representation and poor cross-modal semantic alignment of heterogeneous multimodal perception data in UAV scenarios.

[0062] As an optional embodiment, a feature fusion operation is performed based on the aligned embedding vector to obtain a fused representation vector. This includes: when the target task is a fine-grained perception task, acquiring multiple feature sequences, and performing interactive fusion between the multiple feature sequences through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector. The multiple feature sequences include visual feature sequences, point cloud feature sequences, and acoustic feature sequences. The visual feature sequences include spatial grid feature sequences obtained after feature extraction from visual imaging data. The point cloud feature sequences include local geometric feature sequences of key points in spatial depth point cloud data. The acoustic feature sequences include acoustic signal data after time-frequency variation... The resulting time-frequency block feature sequence includes at least one of the following fine-grained perception tasks: wire tracking, flight obstacle avoidance, and defect detection and localization; and / or, when the target task is a semantic reasoning task, multiple global embedding vectors are obtained, and the multiple global embedding vectors are weighted and summed through a gated multimodal fusion strategy to obtain a fused embedding vector as a fused representation vector. The multiple global embedding vectors include global embedding vectors corresponding to visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors, respectively. The semantic reasoning task includes at least one of the following: equipment status diagnosis and flight path planning.

[0063] Among them, fine-grained perception tasks refer to inspection tasks that require the preservation of spatial detail information for accurate perception and positioning. They can make full use of fine-grained features at the feature sequence level for accurate spatial reasoning and decision-making. For example, conductor tracking tasks require accurate identification of the three-dimensional spatial direction of the conductor, flight obstacle avoidance tasks require accurate perception of the spatial position and outline of obstacles, and defect detection and positioning tasks require pixel-level or point-level accurate positioning of defects such as insulator damage and conductor strand breakage.

[0064] Among them, the visual feature sequence refers to the set of feature blocks that retain spatial grid position information after feature extraction from visual imaging data. It can retain detailed information of each spatial position in the image and provide fine-grained visual features for cross-modal spatial-level interaction, such as the feature vector corresponding to a certain pixel region in the image.

[0065] Among them, the point cloud feature sequence refers to the set of local geometric feature blocks obtained after extracting features from key points in spatial depth point cloud data. It can retain the spatial geometric structure information of each key point and provide three-dimensional spatial details for cross-modal interaction, such as the local neighborhood geometric features of a key point in the tower point cloud.

[0066] Among them, the acoustic feature sequence refers to the set of time-frequency block features obtained after time-frequency transformation of acoustic signal data. It can retain the spatial distribution information of acoustic signals in the time and frequency dimensions, and provide acoustic details for cross-modal interaction, such as the feature vector corresponding to a certain time-frequency block in the spectrogram.

[0067] Among them, the cross-modal attention mechanism refers to the mechanism of information interaction between modalities by calculating the attention weights between feature positions in different modal feature sequences. It enables a feature position in one modality to pay attention to all feature positions in another modality that are semantically related to it, realizing the free flow of fine-grained information between modalities. For example, a point in a point cloud that is suspected to be an insulator pays attention to the pixels in the corresponding insulator region in the image.

[0068] Among them, semantic reasoning tasks refer to inspection tasks that require high-level semantic understanding to make global judgments and decisions. They can make full use of high-level semantic information at the global embedding vector level for reasoning. For example, equipment status diagnosis tasks need to comprehensively judge whether the equipment has defects and the type of defects, while flight path planning tasks need to select the optimal flight path based on the overall topology of the line.

[0069] Among them, the global embedding vector refers to the fixed-dimensional feature vector obtained by the encoder after extracting and globally pooling each modality data. It compresses global information and can represent the overall semantic content of the corresponding modality data. It is suitable for high-level semantic reasoning tasks. For example, the visual imaging embedding vector represents the global visual semantics of the entire image after global pooling.

[0070] Among them, the gated multimodal fusion strategy refers to the strategy of adaptively weighting and fusing the global embedding vectors of each modality by learning the gate weights. It can automatically adjust the contribution of each modality according to the current environmental context and improve the robustness of the fusion representation. For example, it can automatically increase the weights of infrared modality and lidar modality at night or in foggy weather.

[0071] In this embodiment, two different fusion strategies are used to generate fused representation vectors depending on the type of the target task. When the target task is a fine-grained perception task, visual feature sequences, point cloud feature sequences, and acoustic feature sequences are acquired. Attention weights between feature positions in each feature sequence are calculated using a cross-modal attention mechanism. The feature information at each feature position is then weighted and fused according to the attention weights to obtain a fused feature sequence as the fused representation vector. When the target task is a semantic reasoning task, global embedding vectors corresponding to each modality are acquired. Multiple global embedding vectors are adaptively weighted and summed using a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector.

[0072] This approach allows for the flexible selection of the most suitable fusion strategy based on the varying spatial detail and high-level semantic requirements of downstream tasks, achieving an optimal balance between computational efficiency and perceptual accuracy. The cross-modal attention mechanism, by preserving the spatial structure of feature sequences, enables fine-grained perception tasks such as wire tracking, obstacle avoidance, and defect localization to obtain pixel-level or point-level spatial interaction information, significantly improving spatial perception accuracy. The gated multimodal fusion strategy, through an adaptive weighting mechanism, enables semantic reasoning tasks such as equipment status diagnosis and flight path planning to automatically increase the weights of other reliable modalities in visually degraded environments such as nighttime and foggy conditions, effectively suppressing interference from unreliable modal information and ensuring the robustness and accuracy of high-level decision-making. The synergistic cooperation of these two fusion strategies ensures that the fused representation vector possesses optimal expressive power across different task scenarios.

[0073] As an optional embodiment, multiple feature sequences are acquired, and the interaction between the multiple feature sequences is fused through a cross-modal attention mechanism to obtain a fused feature sequence as a fused representation vector. This includes: acquiring multiple feature sequences; determining attention weights corresponding to the multiple feature sequences respectively, wherein the corresponding attention weights represent the degree of correlation between the target feature position in the corresponding feature sequence and the predetermined feature position in other feature sequences; and performing an interactive fusion operation through a cross-modal attention mechanism based on the attention weights corresponding to the multiple feature sequences to obtain a fused feature sequence, wherein the interactive fusion operation includes weighted fusion of feature information at each feature position in the multiple feature sequences.

[0074] Among them, attention weight refers to the weight value calculated in the cross-modal attention mechanism, which represents the degree of semantic association between each feature position in the feature sequence of different modalities. It can quantify the correlation between a certain feature position in one modality and each feature position in another modality, such as the degree of association between a key point in a point cloud feature sequence and each spatial grid feature in a visual feature sequence.

[0075] The target feature location refers to a specific feature location in the feature sequence where attention weights are currently being calculated. It can serve as the starting point for cross-modal information interaction and guide the model to focus on other modal features that are semantically related to that location.

[0076] Among them, the predetermined feature position refers to each feature position in other feature sequences that is associated with the target feature position and can provide contextual information that may be semantically related to the target feature position.

[0077] In this embodiment, after obtaining the visual feature sequence, point cloud feature sequence and acoustic feature sequence, the attention weight corresponding to each feature sequence is determined. The attention weight represents the degree of semantic association between each feature position in the corresponding feature sequence and each feature position in other feature sequences. Then, the feature information of each feature position is weighted and fused according to the attention weight to obtain the fused feature sequence.

[0078] In this way, the cross-modal attention mechanism enables full semantic interaction between feature sequences of different modalities in the spatial dimension. For example, pixel regions in the visual feature sequence can be directly associated with corresponding spatial points in the point cloud feature sequence, and time-frequency blocks in the acoustic feature sequence can focus on corresponding device regions in the visual feature sequence. This breaks down the information barriers between modalities and achieves true cross-modal fine-grained perception fusion. The fused feature sequence simultaneously possesses visible light appearance details, three-dimensional geometric information from LiDAR, and acoustic anomaly information, providing information redundancy and complementarity for fine-grained perception tasks such as wire tracking, obstacle avoidance, and defect detection and localization, significantly improving the accuracy and robustness of perception.

[0079] As an optional embodiment, multiple global embedding vectors are obtained, and a weighted sum of the multiple global embedding vectors is performed through a gated multimodal fusion strategy to obtain a fused embedding vector as a fused representation vector. This includes: obtaining multiple global embedding vectors; performing a vector concatenation operation on the multiple global embedding vectors to obtain a joint embedding vector; inputting the joint embedding vector into a gated network, performing a linear transformation on the joint embedding vector through the learning parameter matrix in the gated network, and then processing it through a normalized exponential function to obtain the gate weights corresponding to the multiple global embedding vectors respectively; and obtaining the fused embedding vector based on the multiple global embedding vectors and the gate weights corresponding to the multiple global embedding vectors respectively.

[0080] Among them, the joint embedding vector refers to the high-dimensional vector obtained by concatenating the global embedding vectors of multiple modalities in sequence. It can gather the global semantic information of each modality into a single vector, providing all available modal information for the gating network.

[0081] Among them, a gating network refers to a small neural network module consisting of a learnable parameter matrix and a normalized exponential function, which can adaptively output the weight coefficients of each modality according to the current input, thereby realizing the dynamic evaluation of the importance of different modalities.

[0082] Among them, the gating weights refer to the weight coefficients output by the gating network that correspond one-to-one with the global embedding vectors of each modality. They can adaptively adjust the contribution of each modality in the fusion process according to the current environmental context. The values ​​range from zero to one and the sum of each weight is one. For example, in nighttime scenes, higher weights are automatically assigned to the infrared modality.

[0083] In this embodiment, after obtaining multiple global embedding vectors, a vector concatenation operation is performed on the multiple global embedding vectors to obtain a joint embedding vector. The joint embedding vector is input into a gating network. After linear transformation of the joint embedding vector through the learning parameter matrix, it is processed by a normalized exponential function to obtain the gating weights corresponding to the multiple global embedding vectors. The global embedding vectors are weighted and summed according to the gating weights to obtain the fused embedding vector.

[0084] In this approach, the gated multimodal fusion strategy adaptively determines the fusion weights of each modality through learning, rather than using fixed weights or manually preset rules. This allows the system to dynamically adjust the level of trust in each modality based on changes in the environmental context. For example, in well-lit daytime conditions, the visual and infrared modalities receive higher weights to leverage their appearance perception advantages. In visually degraded environments such as nighttime, fog, or backlighting, the weights of the lidar and acoustic modalities automatically increase, while unreliable information from the visual modality is effectively suppressed. This ensures the robustness and accuracy of the fused representation vector under various environmental conditions, providing high-quality perceptual input for high-level semantic reasoning tasks such as equipment status diagnosis and flight path planning.

[0085] As an optional embodiment, based on multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, including: encoding visual imaging data using a deep separable convolutional neural network to obtain a visual imaging embedding vector; converting acoustic signal data into a Mel spectrogram, and then encoding the Mel spectrogram using a convolutional neural network with progressively reduced channel counts to obtain an acoustic embedding vector; downsampling spatial depth point cloud data to obtain downsampled point cloud data, and then encoding the downsampled point cloud data using a multilayer perceptron-based point cloud processing network to obtain a point cloud embedding vector; encoding attitude inertial sequence data using a target neural network to obtain an inertial embedding vector, wherein the target neural network includes at least one of the following: a gated recurrent unit network, a dilated convolutional temporal network; and encoding position coordinate data using a fully connected network to obtain a position embedding vector.

[0086] Among them, depthwise separable convolutional neural networks refer to lightweight convolutional neural network structures that decompose standard convolution into two stages: channel-wise convolution and point-wise convolution. This can significantly reduce the number of parameters and computational cost of convolution operations, while maintaining feature extraction capabilities and adapting to the resource constraints of UAV edge computing platforms.

[0087] Mel spectrum refers to a two-dimensional spectral feature map obtained by time-frequency transformation of acoustic signal data and expressed on a Mel frequency scale. It can simulate the nonlinear perception characteristics of human ear for sound frequency, highlight the frequency domain features related to partial discharge, corona discharge and mechanical abnormalities, and compress redundant information in the high-frequency part.

[0088] Among them, the convolutional neural network with the number of channels decreasing layer by layer refers to a convolutional neural network structure in which the number of output channels of each convolutional layer decreases layer by layer. It can compress the dimension of features layer by layer while ensuring feature extraction capabilities, reduce computational overhead, and adapt to edge computing scenarios.

[0089] Downsampled point cloud data refers to point cloud data with fewer points obtained after downsampling the original spatial depth point cloud data. This can reduce the input scale of point cloud processing and reduce the computational load of the point cloud processing network.

[0090] Among them, the point cloud processing network based on multilayer perceptron refers to a network structure that learns the features of the local neighborhood geometric relationships of each point in the point cloud by stacking multilayer perceptrons. It can directly process unordered point cloud data and extract the local geometric features of the points.

[0091] Among them, gated recurrent unit network refers to a recurrent neural network structure that uses a gating mechanism to control the flow of information. It can effectively capture long-term dependencies in time-series data and has the advantage of fewer parameters compared to long short-term memory networks.

[0092] Among them, dilated convolutional temporal networks refer to convolutional neural network structures that use dilated convolutional kernels to extract features from time-series data. They can expand the receptive field without increasing the number of parameters and efficiently capture multi-scale temporal features in time-series signals.

[0093] Among them, a fully connected network refers to a neural network structure in which neurons in each layer are connected to all neurons in the layer above, and can map low-dimensional coordinate data into a high-dimensional embedding space.

[0094] In this embodiment, differentiated network structure designs are used to perform multimodal encoding operations based on the characteristics and computational requirements of different modal data. Visual imaging data is encoded using a deep separable convolutional neural network, which significantly reduces computational load through channel-wise and point-wise convolution decomposition. Acoustic signal data is first converted into Mel spectrograms, and then extracted for dimensionality compression using a convolutional neural network with progressively reduced channel count. Spatial depth point cloud data is first downsampled to reduce the number of points, and then local geometric features of each point are extracted using a point cloud processing network based on a multilayer perceptron. Attitude inertial sequence data captures temporal motion features through a gated recurrent unit network or a dilated convolutional temporal network. Position coordinate data is mapped to the embedding space through a fully connected network.

[0095] In this approach, each modal encoder is structurally optimized for the resource constraints of edge computing platforms, reducing the number of model parameters and computational overhead while maintaining feature extraction capabilities. Depthwise separable convolutions reduce convolutional computation, point cloud downsampling and layer-by-layer channel reduction effectively control network size, gated recurrent unit networks have fewer parameters than long short-term memory networks, and dilated convolutional temporal networks extract temporal features more efficiently. Ultimately, this enables multimodal coding operations to run in real time on UAV-borne edge computing platforms without relying on high-performance computing resources from ground stations or the cloud, ensuring the real-time nature and autonomy of multimodal perception.

[0096] As an optional embodiment, before inputting the multimodal embedding vectors into the modality alignment model to obtain the aligned embedding vectors in the semantic coreference embedding space, the method further includes: acquiring sample data and a cross-modal contrastive alignment loss function, wherein the sample data includes positive sample data and negative sample data, the positive sample data includes any combination of sample embedding vectors of two different modalities at the same acquisition time, and the negative sample data includes any combination of sample embedding vectors of two different modalities at different acquisition times; training the initial alignment model, and using the cross-modal contrastive alignment loss function to maximize the similarity between positive sample data and minimize the similarity between negative sample data, until the target condition is reached to obtain the modality alignment model, wherein the target condition includes at least one of the following: reaching a predetermined number of iterations, and the value of the cross-modal contrastive alignment loss function being lower than a preset loss threshold.

[0097] The sample data refers to the synchronously collected multimodal embedding vector dataset used to train the modality alignment model, including positive and negative sample data, which can provide paired supervision signals for contrastive learning, enabling the model to learn cross-modal semantic associations.

[0098] Positive sample data refers to any combination of two different modalities' sample embedding vectors collected at the same time, which can serve as positive sample pairs for comparative learning, conveying the training objective to the model that "these two modalities describe the same power scenario".

[0099] Negative sample data refers to any combination of two different modalities of sample embedding vectors collected at different times. These can be used as negative sample pairs for comparative learning, conveying the training objective to the model that "these two modalities describe different power scenarios", thus enabling the model to distinguish between different scenarios.

[0100] Among them, sample embedding vector combination refers to the pairing of embedding vectors of any two different modalities selected from the sample data, which can form the basic unit of contrastive learning training.

[0101] The predetermined number of iterations refers to the maximum number of loops preset during the training process, which can terminate training when the model reaches this number of loops, thus avoiding overtraining or excessively long training time.

[0102] The preset loss threshold refers to the target lower limit of the cross-modal comparison alignment loss function value. It can determine that the model has converged and terminate training when the loss function value is lower than the threshold.

[0103] In this embodiment, before applying the modal alignment model for inference, an initial alignment model is first trained using sample data and a cross-modal contrastive alignment loss function. Sample data consisting of positive and negative samples is obtained, and the cross-modal contrastive alignment loss function is used to maximize the similarity of positive samples and minimize the similarity of negative samples. The parameters of the initial alignment model are continuously optimized and updated until a predetermined number of iterations is reached or the loss function value is lower than a preset loss threshold, resulting in a deployable modal alignment model.

[0104] In this approach, the model learns cross-modal semantic associations through a large amount of synchronously collected multimodal sample data during the training phase, establishing semantic correspondences between different modalities without the need for manual labeling. Embedding vectors from different modalities at the same time are brought closer together to neighboring positions in the semantic coreference embedding space, while embedding vectors from different times are pushed apart. This allows different modalities describing the same power equipment or event to naturally cluster in the semantic coreference embedding space after training, providing a geometric basis for cross-modal semantic alignment in the application phase. This reduces the dependence on large amounts of labeled pairing data and improves the system's scalability.

[0105] As an optional embodiment, after inputting the multimodal embedding vectors into the modality alignment model to obtain the aligned embedding vectors in the semantic coreference embedding space, the method further includes: obtaining the abnormal embedding vector of the target modality, wherein the target modality is the modality in the multimodal embedding vectors that detects anomalies, and the abnormal embedding vector is the feature vector in the target modality corresponding to the abnormal event; determining the similarity between the abnormal embedding vector and the embedding vectors of other modalities in the semantic coreference embedding space; determining the associated embedding vector that matches the abnormal embedding vector from the embedding vectors corresponding to other modalities based on the similarity; and outputting fault association data corresponding to other modalities based on the data acquisition time and spatial location corresponding to the associated embedding vector.

[0106] Among them, the target mode refers to a specific mode in the multimodal embedding vector that detects an anomaly. It can serve as the starting point for cross-modal fault association queries and trigger the retrieval and location of other modal data. For example, when the acoustic mode detects an abnormal corona discharge, the acoustic mode is the target mode.

[0107] Among them, the abnormal embedding vector refers to the embedding vector corresponding to the abnormal segment in the target modal data. It can be used as the query vector for cross-modal retrieval to retrieve other modal data that are semantically related to it in the semantic coreference embedding space, such as the embedding vector of the acoustic segment of corona discharge.

[0108] Among them, the associated embedding vector refers to the embedding vector of other modalities that has the highest similarity to the anomalous embedding vector in the semantic co-reference embedding space. It can point to other modal data in the same scene or the same device as the anomalous event, such as the visual image embedding vector that best matches the acoustic embedding vector of corona discharge.

[0109] Among them, fault-related data refers to the original data of other modalities associated with abnormal events, which are output based on the acquisition time and spatial location corresponding to the association embedding vector. It can provide maintenance personnel with multimodal fault judgment basis, such as infrared thermal images or visible light images associated with abnormal sounds, highlighting possible faulty equipment areas in the images.

[0110] In this embodiment, when an anomaly is detected in a certain modality, the anomaly embedding vector of the target modality is obtained. The similarity between the anomaly embedding vector and the embedding vectors of other modalities in the semantic coreference embedding space is calculated. Based on the similarity, the associated embedding vector matching the anomaly embedding vector is determined from the embedding vectors corresponding to other modalities. Based on the data acquisition time and spatial location corresponding to the associated embedding vector, fault association data corresponding to other modalities is output. In this way, the cross-modal semantic alignment characteristic of the semantic coreference embedding space is utilized to achieve zero-sample cross-modal fault association and localization. Even if the model is not directly trained using specific fault type paired data, when an anomaly is detected in a certain modality, it can automatically associate the most likely corresponding fault evidence in other modalities by searching for nearest neighbors in the shared embedding space, providing maintenance personnel with multi-dimensional fault judgment information. This method reduces the dependence on a large amount of labeled paired data and improves the system's adaptability and scalability in the face of unknown faults or new fault types.

[0111] Based on the above embodiments and optional embodiments, an optional implementation method is provided, which is described in detail below.

[0112] In related technologies, distribution network topology is a structured model describing the electrical connections between various devices (such as lines, switches, transformers, and load points) in a power network. It serves as the foundation for distribution network operation, analysis, control, and optimization. Accurate, real-time, and reliable topology information is crucial for achieving intelligent management of distribution networks, directly supporting advanced applications such as power flow calculation, state estimation, fault location, isolation and recovery, reactive power optimization, and network reconfiguration.

[0113] Currently, distribution network topology identification and management mainly rely on the following traditional or rudimentary information-based methods:

[0114] 1. Manual on-site inspection and drawing comparison: This is the most traditional method, relying on maintenance personnel to conduct regular or irregular on-site inspections. They manually record and compare the physical equipment and wiring with existing primary wiring diagrams and Geographic Information System (GIS) drawings to ensure consistency. The core problem with this method is its significant time lag. Distribution networks, especially medium and low voltage distribution networks, experience relatively frequent structural changes (such as business expansion applications, line relocation, temporary power transfer, and network optimization). Manual inspections are time-consuming and costly, making it difficult to capture and update these changes in a timely manner. This results in a long-term inconsistency between the topology information recorded in the system and the actual physical connections.

[0115] 2. Offline Integration Based on Static GIS / SCADA: Currently, most distribution management systems (DMS) attempt to correlate spatial equipment information provided by GIS with real-time telemetry and telecontrol information provided by SCADA (Data Acquisition and Monitoring Control System). However, this integration is often static and unidirectional. GIS typically provides the initial topology as the "source system," but subsequent network changes (especially changes in the operating topology caused by changes in the status of field switches) mainly rely on SCADA communication signals. The problem is that when field equipment is modified, added, or removed, if it is not updated in the GIS in a timely manner, its static topology model becomes disconnected from the dynamic signals of SCADA. More importantly, existing systems generally lack an effective dynamic verification and reverse correction mechanism, and cannot use real-time measurement data from SCADA (such as voltage, current, and power) to automatically detect and correct topology errors in the GIS.

[0116] 3. Attempts at Automated Identification from Single Data Sources: In recent years, some studies have attempted to infer topology using single data sources such as voltage correlation from Advanced Measurement Systems (AMI), power flow information from SCADA, or fault indicator signals, through data mining or state estimation methods. However, these methods have significant limitations: they are highly dependent on data quality and integrity; insufficient AMI coverage, SCADA communication interruptions, or signal errors can all lead to identification failures or errors. They cannot distinguish between physical connections and operational status: it is difficult to accurately identify changes in operational topology caused by switch opening and closing and permanent alterations to the network's physical structure. They lack verification with spatial geographic information: the inferred electrical connections lack a visual and locatable correlation with the spatial location of physical equipment, which is detrimental to on-site maintenance operations. They create "information silos": the above methods are usually conducted independently, failing to deeply integrate and cross-validate with multi-source information such as GIS graphics, equipment ledgers, and inspection images; the topology information between different systems remains isolated and contradictory.

[0117] It can be seen that, based on the existing methods described above, current power distribution network topology identification and management technologies mainly suffer from the following prominent defects and shortcomings:

[0118] 1. Information updates are severely lagging, and static management has significant drawbacks: The management model, which is based on manual processes and static GIS, cannot keep up with the frequency of actual structural changes in the power distribution network. The "shelf life" of topological information is extremely short, and it quickly "deteriorates" after frequent operations and modifications, causing all advanced applications that rely on it to be built on a flawed foundation. The so-called "panoramic perception" is actually an "outdated map".

[0119] 2. Maintenance relies heavily on manual intervention, resulting in low efficiency and continuously deteriorating reliability: Topology maintenance is highly dependent on human intervention, which is not only inefficient and costly, but also prone to subjective negligence and operational errors. Every failure to update in a timely or accurate manner causes a cumulative decline in the reliability of topology information within the system, forming a "deterioration cycle" that seriously erodes the foundation of the automated system.

[0120] 3. Topology errors trigger a cascading failure in advanced applications: Inaccurate topology information directly leads to distorted power flow calculations, divergent state estimations, inaccurate fault location, and ineffective recovery strategies. For example, fault zone identification based on incorrect topology can mislead line inspections and delay emergency repairs; incorrect network reconfiguration schemes may introduce new security risks. This significantly reduces the effectiveness of many intelligent algorithms in real-world applications, severely impacting power supply reliability and operational accuracy.

[0121] 4. Isolated Multi-Source Data and Lack of Dynamic Fusion and Verification Mechanism: Under the existing system architecture, GIS graphic data, SCADA real-time signals, AMI user data, and inspection imagery are isolated "information silos." The system lacks a core mechanism capable of proactively fusing multi-source heterogeneous data and performing consistency verification and conflict resolution. It cannot utilize real-time electrical measurements to dynamically verify the static topology, nor can it use visual evidence from on-site inspections to corroborate or update equipment connection relationships; the collaborative value between data remains untapped.

[0122] 5. Lack of adaptive and automated error correction capabilities: Existing technical solutions are essentially passive and reactive, only allowing for manual verification after problems are exposed (such as application errors caused by topology errors). There is an urgent need for a technology that can proactively sense network changes, automatically verify multi-source information, intelligently infer true connections, and autonomously update the topology model, achieving a fundamental shift from "static recording" to "dynamic sensing and self-correction."

[0123] Therefore, to address the aforementioned issues, an innovative technical solution is urgently needed in this field. In resource-constrained scenarios such as drones, embedded devices, and edge computing, this invention proposes a method for determining fused representation vectors, also known as a lightweight multimodal embedding model and cross-modal semantic alignment method, to achieve deeper levels of perception and reasoning. It leverages advanced technologies such as drones, the Internet of Things, and edge computing to fuse multimodal perception data, enabling real-time, dynamic, and automated identification and updating of the physical and operational topology of the distribution network. This lays a solid data foundation for building a truly intelligent, reliable, and adaptive modern distribution network.

[0124] Figure 2 This is a system overall architecture and workflow diagram provided by an optional embodiment of the present invention, such as... Figure 2 The diagram illustrates the overall architecture and workflow of a lightweight multimodal perception and fusion system based on unmanned aerial vehicles (UAVs). This includes:

[0125] 1. Data Acquisition Layer: The UAV is equipped with multiple types of sensors (visible light camera, infrared thermal imager, acoustic imager, lidar, IMU, GPS) to simultaneously collect raw data in six modalities during the inspection flight (similar to the acquisition of multimodal sensor data mentioned above, where multimodal sensor data is collected by the sensor system on the UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data).

[0126] 2. Edge Computing Layer (Core Processing Unit): Deploy the model described in this invention on airborne or near-field edge computing devices. This includes a lightweight multimodal encoder group: extracting features from six types of raw data to generate unified low-dimensional embedding vectors or feature sequences (similar to the above-mentioned multimodal encoding operation based on multimodal sensor data to obtain multimodal embedding vectors, where the multimodal embedding vectors include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors). A cross-modal semantic alignment module: training the model through contrastive learning (using the InfoNCE loss function) to bring data from different modalities but describing the same entity closer together in the embedding space (similar to the above-mentioned inputting multimodal embedding vectors into the modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space, where the modality alignment model is trained from sample data on an initial alignment model, including a cross-modal contrastive alignment loss function, which maximizes the similarity between different modal embedding vectors at the same acquisition time and minimizes the similarity between different modal embedding vectors at different acquisition times). Hierarchical feature fusion module: Based on task instructions (such as "fine detection" or "state diagnosis"), select the "early interaction fusion" or "late semantic fusion" strategy to generate a robust joint scene representation (align the embedding vector with the above criteria, perform feature fusion operation, and obtain the fused representation vector).

[0127] 3. Application Layer: The fused representation is output to the downstream task module to realize line tracking, defect diagnosis, 3D reconstruction, etc., and supports cross-modal retrieval (such as searching for images with text) and zero-shot transfer (such as locating fault points with sound) (same as above, based on the fused representation vector, the target task is executed, where the target task includes at least one of the following: constructing a line space topology map task, determining UAV flight command task).

[0128] The following is combined Figure 2 The optional embodiments of the present invention are described below. The model in the optional embodiments of the present invention includes the following key components:

[0129] 1. Multimodal encoder group: For the six input modalities, lightweight neural network encoders are designed or adapted to map the original high-dimensional heterogeneous data to low-dimensional, dense embedding vectors.

[0130] 2. Cross-modal semantic alignment module: Based on the contrastive learning framework, during the model training phase, it drives different modal encoders to project data describing the same power scene or entity into positions close to each other in a unified semantic embedding space.

[0131] 3. Hierarchical Feature Fusion Module: Based on the complexity and requirements of downstream tasks, it provides two optional fusion strategies: early interaction fusion and late semantic fusion, to generate robust joint scene representations.

[0132] 4. Task Decoder / Retrieval Interface: Based on the fused representation, it supports multiple downstream tasks, such as device identification, fault diagnosis, and path planning, and provides a cross-modal retrieval interface.

[0133] The specific implementation steps of this invention are as follows:

[0134] (I) Multimodal data acquisition and preprocessing:

[0135] The system synchronously collects time-series data for six modalities:

[0136] Visual images: captured from a high-definition visible light camera, showing the device's appearance, logo, and environment.

[0137] Thermal imaging: From an infrared camera, it acquires the temperature distribution of the device and detects hotspots.

[0138] The aforementioned visual images and thermal images are the same as the aforementioned visual imaging data.

[0139] Acoustic signals (same as the acoustic signal data mentioned above): These are collected from acoustic imagers or ultrasonic testing instruments, capturing abnormal acoustic characteristics such as equipment operating noise and corona discharge, and are considered as a special type of audio stream.

[0140] Spatial Depth / Point Cloud (same as the spatial depth point cloud data mentioned above): obtained from LiDAR, acquiring precise three-dimensional geometric information of power lines, towers and the surrounding environment.

[0141] Attitude inertial sequence (same as the attitude inertial sequence data above): from the inertial measurement unit, providing the UAV's own flight attitude, angular velocity and acceleration data.

[0142] Location coordinates (same as the location coordinate data above): from the Global Positioning System or fused positioning module, providing spatiotemporal stamp information.

[0143] All data is synchronized via hardware or software timestamps during acquisition to ensure that modal data at the same moment correspond to the same scene context.

[0144] (II) Lightweight Multimodal Encoder Design:

[0145] To adapt to the resource constraints of edge computing, lightweight designs are implemented for each modal encoder:

[0146] Visual / thermal imaging encoder: Use lightweight convolutional networks such as MobileNetV3 or EfficientNet-Lite after cropping or knowledge distillation, or design a smaller custom CNN to output global image embedding vectors and optional spatial feature maps (similar to using a deep separable convolutional neural network to encode visual imaging data to obtain visual imaging embedding vectors).

[0147] Acoustic signal encoder: Converts acoustic signals into time-frequency representations such as Mel spectrograms, and uses lightweight 1D / 2D CNN or MobileViT for feature extraction (similar to the above, after converting acoustic signal data into Mel spectrograms, a convolutional neural network with progressively reduced channel count is used to encode the Mel spectrograms to obtain acoustic embedding vectors).

[0148] Point cloud encoder: A lightweight model based on a simplified version of PointNet++ or a sparse convolutional network is used to extract the global geometric embedding and local features of the point cloud (similar to the downsampling of spatial depth point cloud data mentioned above, after obtaining downsampled point cloud data, a point cloud processing network based on a multilayer perceptron is used to encode the downsampled point cloud data to obtain the point cloud embedding vector).

[0149] IMU sequence encoder: Lightweight recurrent neural network or temporal convolutional network is used to process temporal inertial data (similar to the above-mentioned use of target neural network to encode attitude inertial sequence data to obtain inertial embedding vector, wherein the target neural network includes at least one of the following: gated recurrent unit network, dilated convolutional temporal network).

[0150] GPS encoder: Converts latitude and longitude coordinates into local relative coordinates or rasterized representation, and encodes them through a small fully connected network (similar to the above-mentioned use of a fully connected network to encode location coordinate data to obtain a location embedding vector).

[0151] All encoders output embedding vectors of a uniform dimension, which facilitates subsequent alignment and fusion operations.

[0152] (III) Cross-modal semantic alignment method based on contrastive learning:

[0153] To achieve semantic consistency and interchangeability of data from different modalities, a contrastive learning framework based on noise contrast estimation is used for joint training.

[0154] a) Training data construction and definition of positive and negative samples:

[0155] During training, batch data is constructed, with each sample containing multiple modalities of data synchronously collected at the same time and under the same power scenario. Definition:

[0156] Positive sample pair: Embedding vectors of any two modalities from the same training sample (such as wire image embedding and corresponding infrared thermogram embedding, image embedding and simultaneous corona discharge acoustic embedding, etc.).

[0157] Negative sample pair: Within a batch, the embedding vectors of any two modalities from different training samples (i.e., different scenes or different times).

[0158] Similar to the above-mentioned acquisition of sample data and cross-modal comparison alignment loss function, the sample data includes positive sample data and negative sample data. Positive sample data includes the combination of sample embedding vectors of any two different modalities at the same acquisition time, and negative sample data includes the combination of sample embedding vectors of any two different modalities at different acquisition times.

[0159] b) Objective function and training process:

[0160] The InfoNCE loss function is adopted as the core objective of alignment training. For the embedding vectors of mode i and mode j, its loss function is formalized as follows:

[0161]

[0162] in:

[0163] and To form a positive sample pair (e.g., For image embedding, (For embedding the corresponding audio).

[0164] This represents the modality j embedding of other samples in the batch, which is considered a negative sample.

[0165] The similarity function is cosine similarity.

[0166] This is a temperature hyperparameter used to adjust the level of attention given to difficult negative samples.

[0167] N is the batch size.

[0168] This is an indicator function.

[0169] The total loss is the average of the sum of the losses for all modes: , where M is the number of mode pairs.

[0170] The initial alignment model is trained in the same way as described above. The cross-modal contrastive alignment loss function is used to maximize the similarity between positive sample data and minimize the similarity between negative sample data until the target condition is reached, thus obtaining the modal alignment model. The target condition includes at least one of the following: reaching a predetermined number of iterations, and the value of the cross-modal contrastive alignment loss function is lower than a preset loss threshold.

[0171] By optimizing this objective, the model can learn cross-modal semantic associations (such as the intrinsic relationship between "conductor heating" and "specific infrared mode", "corona discharge sound" and "specific acoustic spectrum features", and "insulator" and "specific point cloud structure") without explicit category labels, forming a unified semantic coreference embedding space. In this space, different modal data describing the same power equipment, state, or event have highly similar embedding vectors.

[0172] (iv) Hierarchical feature fusion mechanism:

[0173] After obtaining the semantically aligned modal embeddings, a hierarchical fusion mechanism is used to generate the final scene representation.

[0174] a) Early interaction fusion (suitable for fine-grained perception tasks):

[0175] For tasks requiring spatial detail, such as wire tracking, obstacle avoidance, and fine-grained defect detection, a lightweight cross-modal attention module is used to achieve early feature interaction. This module is a simplified version of the Transformer decoder block.

[0176] Input: Feature sequences (not global vectors) extracted from visual, LiDAR (point cloud), and acoustic modalities. For example, spatial grid features from visual feature maps, features of key points in point clouds, and time-frequency block features from spectrograms.

[0177] Process: Through a multi-head attention mechanism, attention weights can be calculated between a specific feature location in one modality (such as a pixel region in an image) and all feature locations in another modality. For example, a point in a point cloud that appears to be an insulator can focus on the corresponding insulator region in the image, or an abnormal audio clip can focus on a region with an abnormal temperature in a thermal image.

[0178] Output: The fused feature sequence after interaction, which can be used for pixel-level or point-level prediction tasks. The formula is expressed as:

[0179] .

[0180] Similar to the above, when the target task is a fine-grained perception task, multiple feature sequences are acquired, and the interaction and fusion of multiple feature sequences are carried out through a cross-modal attention mechanism to obtain a fused feature sequence as a fused representation vector. Among them, the multiple feature sequences include visual feature sequences, point cloud feature sequences, and acoustic feature sequences. The visual feature sequences include spatial grid feature sequences obtained after feature extraction of visual imaging data. The point cloud feature sequences include local geometric feature sequences of key points in spatial depth point cloud data. The acoustic feature sequences include time-frequency block feature sequences obtained after time-frequency transformation of acoustic signal data. The fine-grained perception task includes at least one of the following: wire tracking task, flight obstacle avoidance task, and defect detection and localization task.

[0181] Among them, Ev: Embedded features of the visual modality. These are typically feature vectors or feature maps extracted from visible light images using an encoder (such as EfficientNet), representing the appearance information of equipment such as conductors, insulators, and towers. El: Embedded features of the LiDAR modality. These are geometric features extracted from laser point clouds using networks such as PointNet++, reflecting the three-dimensional structure, spatial location, and distance relationship with surrounding objects of the equipment. Ea: Embedded features of the acoustic modality. These are acoustic features extracted from the spectrum generated by the short-time Fourier transform of the audio signal using networks such as MobileNet, representing abnormal sound patterns such as partial discharge and corona discharge. Ffusion: The fused output feature. This feature simultaneously contains visual appearance information, LiDAR geometric structure information, and acoustic event information, used for subsequent tasks such as conductor tracking, obstacle avoidance decision-making, or defect identification.

[0182] b) Late-stage semantic fusion (suitable for high-level semantic reasoning tasks):

[0183] For tasks such as equipment status diagnosis, scene classification, and trajectory planning, the global embedding vectors of each modality are directly fused. A gated multimodal fusion strategy is adopted to adaptively adjust the contribution of each modality.

[0184]

[0185] in, Representing the global embedding vectors for each modality, ⊙ indicates element-wise multiplication. efusio: The final embedding vector output after fusion. This vector integrates information from all modalities and can be used for subsequent equipment status diagnosis (e.g., determining whether an insulator has spontaneously exploded), scene classification (e.g., identifying straight poles or corner poles), or trajectory planning (e.g., deciding on branch direction). ev,et,ea,el,ep: Represent the global embedding vectors for the five modalities: visual (visible light), infrared thermal imaging, acoustic, lidar, and position / attitude, respectively. Each vector has a fixed dimension d (e.g., 256 dimensions), extracted from the original data by the encoder of its respective modality (e.g., EfficientNet, ResNet, MobileNet, PointNet++, LSTM, etc.). These embedding vectors have encoded the high-level semantic information of the corresponding modality. gv,gt,ga,gl,gp: Gated weight vectors corresponding to the five modalities, with the same dimensions as the embedding vectors. The elements of each gated vector are usually constrained to the interval [0,1][0,1], representing the "preservation degree" or "importance" of each feature dimension of the corresponding modality embedding. The values ​​of the gating vectors are dynamically generated by the current environment or task. For example, a small neural network can be used to take environmental visibility, sensor confidence, etc., as input and output the gating weights for each modality. Through learning:

[0186]

[0187] Here, || represents vector concatenation. and These are learnable parameters. This mechanism can automatically increase the weight of infrared or lidar modes based on environmental context (such as nighttime or fog) and suppress unreliable information from visual modes.

[0188] As mentioned above, when the target task is semantic reasoning, multiple global embedding vectors are obtained. The multiple global embedding vectors are weighted and summed through a gated multimodal fusion strategy to obtain a fused global embedding vector as a fused representation vector. The multiple global embedding vectors include global embedding vectors corresponding to visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors, respectively. The semantic reasoning task includes at least one of the following: device status diagnosis task and flight path planning task.

[0189] (v) Zero-sample transfer and cross-modal retrieval applications:

[0190] Based on the alignment and fusion methods described above, the system achieves powerful zero-shot reasoning and cross-modal retrieval capabilities.

[0191] Text-guided cross-modal retrieval: A pre-trained lightweight text encoder (such as a distilled version of BERT) is introduced. During the later stages of training, a small amount of image-text pairs, text-audio pairs, and other data are projected onto an established multimodal embedding space. Subsequently, maintenance personnel input natural language queries (such as "find 10kV tension towers" or "locate areas with corona discharge sounds"), and the system calculates the embedding vector of the query text. It then retrieves the most similar image, infrared image, or audio embedding from the unified embedding space, achieving zero-shot retrieval and localization without additional training.

[0192] Cross-modal fault association and localization: When an anomaly is detected in a certain mode (such as an acoustic sensor identifying a "corona discharge" mode), even if the model is not directly trained using "corona-visual" paired data, it can automatically retrieve and highlight the most likely corresponding faulty equipment (such as insulator damage or conductor breakage) in visible light or infrared images by calculating the similarity between the audio anomaly embedding in the shared space and other modal embeddings, thus achieving cross-modal fault association and visual localization.

[0193] Cross-modal semantic alignment is achieved through contrastive learning, and robust scene representations are generated through a hierarchical fusion mechanism. This not only solves the problem of unified processing of heterogeneous data, but also endows the system with general cross-modal understanding and zero-shot transfer capabilities similar to ImageBind. At the same time, through a carefully designed lightweight network and training strategy, its deployment feasibility on edge computing platforms such as drones is ensured.

[0194] Specifically, let's take an example:

[0195] The embodiments of the present invention are mainly used in the field of power system inspection, especially in the scenario of intelligent drone inspection of 10kV overhead lines, but the application of the present invention is not limited thereto.

[0196] Specifically, regarding the training and deployment of lightweight multimodal embedding models:

[0197] 1.1 Hardware Configuration and Data Acquisition:

[0198] The hardware platform used in this embodiment includes:

[0199] Drone platform: DJI Matrice 300 RTK drone, equipped with DJI Pilot 2 smart remote controller.

[0200] Sensor configuration:

[0201] Visible light camera: Zenmuse H20T integrated camera, resolution 1280×720, frame rate 30fps.

[0202] Infrared thermal imager: Zenmuse H20T integrated thermal imaging module, resolution 640×512, temperature measurement range -40°C to 550°C.

[0203] Acoustic sensor: USB acoustic imager, frequency range 2kHz-48kHz, sampling rate 96kHz.

[0204] LiDAR: Livox Mid-40 LiDAR, detection range 260 meters, point cloud density 240,000 points / second.

[0205] IMU: The drone has a built-in 9-axis inertial measurement unit.

[0206] GPS: RTK positioning module, with positioning accuracy down to the centimeter level.

[0207] All sensors achieve synchronous data acquisition via hardware trigger signals, with a synchronization error of less than 10ms. Data acquisition is conducted in multiple typical power inspection scenarios, including normal lines, faulty lines, and different weather conditions (daytime, nighttime, foggy weather).

[0208] 1.2 Data Preprocessing:

[0209] The collected multimodal data undergoes the following preprocessing steps:

[0210] 1. Spatiotemporal alignment: Data synchronization is based on hardware timestamps, and missing frames are filled in by interpolation.

[0211] 2. Coordinate unification: All spatial data is unified to the UAV's body coordinate system. The transformation formula is as follows:

[0212]

[0213] Where R is the rotation matrix and T is the translation vector.

[0214] 3. Normalization process:

[0215] Image data: Normalized to the range [0,1].

[0216] Infrared data: Normalized based on ambient temperature.

[0217] Acoustic data: converted to Mel spectrogram and resized to 64×64.

[0218] Point cloud data: voxelized, voxel size 0.1m.

[0219] 1.3 Lightweight encoder design:

[0220] Specific parameter configurations for each modal encoder:

[0221] (1) Visual / infrared encoder:

[0222] It adopts the MobileNetV3-Small architecture with an input size of 224×224×3.

[0223] Remove the last two fully connected layers and retain the global average pooling layer.

[0224] Output dimension: 256-dimensional feature vector.

[0225] Number of parameters: 2.5M, computational cost: 0.3GFLOPs.

[0226] (2) Acoustic encoder:

[0227] Structure: 3-layer 1D CNN + global average pooling.

[0228] Input: 64×64 Mel spectrogram.

[0229] Kernel size: [3,3,3], number of channels: [32,64,128].

[0230] Output dimension: 128-dimensional feature vector.

[0231] Number of parameters: 0.8M.

[0232] (3) Point cloud encoder:

[0233] A simplified version of PointNet is used, with an input point cloud of 1024.

[0234] Number of layers in a multilayer perceptron: [64, 128, 256].

[0235] Output dimension: 256-dimensional feature vector.

[0236] Number of parameters: 1.2M.

[0237] (4) IMU encoder:

[0238] Structure: Bidirectional LSTM, hidden layer dimension 64.

[0239] Input: 6-dimensional IMU sequence, length 50.

[0240] Output dimension: 128-dimensional feature vector.

[0241] 1.4 Cross-modal semantic alignment training:

[0242] Key parameter settings during training:

[0243] 1. Training data: Collect 10,000 sets of synchronous multimodal data samples and divide them into training set, validation set and test set in an 8:1:1 ratio.

[0244] 2. Comparison of learning settings:

[0245] Loss function: InfoNCE loss, temperature parameter τ=0.07.

[0246] Batch size: 32.

[0247] Positive samples: Any pair of modes at the same time.

[0248] Negative samples: Modal pairings at different times within a batch.

[0249] 3. Optimizer settings:

[0250] Optimizer: AdamW, initial learning rate 0.001.

[0251] Learning rate scheduling: cosine annealing, warm-up for 5 epochs.

[0252] Weight decay: 0.01.

[0253] 4. Training process:

[0254] Training epochs: 100 epochs.

[0255] Early stopping strategy: Stop if the validation set loss does not decrease after 10 epochs.

[0256] Hardware: A single NVIDIA Jetson AGX Orin edge computing platform.

[0257] Training time: Approximately 48 hours.

[0258] 5. Evaluation Indicators:

[0259] Alignment accuracy: The average similarity of correctly paired objects.

[0260] Search accuracy: Top-1 accuracy in cross-modal retrieval.

[0261] Computation latency: Inference time per frame.

[0262] 1.5 Implementation of a hierarchical integration mechanism:

[0263] Early interactive integration implementation:

[0264] class CrossModalAttention(nn.Module):

[0265] def __init__(self, dim=256, heads=4):

[0266] super().__init__()

[0267] self.visual_proj = nn.Linear(dim, dim)

[0268] self.point_proj = nn.Linear(dim, dim)

[0269] self.audio_proj = nn.Linear(128, dim)

[0270] self.attention = nn.MultiheadAttention(dim, heads, batch_first=True)

[0271] self.norm = nn.LayerNorm(dim)

[0272] def forward(self, visual_feats, point_feats, audio_feats):

[0273] # Features projected into the same space

[0274] visual_tokens = self.visual_proj(visual_feats) # [B, H W, dim]

[0275] point_tokens = self.point_proj(point_feats) # [B, N, dim]

[0276] audio_tokens = self.audio_proj(audio_feats) # [B, T, dim]

[0277] # Concatenate all tokens

[0278] all_tokens = torch.cat([visual_tokens, point_tokens, audio_tokens],dim=1)

[0279] # Cross-modal attention

[0280] attended, _ = self.attention(all_tokens, all_tokens, all_tokens)

[0281] attended = self.norm(all_tokens + attended)

[0282] return attended

[0283] Late-stage semantic fusion implementation:

[0284] class GatedFusion(nn.Module):

[0285] def __init__(self, modalities=5, dim=256):

[0286] super().__init__()

[0287] self.gate_net = nn.Sequential(

[0288] nn.Linear(dim modalities, dim 2),

[0289] nn.ReLU(),

[0290] nn.Linear(dim 2, modalities),

[0291] nn.Softmax(dim=-1) )

[0293] def forward(self, embeddings):

[0294] # embeddings: list of [B, dim] tensors

[0295] concat = torch.cat(embeddings, dim=-1) # [B, dim modalities

[0296] gates = self.gate_net(concat) # [B, modalities]

[0297] # Weighted fusion

[0298] fused = torch.zeros_like(embeddings[0])

[0299] for i, emb in enumerate(embeddings):

[0300] weight = gates[:, i].unsqueeze(-1) # [B, 1]

[0301] fused += weight emb

[0302] return fused

[0303] 1.6 Model Deployment and Optimization

[0304] 1. Model quantization:

[0305] Using INT8 quantization reduces storage usage by 75%.

[0306] Quantized model size: 35MB.

[0307] 2. Inference optimization:

[0308] TensorRT acceleration engine.

[0309] Multi-threaded parallel processing of different modalities.

[0310] Streamlined processing flow.

[0311] 3. Performance Indicators:

[0312] Single frame processing time: <100ms (meets real-time requirements).

[0313] Power consumption: <15W (suitable for drone platforms).

[0314] Memory usage: <500MB.

[0315] Specifically, for power line fault detection applications:

[0316] 2.1 Fault Detection Process:

[0317] 1. Data Acquisition: The UAV flies along the 10kV line and collects multimodal data simultaneously.

[0318] 2. Real-time inference: Edge computing units process data in real time and generate joint representations.

[0319] 3. Fault diagnosis: Fault classification based on fused features:

[0320] Normal state:

[0321] The wire overheats (temperature > 70°C);

[0322] Insulator contamination (infrared and acoustic characteristics);

[0323] Corona discharge (primarily acoustic characteristics);

[0324] Mechanical damage (visual + point cloud features).

[0325] 2.2 Cross-modal retrieval applications:

[0326] When an anomaly is detected, the system supports the following search functions:

[0327] Text-guided search:

[0328] def text_guided_retrieval(query_text, database, top_k=5):

[0329] # Text Encoding

[0330] text_embedding = text_encoder(query_text) # [1, dim]

[0331] # Calculate similarity

[0332] similarities = []

[0333] for modal, embeddings in database.items():

[0334] sim = cosine_similarity(text_embedding, embeddings)

[0335] similarities.append((modal, sim))

[0336] # Sort return

[0337] sorted_results = sorted(similarities, key=lambda x: x[1], reverse=True)

[0338] return sorted_results[:top_k]

[0339] Specific application scenarios:

[0340] 1. Maintenance personnel enter: "Find insulators with corona discharge"

[0341] 2. System response:

[0342] The most similar acoustic segments (timestamp, frequency characteristics);

[0343] The corresponding infrared image (heated area highlighted);

[0344] Visual image (insulator location markings);

[0345] Confidence score: 0.92.

[0346] 2.3 Zero-sample fault location:

[0347] The system can locate novel faults without requiring specialized training.

[0348] 1. Unknown Fault Detection: When the acoustic sensor detects an unknown abnormal mode:

[0349] Extract the audio embedding vector.

[0350] Search for similar embeddings in a unified semantic space.

[0351] Find the most similar visual / infrared sample.

[0352] The fault point is located based on the sample location information.

[0353] 2. Positioning accuracy:

[0354] Horizontal positioning error: <1 meter (RTK positioning assistance).

[0355] Fault type identification accuracy: 85.3% (under zero sample conditions).

[0356] The above optional implementation methods can achieve at least the following beneficial effects:

[0357] 1) Lightweight and energy-efficient: The encoder and fusion module, designed specifically for edge computing, significantly reduce model complexity and computational overhead, enabling complex multimodal AI models to run in real time on resource-constrained platforms such as drones.

[0358] 2) Deep semantic understanding: Cross-modal semantic alignment achieved through contrastive learning enables the model to understand the common semantics behind different perceptual data, achieving deep environmental perception and reasoning that goes beyond a single modality.

[0359] 3) Powerful zero-shot and generalization capabilities: The unified semantic embedding space enables the system to perform zero-shot cross-modal retrieval and transfer, reducing the reliance on a large amount of labeled pairing data and improving the system's adaptability to unseen scenarios or novel faults.

[0360] 4) Flexible and scalable fusion mechanism: The hierarchical fusion mechanism can dynamically select fusion strategies according to task requirements, balancing computational efficiency and perception accuracy, and providing a universal perception base for various UAV power line inspection tasks.

[0361] 5) Enhance the autonomy and intelligence of inspection: This method provides core technologies for UAVs to achieve fully autonomous line tracking, topology reconstruction, hidden danger discovery and diagnosis, which is expected to significantly improve the efficiency and reliability of power line inspection.

[0362] 6) This invention effectively solves the problems of difficult heterogeneous data fusion and limited edge computing resources in power line inspection by constructing a lightweight multimodal embedding model and achieving cross-modal semantic alignment, laying a solid technical foundation for building a new generation of intelligent UAV inspection system.

[0363] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0364] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0365] Example 2

[0366] According to embodiments of the present invention, an apparatus for implementing the above-described method for determining the fusion representation vector is also provided. Figure 3 This is a structural block diagram of a device for determining a fusion representation vector according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: an acquisition module 302, a first determination module 304, a second determination module 306, a third determination module 308, and an execution module 310. The device will be described in detail below.

[0367] The acquisition module 302 is used to acquire multimodal sensor data, which is collected by the sensor system on the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data. The first determination module 304, connected to the acquisition module 302, is used to perform multimodal encoding operations based on the multimodal sensor data to obtain multimodal embedding vectors, which include visual imaging embedding vectors, acoustic embedding vectors, point cloud embedding vectors, inertial embedding vectors, and position embedding vectors. The second determination module 306, connected to the first determination module 304, is used to input the multimodal embedding vectors into a modality alignment model to obtain semantic coreference embeddings. The alignment embedding vectors in the space, wherein the modal alignment model is trained on the initial alignment model by sample data, including a cross-modal contrastive alignment loss function, which is used to maximize the similarity between different modal embedding vectors at the same acquisition time and minimize the similarity between different modal embedding vectors at different acquisition times; a third determining module 308, connected to the second determining module 306, is used to perform feature fusion operation based on the alignment embedding vectors to obtain a fused representation vector; an execution module 310, connected to the third determining module 308, is used to execute a target task based on the fused representation vector, wherein the target task includes at least one of the following: constructing a route spatial topology map task, determining UAV flight command task.

[0368] It should be noted here that the above-mentioned acquisition module 302, first determination module 304, second determination module 306, third determination module 308 and execution module 310 correspond to steps S102 to S110 in the method for determining the fusion representation vector. The multiple modules and the corresponding steps are the same in terms of implementation instances and application scenarios, but are not limited to the content disclosed in the above embodiment 1.

[0369] Example 3

[0370] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the method for determining the fusion representation vector of any of the above embodiments.

[0371] Example 4

[0372] According to another aspect of the present invention, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method for determining the fusion representation vector described above.

[0373] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0374] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0375] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0376] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0377] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0378] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0379] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for determining a fusion representation vector, the method comprising: include: Acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV, including: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data, and position coordinate data; Based on the multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, wherein the multimodal embedding vector includes a visual imaging embedding vector, an acoustic embedding vector, a point cloud embedding vector, an inertial embedding vector, and a position embedding vector. The multimodal embedding vectors are input into the modality alignment model to obtain aligned embedding vectors in the semantic coreference embedding space. The modality alignment model is trained on the initial alignment model using sample data and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times. Based on the aligned embedding vector, a feature fusion operation is performed to obtain a fused representation vector; Based on the fused representation vector, the target task is executed, wherein the target task includes at least one of the following: constructing a route space topology map task, determining UAV flight command task.

2. The method of claim 1, wherein, Based on the aligned embedding vector, a feature fusion operation is performed to obtain a fused representation vector, including: When the target task is a fine-grained perception task, multiple feature sequences are acquired, and the interaction and fusion of these multiple feature sequences are performed through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector. The multiple feature sequences include visual feature sequences, point cloud feature sequences, and acoustic feature sequences. The visual feature sequences include spatial grid feature sequences obtained after feature extraction from the visual imaging data. The point cloud feature sequences include local geometric feature sequences of key points in the spatial depth point cloud data. The acoustic feature sequences include time-frequency block feature sequences obtained after time-frequency transformation of the acoustic signal data. The fine-grained perception task includes at least one of the following: a wire tracking task, a flight obstacle avoidance task, and a defect detection and localization task; and / or... When the target task is a semantic reasoning task, multiple global embedding vectors are obtained, and the multiple global embedding vectors are weighted and summed through a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector. The multiple global embedding vectors include the global embedding vectors corresponding to the visual imaging embedding vector, the acoustic embedding vector, the point cloud embedding vector, the inertial embedding vector, and the position embedding vector, respectively. The semantic reasoning task includes at least one of the following: device status diagnosis task and flight path planning task.

3. The method of claim 2, wherein, Multiple feature sequences are acquired, and the interaction and fusion of these multiple feature sequences are performed through a cross-modal attention mechanism to obtain a fused feature sequence as the fused representation vector, including: Obtain the multiple feature sequences; Determine the attention weights corresponding to multiple feature sequences, where the corresponding attention weights represent the degree of correlation between the target feature position in the corresponding feature sequence and the predetermined feature positions in other feature sequences; Based on the attention weights corresponding to the multiple feature sequences, an interactive fusion operation is performed through a cross-modal attention mechanism to obtain the fused feature sequence. The interactive fusion operation includes weighted fusion of feature information at each feature position in the multiple feature sequences.

4. The method of claim 2, wherein, Multiple global embedding vectors are obtained, and a weighted sum of these vectors is performed using a gated multimodal fusion strategy to obtain a fused embedding vector as the fused representation vector, including: Obtain the multiple global embedding vectors; Perform a vector concatenation operation on the multiple global embedding vectors to obtain a joint embedding vector; The joint embedding vector is input into a gating network. After the joint embedding vector is linearly transformed by the learning parameter matrix in the gating network, it is processed by a normalized exponential function to obtain the gating weights corresponding to the multiple global embedding vectors respectively. The fused embedding vector is obtained based on the plurality of global embedding vectors and the gating weights corresponding to the plurality of global embedding vectors.

5. The method of claim 1, wherein, Based on the multimodal sensor data, a multimodal encoding operation is performed to obtain a multimodal embedding vector, including: The visual imaging data is encoded using a deep separable convolutional neural network to obtain the visual imaging embedding vector; After converting the acoustic signal data into a Mel spectrogram, the Mel spectrogram is encoded using a convolutional neural network with a progressively decreasing number of channels to obtain the acoustic embedding vector. After downsampling the spatial depth point cloud data to obtain downsampled point cloud data, a point cloud processing network based on a multilayer perceptron is used to encode the downsampled point cloud data to obtain the point cloud embedding vector. The attitude inertial sequence data is encoded using a target neural network to obtain the inertial embedding vector, wherein the target neural network includes at least one of the following: a gated recurrent unit network, and a dilated convolutional temporal network; The location coordinate data is encoded using a fully connected network to obtain the location embedding vector.

6. The method of claim 1, wherein, Before inputting the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space, the process further includes: Acquire sample data and a cross-modal contrast alignment loss function, wherein the sample data includes positive sample data and negative sample data, the positive sample data includes a combination of sample embedding vectors of any two different modalities at the same acquisition time, and the negative sample data includes a combination of sample embedding vectors of any two different modalities at different acquisition times. The initial alignment model is trained by maximizing the similarity between positive sample data and minimizing the similarity between negative sample data using the cross-modal contrastive alignment loss function until the target condition is met, thereby obtaining the modal alignment model. The target condition includes at least one of the following: reaching a predetermined number of iterations, and the value of the cross-modal contrastive alignment loss function being lower than a preset loss threshold.

7. The method according to any one of claims 1 to 6, characterized in that, After inputting the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space, the process further includes: Obtain the anomaly embedding vector of the target modality, wherein the target modality is the modality in the multimodal embedding vector in which anomalies are detected, and the anomaly embedding vector is the feature vector in the target modality corresponding to the anomalous event; Determine the similarity between the anomalous embedding vector and the embedding vectors of other modalities in the semantic coreference embedding space; Based on the similarity, an associated embedding vector matching the abnormal embedding vector is determined from the embedding vectors corresponding to the other modalities; Based on the data acquisition time and spatial location corresponding to the associated embedding vector, fault association data corresponding to the other modes is output.

8. A device for determining a fusion representation vector, characterized in that, include: The acquisition module is used to acquire multimodal sensor data, wherein the multimodal sensor data is collected by the sensor system carried by the UAV and includes: visual imaging data, acoustic signal data, spatial depth point cloud data, attitude inertial sequence data and position coordinate data; The first determining module is used to perform a multimodal encoding operation based on the multimodal sensor data to obtain a multimodal embedding vector, wherein the multimodal embedding vector includes a visual imaging embedding vector, an acoustic embedding vector, a point cloud embedding vector, an inertial embedding vector, and a position embedding vector. The second determining module is used to input the multimodal embedding vector into the modality alignment model to obtain the aligned embedding vector in the semantic coreference embedding space. The modality alignment model is trained by the sample data on the initial alignment model and includes a cross-modal contrastive alignment loss function. The cross-modal contrastive alignment loss function is used to maximize the similarity between different modality embedding vectors at the same acquisition time and minimize the similarity between different modality embedding vectors at different acquisition times. The third determining module is used to perform a feature fusion operation based on the aligned embedding vector to obtain a fused representation vector; An execution module is used to execute a target task based on the fused representation vector, wherein the target task includes at least one of the following: constructing a route space topology map task, determining UAV flight command task.

9. An electronic device, comprising: include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for determining the fusion representation vector as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method for determining the fusion representation vector as described in any one of claims 1 to 7.

Images