Edge computing module supporting multi-modal data fusion processing

Abstract

The present application relates to the technical field of multi-modal data fusion processing, and more particularly to an edge computing module supporting multi-modal data fusion processing, comprising a multi-modal data acquisition module for acquiring multi-modal data and unmanned aerial vehicle motion disturbance data of a target area; a disturbance state evaluation module for determining whether to trigger compensation processing of the multi-modal data based on the unmanned aerial vehicle motion disturbance data; a multi-modal data compensation module for determining whether to trigger secondary acquisition through a key area of visible light image data, determining a core influence area of infrared thermal imaging data and performing compensation processing, and determining whether to retain laser radar point cloud data; a multi-modal fusion decision module for generating an abnormal feature fusion confidence of the multi-modal data; and a data management and communication module for uploading the multi-modal data and the fusion confidence to a central dispatch platform, and sending a data processing request and a task inquiry. The present application improves the efficiency and precision of multi-modal data fusion processing in edge computing.

Description

Technical Field

[0001] This invention relates to the field of multimodal data fusion processing technology, and in particular to an edge computing module that supports multimodal data fusion processing. Background Technology

[0002] With the continuous improvement of smart city capabilities, drone inspections have become an important means of smart city operation and maintenance. Equipped with various sensors, drones can collect multimodal data, enabling comprehensive perception of the status of critical equipment. This multi-source data acquisition method provides richer information dimensions for fault diagnosis, helping to discover potential equipment defects and improve the reliability of power grid operation.

[0003] However, existing technologies face significant challenges in practical applications. During inspections, drones are inevitably subject to environmental disturbances such as airflow and mechanical vibration, leading to a decline in the quality of the collected multimodal data. More critically, current data processing methods often neglect the impact of disturbances on multimodal data fusion, lacking dynamic assessment and adaptive compensation mechanisms for data quality, making it difficult to ensure the accuracy and reliability of diagnostic results under complex flight conditions.

[0004] Chinese Patent Publication No. CN120766068A discloses a method and system for real-time mapping data fusion of unmanned aerial vehicles (UAVs) using multimodal edge computing. The method includes: constructing a dynamic time synchronization model for multiple sensors based on sensor physical characteristics; dynamically calibrating the spatial coordinate system of multiple sensors using a kinematically constrained spatial registration algorithm based on an edge computing device; dynamically adjusting the fusion weights based on sensor confidence and environmental features using a multimodal data fusion algorithm with adaptive weight allocation; performing real-time 3D reconstruction using an edge computing device, completing feature-level fusion of point cloud and image data on the UAV; and constructing a distributed edge caching mechanism for data lifecycle management based on the LRU algorithm.

[0005] Therefore, the following problems exist in the existing technology:

[0006] In UAV inspection operations, the instability of multimodal data caused by the disturbance of the aircraft's operation was not taken into account, which resulted in low efficiency and accuracy of the edge computing process for the fusion processing of multimodal data. Summary of the Invention

[0007] To address this issue, the present invention provides an edge computing module that supports multimodal data fusion processing, thereby overcoming the problem in the prior art where the instability of multimodal data caused by the aircraft's movement during UAV inspection operations is not considered, resulting in low efficiency and accuracy in the edge computing process of multimodal data fusion processing.

[0008] To achieve the above objectives, the present invention provides an edge computing module that supports multimodal data fusion processing, comprising: a multimodal data acquisition module, which is used to acquire multimodal data of target components of a UAV during inspection operations in a target area and to acquire a number of UAV motion disturbance data, wherein the multimodal data includes visible light image data, infrared thermal imaging data and lidar point cloud data, and the UAV motion disturbance data includes the UAV's angular velocity, acceleration, speed and positioning;

[0009] The disturbance state assessment module calculates the UAV disturbance index based on real-time collected UAV motion disturbance data and UAV motion disturbance learning data to determine whether to trigger compensation processing for the multimodal data collected under the disturbance state; wherein, the UAV motion disturbance learning data is determined by clustering analysis of UAV motion disturbance data corresponding to the UAV insensitive state determined by the confidence of abnormal features of multimodal data in historical periods.

[0010] The multimodal data compensation module is used to compare the key area of ​​the acquired visible light image data with the key area of ​​the preset standard clear image data to determine whether to trigger the secondary acquisition of visible light image data, to determine the core influence area of ​​infrared thermal imaging data based on the change trajectory of UAV disturbance data and visible light image data, and to determine whether to retain the lidar point cloud data acquired under the disturbance state based on the area of ​​the core influence area.

[0011] The multimodal fusion decision module is used to extract abnormal features from the processed multimodal data and calculate the corresponding abnormal feature confidence scores of the multimodal data to generate abnormal feature fusion confidence scores.

[0012] The data management and communication module is used to upload the processed multimodal data and the confidence level of the anomaly feature fusion to the central scheduling platform, and simultaneously send multimodal data processing requests and task inquiries.

[0013] Furthermore, the disturbance state assessment module determines the UAV insensitive state based on a first preset condition of the confidence level of abnormal features in multimodal data within historical periods;

[0014] The first preset condition is that at least one of the visible light image data, infrared thermal imaging data, and lidar point cloud data collected during the disturbance state in the historical period has an abnormal feature confidence level greater than a preset confidence threshold.

[0015] Furthermore, the disturbance state assessment module determines the drone disturbance index corresponding to each drone based on the distance between the cluster center in the drone motion disturbance learning data and the real-time collected drone motion disturbance data.

[0016] And based on the fact that the UAV disturbance index is less than or equal to a preset disturbance index threshold, it is determined to trigger compensation processing for the multimodal data collected under disturbance conditions.

[0017] Furthermore, the multimodal data compensation module obtains the insensitive feature parameters of key regions of visible light image data and performs similarity calculation with the insensitive feature parameters of key regions of preset standard clear image data. If the similarity is less than the preset similarity threshold, it determines to trigger the secondary acquisition of visible light image data.

[0018] The insensitive feature parameters include the overall outline, shape features, color, and brightness distribution of the key region.

[0019] Furthermore, the multimodal data compensation module is also used to determine the vibration intensity index based on the angular velocity modulus in the UAV motion disturbance data, and to determine the exposure time for secondary acquisition based on the exponential decay relationship of the vibration intensity index.

[0020] Furthermore, the multimodal data compensation module determines the shrinkage coefficient of the motion blur region of the infrared thermal imaging data based on the ratio of the blurred contour area of ​​the key region corresponding to the first acquired visible light image data to the second acquired visible light image data, so as to determine the core influence region of the infrared thermal imaging data.

[0021] The motion-blurred region of the infrared thermal imaging data is determined by the angular velocity change trajectory of the UAV disturbance data.

[0022] Furthermore, the multimodal data compensation module determines that the lidar point cloud data collected under disturbance conditions should not be retained based on the fact that the area ratio of the core influence region is greater than a preset area ratio threshold.

[0023] Furthermore, the multimodal fusion decision module determines the fusion weights of the multimodal data based on the UAV disturbance anomaly index and the anomaly feature confidence level.

[0024] Furthermore, the multimodal fusion decision module performs fusion processing on the confidence scores of the abnormal features based on the fusion weights of the multimodal data to generate an abnormal feature fusion confidence scores.

[0025] Furthermore, when the data management and communication module determines that the lidar point cloud data collected under the disturbance state is not retained, it sends a multimodal data processing request and task solicitation to the central scheduling platform, including: a request to perform three-dimensional modeling based on the collected two-dimensional image, and a solicitation based on the confidence level of the processed multimodal data and the fusion of abnormal features to determine whether to issue an instruction to re-collect multimodal data to the target component affected by the disturbance.

[0026] Compared with existing technologies, the advantages of this invention lie in its ability to systematically guarantee data quality in complex flight environments by constructing a complete UAV disturbance state perception and multimodal data collaborative processing system. First, this invention establishes a multi-sensor synchronous acquisition mechanism to ensure the spatiotemporal uniformity of visible light, infrared thermal imaging, and lidar data, laying a solid foundation for subsequent fusion processing. The disturbance state assessment module monitors the UAV's motion state in real time, accurately identifying the compensable disturbance range and providing a basis for data quality repair decisions. The multimodal data compensation module implements differentiated processing strategies for different sensor characteristics, including intelligent resampling of visible light images, precise compensation of core affected areas of infrared thermal imaging, and key steps in determining the effectiveness of lidar point clouds. The multimodal fusion decision module, based on the quality status of the compensated data, employs an adaptive weight allocation mechanism to achieve optimal fusion results. Finally, through standardized data encapsulation and platform interaction protocols, a complete edge-cloud collaborative processing closed loop is formed. This systematic solution effectively overcomes the shortcomings of traditional inspection technology in not adequately considering the impact of machine disturbances, realizes full-process quality control from data acquisition to fault diagnosis, and improves the efficiency and accuracy of multimodal data fusion processing in edge computing.

[0027] Furthermore, this invention establishes disturbance learning data using historical valid data, providing a scientific and reasonable judgment standard for UAV disturbance state assessment. Reliable diagnostic data samples are selected from historical inspection data to ensure the quality foundation of the learning baseline. By setting reasonable confidence thresholds, data periods that still retain diagnostic value under disturbance conditions are effectively identified, providing a clear basis for the selection of learning samples. Cluster analysis is used to deeply explore the distribution patterns of disturbance data under different flight states, forming a representative disturbance pattern recognition system. By learning the disturbance characteristics under historical valid states, the system can accurately identify whether the current flight state is within an acceptable range, providing a reliable basis for subsequent data processing decisions. The dynamic baseline establishment method based on actual operational data significantly improves the accuracy of the system's disturbance state judgment under different flight conditions and environmental factors.

[0028] Furthermore, this invention achieves precise quantitative evaluation of UAV disturbance states through precise distance calculation and dynamic threshold judgment between cluster centers in UAV motion disturbance learning data and real-time collected UAV motion disturbance data. Distance calculation fully considers the inherent correlation between various motion parameters, accurately measuring the deviation between the current state and historical effective states. Comparison between the UAV disturbance index and a preset disturbance index threshold ensures the scientific nature of the disturbance index judgment, preventing both oversensitivity leading to frequent compensation and slow response causing quality issues. This refined evaluation mechanism provides accurate triggering criteria for data compensation processing, ensuring the system initiates the compensation process at the appropriate time. By establishing a scientific evaluation system, the system can effectively distinguish between acceptable disturbances and abnormal states requiring intervention, improving the overall intelligence level of the processing.

[0029] Furthermore, this invention achieves scientific judgment and intelligent decision-making regarding the quality of visible light images through a multi-feature similarity analysis-based image quality assessment mechanism. Precise image segmentation technology locates key regions, ensuring targeted assessment. A comprehensive image quality assessment system is constructed by comprehensively utilizing insensitive parameters across multiple dimensions, including contour, shape, color, and brightness features. The degree of difference between the current image and a standard image is used to provide an objective basis for quality judgment. A reasonably set similarity threshold ensures the accuracy of decision-making, effectively distinguishing between usable images and inferior images requiring re-acquisition. This systematic assessment method provides reliable technical support for secondary acquisition decisions. By establishing scientific image quality assessment standards, the system can promptly detect and address image quality problems caused by machine disturbances, ensuring the validity and reliability of inspection data.

[0030] Furthermore, this invention achieves intelligent optimization and adaptive adjustment of acquisition parameters through a dynamic correlation mechanism between vibration intensity and exposure time. The vibration intensity is accurately quantified by the angular velocity modulus, providing precise input for parameter adjustment. A mathematical relationship between vibration intensity and exposure time is established based on an exponential decay model, ensuring the scientific validity and rationality of parameter adjustments. This intelligent exposure control mechanism can automatically adopt shorter exposure times under strong vibration conditions, effectively suppressing motion blur. Through real-time dynamic parameter optimization, the system significantly improves its data acquisition capabilities in complex flight environments, ensuring the acquisition of clear images that meet diagnostic requirements under various disturbance conditions.

[0031] Furthermore, this invention achieves precise definition and effective compensation of the affected area in thermal imaging through infrared data compensation processing transmitted from visible light images. Reconstructing the lens motion trajectory using angular velocity data provides a theoretical basis for determining blurred areas. Calculating the contour area ratio using two acquired visible light images quantifies the extent of motion blur, providing an accurate reference for infrared data processing. Through shrinkage coefficient calculation and region shrinkage operations, the core affected area of ​​the infrared data is precisely located, ensuring the targeted nature of the compensation processing. This multimodal data collaborative processing method fully utilizes the complementary advantages of different sensor data, achieving accurate judgment and effective compensation of the infrared data distortion range. By establishing a scientific infrared data processing workflow, the accuracy and reliability of temperature diagnostic data are significantly improved.

[0032] Furthermore, this invention achieves intelligent quality management and precise decision-making for LiDAR point cloud data through an effective arbitration mechanism based on the core impact region area threshold. Area proportion analysis quantifies the impact of disturbances on point cloud data, providing an objective basis for decision-making. Reasonable thresholds set based on historical data ensure the scientific validity and applicability of the judgment criteria. Comparative analysis of area proportions accurately identifies low-quality point cloud data caused by severe disturbances, preventing negative impacts on subsequent analysis and avoiding the use of non-compliant data for subsequent edge computing power. This data-driven arbitration mechanism can make reasonable decisions based on actual conditions, ensuring data quality while optimizing the processing flow. By establishing strict point cloud data quality control standards, the system ensures that all data participating in the fusion analysis has sufficient reliability, maintaining the accuracy of the 3D diagnostic results. This intelligent quality management method enhances the contribution value of point cloud data in equipment status diagnosis.

[0033] Furthermore, this invention establishes an adaptive multimodal data fusion weight allocation system through a dual consideration mechanism of disturbance state and feature confidence. The basic weights are calculated based on the disturbance index, reflecting the impact of the flight environment on data quality. An exponential decay model is used to achieve a reasonable correlation between weights and the degree of disturbance, ensuring an appropriate reduction in the contribution of each modality's data under strong disturbance conditions. Weight adjustments are made based on the feature confidence of each modality, reflecting the reliability level of the data itself. This multi-level weight allocation strategy fully considers factors such as flight environment and data quality, ensuring the scientific nature and adaptability of the fusion weights. By establishing an intelligent weight allocation mechanism, the system can automatically adjust the fusion strategy under complex flight environments, significantly improving the robustness and reliability of the diagnostic results.

[0034] Furthermore, this invention achieves effective integration and optimized utilization of multi-source information through a weighted summation scientific fusion method. It obtains the confidence levels of abnormal features after compensation processing of each modality's data, ensuring the reliability of the input data quality. Based on the calculated fusion weights, it reflects the relative importance of each modality's data in the current environment. The weighted summation algorithm achieves the organic integration of multi-source information, fully leveraging the complementary advantages of each data source. This fusion method can properly handle quality differences between different modalities, ensuring that high-quality data plays a greater role in decision-making. By establishing a standardized fusion calculation process, it guarantees the consistency and comparability of the output results. This scientific fusion method considers both the technical characteristics of each modality's data and practical application needs, ensuring the accuracy and reliability of diagnostic conclusions. Through the synergistic utilization of multi-source information, it significantly improves the comprehensiveness and accuracy of equipment condition assessment.

[0035] Furthermore, this invention establishes an efficient edge-cloud collaborative working mechanism by uploading processed multimodal data and anomaly feature fusion confidence scores to a central scheduling platform, and simultaneously sending multimodal data processing requests and task inquiries. This accurately conveys the reasons for missing point cloud data and the analysis results, providing complete information for platform decision-making and ensuring the platform accurately understands the processing needs and current status at the edge. The request content is intelligently adjusted based on the point cloud data retention status, reflecting the flexibility and adaptability of the processing. Detailed task inquiry instructions provide sufficient technical basis for platform decision-making. This systematic communication mechanism effectively promotes the deep integration of edge computing and cloud intelligence, forming a complete intelligent inspection closed-loop management system. By establishing an efficient collaborative working mode, the system fully leverages the respective advantages of edge computing and the cloud platform, improving the operational efficiency and intelligence level of the entire inspection system. Attached Figure Description

[0036] Figure 1 This is a module connection diagram of an edge computing module that supports multimodal data fusion processing in an embodiment of the present invention;

[0037] Figure 2 This is a logic diagram for determining the insensitive state of a drone based on a first preset condition, according to an embodiment of the present invention.

[0038] Figure 3 This is a logic decision diagram for determining the trigger for compensating multimodal data collected under disturbance conditions in an embodiment of the present invention;

[0039] Figure 4 This is a logic decision diagram for determining the trigger for secondary acquisition of visible light image data in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0041] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0042] It should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0043] The edge computing module supporting multimodal data fusion processing described in this invention is particularly suitable for refined inspection and monitoring tasks in various complex environments within smart city management. For example, in security inspections of key areas, the module achieves all-weather, high-precision security monitoring and intrusion identification by collecting and fusing multimodal data on perimeter walls, entrances and exits as target inspection areas, and access control devices and protective nets as target components. In health inspections of municipal facilities, it can perform defect detection and deformation analysis on bridge structures, tunnel linings, and building facades as target inspection areas, and expansion joints, support anchors, and curtain wall glass as key target components. In environmental supervision inspections, it can perform precise monitoring and quantitative analysis on monitoring areas and controlled areas as target inspection areas, and sewage outlets and stockpiles as target components. In power line inspections, it can perform efficient inspections on transmission channels and tower areas as target inspection areas, and insulators and conductor joints as target components.

[0044] Please see Figure 1 The diagram shown is a module connection diagram of an edge computing module supporting multimodal data fusion processing according to an embodiment of the present invention. The edge computing module supporting multimodal data fusion processing according to an embodiment of the present invention includes:

[0045] The multimodal data acquisition module is used to acquire multimodal data of target components during UAV inspection operations in the target area and to obtain UAV motion disturbance data. The multimodal data includes visible light image data, infrared thermal imaging data and lidar point cloud data. The UAV motion disturbance data includes the UAV's angular velocity, acceleration, speed and positioning.

[0046] The disturbance state assessment module calculates the UAV disturbance index based on real-time collected UAV motion disturbance data and UAV motion disturbance learning data to determine whether to trigger compensation processing for the multimodal data collected under the disturbance state; wherein, the UAV motion disturbance learning data is determined by clustering analysis of UAV motion disturbance data corresponding to the UAV insensitive state determined by the confidence of abnormal features of multimodal data in historical periods.

[0047] The multimodal data compensation module is used to compare the key area of ​​the acquired visible light image data with the key area of ​​the preset standard clear image data to determine whether to trigger the secondary acquisition of visible light image data, and to determine the core influence area of ​​infrared thermal imaging data based on the change trajectory of UAV disturbance data and visible light image data and perform compensation processing, and to determine whether to retain the lidar point cloud data acquired under the disturbance state based on the area of ​​the core influence area.

[0048] The multimodal fusion decision module is used to extract abnormal features from the processed multimodal data and calculate the corresponding abnormal feature confidence scores of the multimodal data to generate abnormal feature fusion confidence scores.

[0049] The data management and communication module is used to upload the processed multimodal data and the confidence level of the anomaly feature fusion to the central scheduling platform, and simultaneously send multimodal data processing requests and task inquiries.

[0050] It is understood that each drone participating in the inspection of the target area of ​​the smart city is a node of the edge computing in the embodiment of the present invention.

[0051] In this embodiment, the multimodal data acquisition module simultaneously activates the visible light camera, infrared thermal imager, and lidar to collect visible light image data, infrared thermal imaging data, and lidar point cloud data of the target component at a unified timestamp. At the same time, it continuously acquires the angular velocity, acceleration, speed, and positioning information of the UAV through the IMU and GNSS receiver.

[0052] Understandably, the target components are the key vulnerable parts in the inspection area. For example, in the inspection of power lines, the target components include, but are not limited to, insulators, conductor joints, surge arresters, tower hardware, and cable terminations, as well as other key vulnerable components in the power lines. The operating status of these components directly affects the safety and stability of the power lines and is the core focus of the inspection.

[0053] Understandably, visible light image data, which is RGB format image data acquired by a high-definition camera, is used to identify physical defects on the surface of components, such as cracks, damage, and dirt; infrared thermal imaging data, which is temperature matrix data acquired by a thermal imager, is used to detect abnormal heat points on components; and lidar point cloud data, which is a set of three-dimensional coordinate points acquired by lidar, is used to measure the geometric parameters and safe distances of components. The IMU and GNSS are the Inertial Measurement Unit and the Global Navigation Satellite System, respectively, used to provide the UAV's motion status and precise positioning information.

[0054] Please see Figure 2 As shown, it is a logic determination diagram of the unsensitive state of the UAV based on the first preset condition in an embodiment of the present invention.

[0055] Specifically, the disturbance state assessment module determines the UAV insensitive state based on a first preset condition of the confidence level of abnormal features in multimodal data within historical periods;

[0056] The first preset condition is that at least one of the visible light image data, infrared thermal imaging data, and lidar point cloud data collected during the disturbance state in the historical period has an abnormal feature confidence level greater than a preset confidence threshold.

[0057] In this embodiment, when constructing UAV motion disturbance learning data, data meeting a first preset condition is selected from the multimodal data set collected during UAV motion disturbances within historical periods. The first preset condition requires that at least one of the visible light image data, infrared thermal imaging data, and lidar point cloud data has an anomaly feature confidence level greater than a preset confidence threshold. Preferably, the confidence threshold is set to 0.7. It is understood that data meeting the first preset condition represents a state where the UAV is insensitive to disturbances during motion disturbances; the collected multimodal data is not completely abnormal or invalid and still has value for compensation and analysis.

[0058] In this embodiment, the anomaly feature confidence score is used to quantify the credibility of anomaly features extracted from each modality of data. Its calculation is based on existing deep learning and pattern recognition technologies or any other existing method. For example, it can be achieved by inputting the extracted deep feature vectors into a pre-trained fully connected neural network classifier. The output layer of this classifier uses a sigmoid activation function, mapping the output value to the (0,1) interval. This value is the anomaly feature confidence score of the corresponding modality of data. The closer the confidence score is to 1, the higher the probability that the modality of data is anomaly; the closer it is to 0, the higher the probability that it is normal. The classifier is trained based on historical multimodal datasets labeled "normal" and "abnormal," which is a mature existing technology and will not be elaborated further here.

[0059] The UAV motion disturbance data corresponding to these multimodal data with compensation value, including angular velocity, acceleration, velocity, and positioning, are extracted. K-means clustering is used to analyze this disturbance data. The optimal number of clusters is determined according to the elbow rule, and the center point and boundary range of each cluster are calculated to form a representative UAV motion disturbance learning dataset, providing an accurate reference benchmark for real-time disturbance assessment. The process of determining the optimal number of clusters includes: calculating the sum of squares within each cluster under different numbers of clusters, and selecting the number of clusters corresponding to the inflection point as the optimal value; the boundary range of the clusters is set as a hyperspherical region with a radius of 3 times the standard deviation centered on the cluster center.

[0060] Please see Figure 3 As shown, it is a logic decision diagram for determining the trigger to perform compensation processing on multimodal data collected under disturbance conditions in an embodiment of the present invention.

[0061] Specifically, the disturbance state assessment module determines the drone disturbance index corresponding to each drone based on the distance between the cluster center in the drone motion disturbance learning data and the real-time collected drone motion disturbance data.

[0062] And based on the fact that the UAV disturbance index is less than or equal to a preset disturbance index threshold, it is determined to trigger compensation processing for the multimodal data collected under disturbance conditions.

[0063] In this embodiment, the disturbance state assessment module receives UAV motion data from IMU and GNSS in real time, including three-axis angular velocity, three-axis acceleration, three-dimensional velocity, and precise positioning information. The disturbance state assessment module calculates the Mahalanobis distance between the real-time collected UAV motion disturbance data and the cluster centers in the UAV motion disturbance learning dataset. It is understood that the Mahalanobis distance calculation fully considers the covariance structure of the dataset and can accurately reflect the deviation of the current motion state from the historical effective state. The system takes the minimum value among all Mahalanobis distances as the UAV disturbance index; the smaller the index, the closer the current state is to the historical effective state. When the disturbance index is less than or equal to the disturbance index threshold, the module determines that the current disturbance is within an acceptable range and triggers the compensation processing flow for multimodal data. Preferably, the disturbance index threshold is set to the 95th percentile of the Mahalanobis distance between the historically insensitive UAV motion disturbance data and the cluster centers in the UAV motion disturbance learning dataset. This effectively distinguishes between compensable and unrecoverable disturbances, ensuring that the compensation mechanism is activated at the appropriate time.

[0064] Please see Figure 4 As shown, it is a logic decision diagram for determining the trigger for secondary acquisition of visible light image data in an embodiment of the present invention.

[0065] Specifically, the multimodal data compensation module obtains the insensitive feature parameters of key regions of visible light image data and performs similarity calculation with the insensitive feature parameters of key regions of preset standard clear image data. If the similarity is less than the preset similarity threshold, it determines to trigger the secondary acquisition of visible light image data.

[0066] The insensitive feature parameters include the overall outline, shape features, color, and brightness distribution of the key region.

[0067] It is understandable that the key areas of the target component include the outline area of ​​the insulator disc, the connection part of the conductor splice tube, and the suspension point of the vibration damper. These can all be set as key parts of the target component, and can be set according to the specific scenario, which will not be elaborated here.

[0068] In this embodiment, after receiving the quality assessment instruction, the multimodal data compensation module first segments the key region from the current visible light image, and then extracts the insensitive feature parameters of the region, including the overall contour described by the contour perimeter and the number of inflection points; shape features described by circularity and rectangularity; color described by the RGB mean; and brightness distribution described by the variance of the grayscale histogram. A preset standard clear image, i.e., an image of the same component acquired under undisturbed conditions, is called, and the above-mentioned insensitive feature parameters of the same key region are extracted. The cosine similarity algorithm is used to calculate the similarity between the two sets of parameters. When the similarity calculation result is less than a preset similarity threshold, the module determines that the key region of the current visible light image is severely blurred and cannot meet the detection requirements, and immediately triggers a secondary acquisition process of visible light image data. Preferably, the similarity threshold is set to 0.75. This mechanism ensures that re-acquisition is only initiated when the image quality is indeed damaged, avoiding unnecessary acquisition operations.

[0069] In this embodiment, the multimodal data compensation module is also used to determine the exposure time for secondary acquisition based on the vibration intensity of the UAV disturbance data;

[0070] Specifically, the multimodal data compensation module is also used to determine the vibration intensity index based on the angular velocity modulus in the UAV motion disturbance data, and to determine the exposure time for secondary acquisition based on the exponential decay relationship of the vibration intensity index.

[0071] In this embodiment, after triggering the secondary acquisition, the multimodal data compensation module first calculates the modulus of the current angular velocity vector as an indicator of vibration intensity. This indicator comprehensively reflects the angular vibration intensity of the UAV in three axes. The module calculates the exposure time for the secondary acquisition based on the exponential decay relationship between the vibration intensity indicator and exposure time. The formula is: Secondary Exposure Time = Base Exposure Time × e^(-k × Modulus). The base exposure time is set to 1 / 500 second, the unit of the angular velocity modulus is degrees / second, the formula ensures that the exponent is dimensionless, and k is the decay coefficient set to 0.02 seconds / degree, which physically represents the attenuation intensity per unit angular velocity to the exposure time. As the vibration intensity indicator increases, the exposure time decreases exponentially, ensuring that a shorter exposure time is used to "freeze" the image under strong vibration conditions. For example, when the modulus is 10° / s, the secondary exposure time = 1 / 500 × e^(-0.02 × 10) ≈ 1 / 550s. This series of coordinated operations ensures that clear images meeting diagnostic requirements can still be acquired even in disturbed environments.

[0072] Specifically, the multimodal data compensation module determines the shrinkage coefficient of the motion blur region of the infrared thermal imaging data based on the ratio of the blurred contour area of ​​the key region corresponding to the first acquired visible light image data to the second acquired visible light image data, so as to determine the core influence region of the infrared thermal imaging data.

[0073] The motion-blurred region of the infrared thermal imaging data is determined by the angular velocity change trajectory of the UAV disturbance data.

[0074] In this embodiment, the multimodal data compensation module first determines the motion blur region of the infrared thermal imaging data based on the angular velocity change trajectory in the UAV disturbance data, i.e., the curves of the change of angular velocity along the x, y, and z axes over time, using a fuzz kernel calculation based on the assumption of uniform linear motion. This region is the area where temperature distribution distortion may occur due to UAV vibration. The fuzz kernel calculation based on the assumption of uniform linear motion to determine the motion blur region is existing technology and will not be described in detail here. Then, the contour area of ​​the motion blur region in the key area of ​​the first acquired visible light image and the blurred contour area of ​​the same key area in the second acquired visible light image are extracted. It can be understood that the second acquired image has reduced blur due to exposure adjustment. The ratio of the blurred contour area of ​​the key area in the first acquired visible light image to the blurred contour area of ​​the same key area in the second acquired visible light image is determined as the shrinkage coefficient. The motion-blurred region of the infrared thermal image is shrunk according to a shrinkage factor. Taking the geometric center of the motion-blurred region as a fixed point, the boundary of the region is uniformly scaled radially towards the center according to the shrinkage factor. For example, if the original blurred region is 100 pixels and the shrinkage factor is 0.6, it will be shrunk to 60 pixels. The shrunk region is the core influence region of the infrared thermal image data, that is, the region most likely to have temperature distortion.

[0075] A bilinear interpolation algorithm is used to compensate for the core affected area, and temperature data from the surrounding unblurred area is used to correct the distortion, ensuring that the infrared thermal imaging data accurately reflects the temperature status of the component. The bilinear interpolation algorithm is existing technology and will not be described in detail here.

[0076] Specifically, the multimodal data compensation module determines that the lidar point cloud data collected under disturbance conditions should not be retained based on the fact that the area ratio of the core affected area is greater than a preset area ratio threshold.

[0077] In this embodiment, after determining the core influence area of ​​the infrared thermal imaging data, the multimodal data compensation module calculates the area of ​​this area and compares it with the average standard reference area of ​​the key component acquired under undisturbed conditions within a historical period to calculate the area ratio. The standard reference area refers to the average pixel area occupied by the outline of the key region in clear images obtained from multiple acquisitions of the key region of the same target component under undisturbed conditions within a historical period. Preferably, the area ratio threshold is set to 0.3. When the area ratio of the core influence area is greater than or equal to the preset area ratio threshold, the module determines that the current disturbance has seriously affected the lidar acquisition, and even after motion distortion correction, the accuracy of the point cloud data is difficult to guarantee. Therefore, it makes the decision to discard the point cloud data of the current frame. This mechanism ensures that only reliable point cloud data can enter the subsequent processing flow, avoids the negative impact of low-quality data on the fusion diagnostic results, maintains the reliability of the system output, and improves the efficiency of data processing and fusion.

[0078] In this embodiment, after compensating the visible light image data and infrared thermal imaging data, and determining whether to retain the lidar point cloud data, anomaly feature extraction is performed on the multimodal data. This can be done using any method in the prior art, such as using a trained deep learning model to extract features from each modality. Specifically, a CNN network is used to extract texture features from the visible light image, a thermal feature extraction network is used to extract temperature distribution features from the infrared thermal imaging, and a PointNet++ network is used to extract geometric features from the lidar point cloud. The anomaly feature confidence level of the corresponding multimodal data is calculated, which can also be determined using any method in the prior art. For example, the extracted features are input into a classifier, and the output is mapped to a confidence value between 0 and 1 using a sigmoid function, indicating the possibility that the modality data is anomaly-prone.

[0079] It is understandable that the abnormal features in visible light image data mainly include cracks, missing parts, broken parts, rust, corrosion, and deformation of key vulnerable components in the inspection area; the abnormal features in infrared thermal imaging data mainly include temperature abnormalities of key vulnerable components in the inspection area; and the abnormal features in lidar point cloud data include abnormal pose and bending deformation of key vulnerable components in the inspection area.

[0080] For example, in power line inspections, abnormal features in visible light image data mainly include: radial or circumferential cracks in insulator discs, missing or damaged insulator strings, rust and corrosion of metal components, broken or scattered strands on the conductor surface, deformation and displacement of grading rings, slippage and detachment of vibration dampers, and structural defects such as bird nests or other foreign objects hanging from the insulator surface; these also include abnormal conditions such as large areas of dirt on the insulator surface and loose or missing bolts on hardware. These features are identified by analyzing the visual characteristics of image texture changes, edge continuity, and shape integrity.

[0081] For example, in power line inspections, abnormal features in infrared thermal imaging data mainly include: localized overheating caused by poor contact of conductor splices or tension clamps; abnormal temperature rise of zero or low-value insulators due to insulator deterioration; temperature anomalies caused by internal faults in equipment such as surge arresters or instrument transformers; increased resistance and heat generation at electrical connection points due to oxidation or loosening; and abnormal temperature distribution caused by uneven line load. These anomalies manifest as localized hot spots in the temperature field, abnormal temperature gradients, and excessive interphase temperature differences.

[0082] For example, in power line inspections, anomalous features in lidar point cloud data mainly include: conductor sag exceeding safety limits, insufficient safe distance between conductors and trees or buildings, abnormal posture caused by tower tilting or foundation settlement, bending deformation of insulator strings, abnormal spatial trajectory caused by conductor galloping, and geometric features of newly added obstacles in the line corridor. These anomalies are detected by analyzing the spatial distribution, density changes, and three-dimensional geometric features of the point cloud.

[0083] Specifically, the multimodal fusion decision module determines the fusion weights of the multimodal data based on the UAV disturbance anomaly index and the anomaly feature confidence level.

[0084] In this embodiment, the multimodal fusion decision module first calculates the basic weights of each modality based on the UAV disturbance index. The weight calculation adopts an exponential decay model with a decay coefficient of 0.5 to ensure that the basic weights of each modality decrease accordingly as the disturbance index increases. The module simultaneously obtains the confidence scores output after anomaly feature extraction for each modality, and multiplies the basic weights by the confidence scores to obtain the preliminary weight allocation. Therefore, the formula for calculating the fusion weight of each multimodal data is: Fusion weight of each multimodal data = [e^(-0.5 × disturbance index)] × confidence score of each multimodal data. Finally, all weights are normalized to make the sum equal to 1. In one specific embodiment, the disturbance anomaly index is 2.0, the confidence level of visible light anomaly features is 0.8, and the visible light fusion weight is [e^(-0.5×2.0)]×0.8≈0.29; the confidence level of infrared thermal imaging data anomaly features is 0.9, and the infrared thermal imaging data fusion weight is [e^(-0.5×2.0)]×0.9≈0.33; lidar point cloud data has been discarded and is not included in the calculation. This dual weight determination mechanism based on disturbance state and confidence level ensures the adaptability and reliability of the fusion strategy in complex flight environments.

[0085] Specifically, the multimodal fusion decision module performs fusion processing on the confidence scores of the abnormal features based on the fusion weights of the multimodal data to generate an abnormal feature fusion confidence scores.

[0086] In this embodiment, after obtaining the fusion weights of each modality's data, the multimodal fusion decision module acquires the confidence level of the abnormal features corresponding to each modality's data. A weighted summation algorithm is used to calculate the fusion confidence level of the abnormal features, with the formula: Fusion Confidence Level = Σ (Fusion weights of each multimodal data × Confidence level of the abnormal features of each multimodal data). Through fusion processing, the confidence levels of each modality's data after data compensation are combined, providing data support for subsequent feedback to the central scheduling platform.

[0087] Specifically, when the data management and communication module determines that the lidar point cloud data collected under the disturbance state is not retained, it sends a multimodal data processing request and task solicitation to the central dispatch platform, including: a request to perform three-dimensional modeling based on the collected two-dimensional image, and a solicitation based on the confidence level of the processed multimodal data and the fusion of abnormal features to determine whether to issue an instruction to re-collect multimodal data to the target component affected by the disturbance.

[0088] In this embodiment, after other modules have completed all data processing, the data management and communication module generates a data packet from the two acquired visible light images, the compensated infrared thermal imaging data marked with the core influence area, and the confidence score of the anomaly feature fusion. When the system determines that the lidar point cloud data acquired under the disturbed state should not be retained, the reason for the missing point cloud data and the analysis results of the core influence area are clearly marked in the data packet. The data packet is then sent to the central dispatch platform, along with a request for 3D modeling based on the acquired 2D images. This can be described as "requesting spatial registration and mapping based on the uploaded 2D multimodal data and the existing 3D digital twin model." When the system determines that the lidar point cloud data should be retained, the lidar point cloud data is synchronously added to the data packet and sent to the central dispatch platform. The request for 3D modeling based on the acquired 2D images is no longer sent synchronously. At the same time, a task solicitation is sent. The solicitation platform requests the central dispatch platform to clarify whether it is necessary to issue a re-acquisition command for the target component affected by the disturbance or to use the currently acquired data, based on the processed multimodal data and the confidence score of the anomaly feature fusion.

[0089] This embodiment also provides an edge computing module supporting multimodal data fusion processing. This module adopts an embedded hardware architecture, with core components including a multi-core processor, a dedicated AI acceleration chip, multiple sensor interfaces, and a communication module, which can be directly integrated into the UAV. The module integrates a multimodal data acquisition module, a disturbance state assessment module, a multimodal data compensation module, a multimodal fusion decision module, and a data management and communication module, forming a complete edge-side intelligent processing system. The module connects to various UAV subsystems through standard interfaces: the visible light camera and infrared thermal imager connect to the multimodal data acquisition module via MIPI CSI-2 and GigE interfaces; the lidar connects via an Ethernet interface; the IMU / GNSS communicates via SPI / UART bus and interacts with the flight control system via a CAN bus. The acquisition module synchronizes all sensor data based on GPS timestamps or a local high-precision clock (RTC, accuracy ±5ppm), with a synchronization error ≤1ms, ensuring the consistency of multimodal data in the spatiotemporal dimensions and laying the foundation for subsequent fusion processing. The disturbance state assessment module calculates the UAV disturbance index in real time based on the collected motion data. The multimodal data compensation module performs intelligent compensation processing on the affected visible light image and infrared thermal imaging data according to the assessment results. The multimodal fusion decision module performs feature extraction and confidence fusion on the data after each mode compensation. Finally, the data management and communication module uploads the processing results to the central dispatch platform through the 5G network.

[0090] The module integrates multiple communication interfaces, including a 5G module (via M.2 interface, supporting SA / NSA dual-mode, latency <20ms), a Wi-Fi 6 module (supporting IEEE 802.11ax, speed 1.2Gbps), and a LoRa module (via SPI interface, supporting LoRaWAN protocol, transmission distance 5km), adapting to different scenario requirements. All communications support TLS 1.3 encrypted transmission to ensure data security.

[0091] The module also includes a power and management unit to ensure stable operation and optimize energy consumption, making it particularly suitable for long-term operation of mobile platforms such as drones. Dynamic power consumption is achieved by using a PMIC (Power Management Component, such as the TI TPS65987) to dynamically adjust the supply voltage and current according to the load of heterogeneous computing units. For example, when idle, the CPU frequency is reduced to 1.0GHz and the NPU is put into hibernation, reducing the module power consumption from 8W at full load to 1.5W, significantly improving energy efficiency. Fault self-recovery is achieved through a built-in voltage and temperature monitoring module, which automatically triggers frequency reduction or restart when overvoltage or overtemperature is detected to avoid module damage. Remote management supports remote wake-up and scheduled power on / off via cloud commands, adapting to unattended inspection scenarios.

[0092] The module adopts an embedded design, with core components including a multi-core processor (such as an ARM Cortex-A55 quad-core processor with a main frequency of 2.0GHz), a dedicated AI acceleration chip (NPU, computing power 8TOPS@INT8), a GPU (such as an ARM Mali-G52), and a DSP (such as a TMS320C6748). It dynamically allocates computing resources through a heterogeneous computing scheduling algorithm. The module is directly integrated inside the UAV and interacts with the flight control system through a standard interface (such as a CAN bus). It achieves real-time acquisition of multimodal data, disturbance status monitoring, data quality assessment, adaptive compensation, and intelligent fusion diagnosis. By establishing a disturbance learning baseline, dynamic weight fusion, and a closed-loop quality control mechanism, it effectively overcomes data quality problems caused by airframe disturbances, outputs accurate equipment status assessment results, and maintains efficient data interaction with the central scheduling platform through the collaborative work of each module.

[0093] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An edge computing module supporting multimodal data fusion processing, characterized in that, include: The multimodal data acquisition module is used to acquire multimodal data of target components during UAV inspection operations in the target area and to obtain UAV motion disturbance data. The multimodal data includes visible light image data, infrared thermal imaging data and lidar point cloud data. The UAV motion disturbance data includes the UAV's angular velocity, acceleration, speed and positioning. The disturbance state assessment module calculates the UAV disturbance index based on real-time collected motion disturbance data of each UAV and UAV motion disturbance learning data to determine whether to trigger compensation processing of the multimodal data collected under the disturbance state. The drone motion disturbance learning data is determined by clustering analysis of drone motion disturbance data corresponding to drone insensitive states, which is determined by the confidence level of abnormal features of multimodal data within historical periods. The multimodal data compensation module is used to compare the key area of ​​the acquired visible light image data with the key area of ​​the preset standard clear image data to determine whether to trigger the secondary acquisition of visible light image data, to determine the core influence area of ​​infrared thermal imaging data based on the change trajectory of UAV disturbance data and visible light image data, and to determine whether to retain the lidar point cloud data acquired under the disturbance state based on the area of ​​the core influence area. The multimodal fusion decision module is used to extract abnormal features from the processed multimodal data and calculate the corresponding abnormal feature confidence scores of the multimodal data to generate abnormal feature fusion confidence scores. The data management and communication module is used to upload the processed multimodal data and the confidence level of the anomaly feature fusion to the central scheduling platform, and simultaneously send multimodal data processing requests and task inquiries.

2. The edge computing module supporting multimodal data fusion processing according to claim 1, characterized in that, The disturbance state assessment module determines the UAV insensitive state based on a first preset condition of the confidence level of abnormal features in multimodal data within historical periods. The first preset condition is that at least one of the visible light image data, infrared thermal imaging data, and lidar point cloud data collected during the disturbance state in the historical period has an abnormal feature confidence level greater than a preset confidence threshold.

3. The edge computing module supporting multimodal data fusion processing according to claim 2, characterized in that, The disturbance state assessment module determines the drone disturbance index corresponding to each drone based on the distance between the cluster center in the drone motion disturbance learning data and the real-time collected drone motion disturbance data. And based on the fact that the UAV disturbance index is less than or equal to a preset disturbance index threshold, it is determined to trigger compensation processing for the multimodal data collected under disturbance conditions.

4. The edge computing module supporting multimodal data fusion processing according to claim 3, characterized in that, The multimodal data compensation module obtains the insensitive feature parameters of key regions of visible light image data and calculates the similarity between them and the insensitive feature parameters of key regions of preset standard clear image data. If the similarity is less than the preset similarity threshold, it determines to trigger the secondary acquisition of visible light image data. The insensitive feature parameters include the overall outline, shape features, color, and brightness distribution of the key region.

5. The edge computing module supporting multimodal data fusion processing according to claim 4, characterized in that, The multimodal data compensation module is also used to determine the vibration intensity index based on the angular velocity modulus in the UAV motion disturbance data, and to determine the exposure time for secondary acquisition based on the exponential decay relationship of the vibration intensity index.

6. The edge computing module supporting multimodal data fusion processing according to claim 5, characterized in that, The multimodal data compensation module determines the shrinkage coefficient of the motion blur region of the infrared thermal imaging data based on the ratio of the blurred contour area of ​​the key region corresponding to the first acquired visible light image data to the second acquired visible light image data, so as to determine the core influence region of the infrared thermal imaging data. The motion-blurred region of the infrared thermal imaging data is determined by the angular velocity change trajectory of the UAV disturbance data.

7. The edge computing module supporting multimodal data fusion processing according to claim 6, characterized in that, The multimodal data compensation module determines that the lidar point cloud data collected under disturbance conditions will not be retained if the area ratio of the core affected region is greater than a preset area ratio threshold.

8. The edge computing module supporting multimodal data fusion processing according to claim 7, characterized in that, The multimodal fusion decision module determines the fusion weights of the multimodal data based on the UAV disturbance anomaly index and the anomaly feature confidence level.

9. The edge computing module supporting multimodal data fusion processing according to claim 8, characterized in that, The multimodal fusion decision module performs fusion processing on the confidence scores of the abnormal features based on the fusion weights of the multimodal data to generate an abnormal feature fusion confidence scores.

10. The edge computing module supporting multimodal data fusion processing according to claim 1, characterized in that, When the data management and communication module determines that the lidar point cloud data collected under the condition of not retaining disturbance is to send a multimodal data processing request and task solicitation to the central scheduling platform, including: The system requests 3D modeling based on acquired 2D images and inquires whether to issue an instruction to reacquire multimodal data for the affected target component based on the confidence level of the processed multimodal data and anomaly feature fusion.

Images