Unmanned aerial vehicle intelligent equipment detection and fault diagnosis method based on computer vision

Abstract

The invention discloses an unmanned aerial vehicle intelligent equipment detection and fault diagnosis method based on computer vision, and the method comprises the following steps: 1, obtaining multi-modal image data and synchronous sensor data of key parts of an unmanned aerial vehicle, and carrying out the preprocessing of the data; 2, constructing a component state topological graph at the current observation moment; 3, performing local disturbance enhancement operation to generate a multi-scale disturbance image set; 4, extracting physical response vectors to form a component physical response vector set; step 5, on the basis of a Gromov-Wasserstein distance, an evolution path of optimal matching is determined; step 6, fault inversion simulation is carried out, a push-back state image is generated, and a structure residual error between the push-back state image and the initial state image at the current observation moment is calculated; and step 7, generating a natural language diagnosis report based on the optimal matching evolution path. According to the invention, through image disturbance and virtual fault evolution, fault evolution simulation and accurate diagnosis of the unmanned aerial vehicle are realized.

Description

Technical Field

[0001] This invention relates to the field of data analysis and intelligent diagnostic technology, and in particular to a method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision. Background Technology

[0002] In the current field of intelligent operation and maintenance and equipment management for unmanned aerial vehicles (UAVs), the detection and fault diagnosis of key components generally rely on single-modal image recognition or threshold-based sensor monitoring mechanisms. These methods often employ traditional computer vision algorithms, such as image recognition techniques based on edge detection, template matching, or shallow feature extraction. Their recognition capabilities are limited by changes in image quality under complex environments. Alternatively, they may set fixed thresholds to trigger alarms for exceeding limits in sensor data such as stress, temperature, and acceleration, lacking in-depth modeling and understanding of the actual operating state and evolutionary process of the equipment. This isolated data processing approach not only fails to comprehensively capture the potential fault evolution mechanisms of the equipment, but also significantly reduces diagnostic accuracy under conditions of multiple coupled factors, strong environmental disturbances, or ambiguous initial fault characteristics, easily leading to false alarms or missed alarms. Furthermore, traditional methods typically focus on static detection and current state analysis, lacking the ability to predict the evolution trend of equipment state, making it difficult to achieve early intervention for future risks and reverse tracing of the fault process.

[0003] In existing research, some scholars have attempted to introduce deep learning models for classification and recognition of image or sensor data. However, these methods are often limited to feature mapping at a single time step, lacking modeling of cross-time evolution paths. They also lack mechanisms to fuse multimodal data into a unified topological structure, resulting in low utilization of multi-source information. Furthermore, when multiple fault possibilities coexist, existing methods struggle to establish a formalized structural comparison mechanism to find fault paths that best approximate the actual evolutionary process. This is especially true when dealing with the mapping relationship between complex topological changes and high-dimensional data, where existing algorithms lack robustness and interpretability.

[0004] Therefore, how to provide computer vision-based methods for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a computer vision-based method for detecting and diagnosing intelligent devices for unmanned aerial vehicles (UAVs). This invention integrates multimodal image perturbation enhancement and sensor physical modeling to construct a state topology diagram and fault evolution path. It achieves structural comparison through Gromov-Wasserstein distance and combines backtracking simulation and natural language generation to realize UAV fault type identification, severity assessment, and visualization of diagnostic results.

[0006] The computer vision-based method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) according to embodiments of the present invention includes the following steps: Step 1: Acquire multimodal image data and synchronous sensor data of key components of the UAV, perform preprocessing on each, and generate multimodal image data and synchronous sensor data with unified structure; Step 2: Based on the unified structure of multimodal image data and unified structure of synchronous sensor data, construct the component state topology diagram at the current observation time; Step 3: Divide the multimodal image data with a unified structure into layers according to the image region size, and perform local perturbation enhancement operations to generate a multi-scale perturbation image set; Step 4: Based on the synchronous sensor data and component material parameters of the unified structure, extract the physical response vector to form a set of component physical response vectors; Step 5: Based on the multi-scale perturbation image set and the physical response vector set, perform virtual fault evolution to form a set of topology evolution diagrams, and perform structural comparison between the component state topology diagram at the current observation time and the topology evolution diagrams of each component in the set of topology evolution diagrams based on the Gromov-Wasserstein distance to determine the optimal matching evolution path; Step 6: Perform fault inversion simulation on the optimal matching evolution path to generate a backtracked state image, and calculate the structural residual between the backtracked state image and the initial state image at the current observation time. Step 7: Generate a natural language diagnostic report based on the optimal matching evolution path.

[0007] Optionally, step one specifically includes: The key components of the UAV include the rotor assembly, motor, arm, battery compartment, flight control sensors, and GPS communication module area, and each key component is numbered. The multimodal image data includes RGB images, infrared images, and structured light images; The synchronous sensor data includes stress, acceleration, vibration frequency, and temperature information; The preprocessing steps include: Image size standardization, spatial coordinate registration, bilateral filtering for noise reduction, and timestamp alignment are performed on multimodal image data to generate multimodal image data with a unified structure. The synchronous sensor data is timestamped, outliers are removed, and normalized to generate synchronous sensor data with a unified structure.

[0008] Optionally, step two specifically includes: Visual feature vectors are extracted from the preprocessed multimodal image data with a unified structure using Sobel edge detection. The visual feature vectors include edge direction, texture distribution, surface reflectivity, and regional crack boundaries. Physical state vectors are extracted from preprocessed, unified structured synchronous sensor data. These physical state vectors include stress fluctuation values, acceleration vectors, temperature gradients, and vibration spectra. The visual feature vector and physical state vector corresponding to each key component are concatenated according to the numerical dimension, and Z-score normalization is performed to generate a unified node state vector. Each key component is used as a node in the component state topology graph, and the node state vector is the attribute vector of the corresponding node. Based on the actual physical connection relationship of each key component of the UAV, a directed edge of the component state topology graph is constructed. The direction of the directed edge is from the force source component to the force receiving component. For each directed edge, the angle between the acceleration vectors of the two connected components at the current moment is calculated as a weight value.

[0009] Optionally, step three specifically includes: Multimodal image data with a unified structure is hierarchically divided according to image region size to construct image patch sets of three scales. The construction steps are as follows: Image regions smaller than 32×32 pixels are defined as microscale image blocks to characterize crack edges, hole initiation points, and micro-damage areas. Image regions with sizes between 32×32 and 96×96 pixels are defined as mesoscale image blocks to characterize local deformation, edge warping, and loosening of spiral connection components. Image regions with a size greater than 96×96 pixels are defined as macro-scale image blocks, used to characterize overall anomalies such as structural tilt, morphological deviation, and component detachment. Local perturbation enhancement operations are performed on image patches of each scale, specifically including: Perform affine transformation and Gaussian perturbation on microscale image patches; Perform brightness perturbation and contrast stretching on mesoscale image patches; Perform random rotation, cropping, and edge occlusion processing on macroscopic image patches; Image patches processed at different scales and with different perturbation methods are combined to form a multi-scale perturbation image set.

[0010] Optionally, step four specifically includes: Based on a pre-set component material library, material parameter information corresponding to each key component of the UAV is obtained. The material parameter information includes elastic modulus, thermal conductivity, mass density, specific heat capacity and Poisson's ratio. The material parameters and synchronous sensor data of the unified structure are aligned with the timestamp to construct a joint data pair for each key component at the current observation time. The joint data pair includes stress-elastic modulus relationship, temperature-thermal conductivity relationship, acceleration-mass density relationship, and vibration frequency-elastic modulus and mass density relationship. Based on the stress-elastic modulus relationship, and combining the stress measurement value in the sensor with the elastic modulus of the component, the current strain value is calculated using Hooke's Law; By combining the temperature-thermal conductivity relationship, the change in heat flux density per unit time is calculated using Fourier's law of heat conduction. Based on the acceleration-mass density relationship, the impact load response value is calculated according to Newton's second law. The frequency shift coefficient is calculated using the natural frequency theory based on the relationship between vibration frequency, elastic modulus, and mass density. The strain value, heat flux density change value, impact load response value, and frequency offset coefficient are arranged in a preset order to form the physical response vector of the current key component; The physical property response vectors of all key components are aggregated to form a set of component physical response vectors at the current observation time.

[0011] Optionally, step five specifically includes: The multi-scale perturbation image set and the component physical response vector set are combined according to the key component number and the observation time to form a joint input pair for each component; Based on the initial state images of each key component, combined with the strain value, heat flux density change value, impact response value and frequency offset coefficient in the corresponding physical response vector, image state samples at multiple times are generated step by step according to the preset evolution step size and virtual state evolution strategy. The initial state images of each key component are micro-scale image blocks, meso-scale image blocks and macro-scale image blocks of each key component. The virtual state evolution strategy includes: The cumulative increase in strain value is mapped to the extension and depth increase of crack path in the initial state image, generating a crack evolution image sequence; The heat flux density change value is mapped to the expansion rate of the thermal anomaly region and the edge temperature difference gradient in the initial state image to generate a hot spot evolution image sequence; The impact load response value and frequency offset coefficient are jointly mapped to the structural torsion amplitude and edge misalignment degree in the initial state image to generate a morphological deformation image sequence. The crack evolution image sequence, hot spot evolution image sequence, and morphological deformation image sequence are arranged in chronological order to form a virtual fault evolution image sequence. Each virtual fault evolution image represents the potential fault morphology of the corresponding component at each evolution moment. Each virtual fault evolution image in the virtual fault evolution image sequence is used as a node in the topological evolution graph. In accordance with the time generation order of the virtual fault evolution image sequence, a directed edge is established by pointing the node corresponding to the previous image frame to the node corresponding to the next image frame, thus forming a unidirectional evolution path of the fault state. The image frame is the virtual fault evolution image at the corresponding time step. Generate a topology evolution graph for each key component and aggregate them to construct a set of topology evolution graphs; The Gromov-Wasserstein distance was used to calculate the structural similarity between the component state topology graph generated at the current observation time and each topology evolution graph in the set of topology evolution graphs. Based on the minimum Gromov-Wasserstein distance, the topology evolution graph most similar to the current component state topology graph is determined. The most similar topology evolution graph is the optimal matching evolution path, and the optimal matching evolution path is the most likely failure evolution process of the component. Extract the fault morphology information of the terminal node of the optimal matching evolution path. The fault morphology information includes crack path, expansion rate and edge temperature gradient of thermal anomaly region, structural distortion amplitude and edge misalignment degree. Finally, the evolution path number of the optimal match and the corresponding match confidence score are output. The match confidence score is the match similarity score obtained after Gromov-Wasserstein distance normalization.

[0012] Optionally, step six specifically includes: Obtain the topological evolution graph node information and physical response vector corresponding to each evolution stage on the optimal matching evolution path, and perform reverse state reconstruction according to the reverse time order. Generate a back-pull state image sequence using a simulation method opposite to the virtual fault evolution. The reverse reconstruction steps include: Perform path contraction and boundary reconstruction on the crack propagation path to simulate the microcrack retreat process; Temperature field reduction and thermal gradient smoothing are performed on the thermal anomaly region to simulate the thermal diffusion retreat process; Structural correction and edge alignment operations are performed on the deformed areas to simulate the macroscopic offset recovery process; The last frame of the back-pull state image in the back-pull state image sequence is selected as the back-pull state image, and structural residual analysis is performed with the initial state image at the current observation time. Compare the structural residual values ​​with a preset residual threshold: If the structural residual is less than the preset residual threshold, the optimal matching evolution path is considered to have high credibility, and the fault judgment is valid. If the structural residual is greater than the preset threshold, the current evolution path is considered to have a deviation, the current evolution path is excluded, and the process returns to step five to recalculate the Gromov-Wasserstein distance until the structural residual is less than the preset residual threshold.

[0013] Optionally, step seven specifically includes: Based on the optimal matching evolution path, a natural language diagnostic report is generated, which includes: Name of the faulty component and its location within the UAV structure; The identified fault types include crack evolution, hot spot evolution, and morphological deformation; The matching similarity score obtained after Gromov-Wasserstein distance normalization; Based on the type of faulty component, recommended handling suggestions are generated, including suggestions for replacement, suggestions for shutdown and inspection, and suggestions for continued observation. Meanwhile, the initial state image, the backward state image, and the virtual fault evolution image are visualized in chronological order.

[0014] The beneficial effects of this invention are: This invention integrates multimodal image data and synchronous sensor data to construct a unified component state topology map. By combining perturbation-enhanced multi-scale images and material physical parameter responses, it utilizes a virtual fault evolution mechanism to simulate and predict potential structural anomalies in critical UAV components, significantly improving the foresight and accuracy of fault identification. Compared to traditional methods based on single-modal data or static image analysis, this invention not only dynamically tracks the evolution of fault states but also performs structural-level comparison and optimal path selection based on Gromov-Wasserstein distance across multiple potential fault paths, exhibiting stronger diagnostic robustness and interpretability. Furthermore, by introducing a structural residual verification mechanism and a natural language diagnostic report output mode, it achieves visualized expression and user-friendly presentation of fault identification results, enhancing operability and application value in engineering practice. In complex operating conditions and high-dimensional heterogeneous data scenarios, this method demonstrates higher fault identification accuracy and system intelligence, overcoming bottlenecks in existing technologies such as weak multi-source data integration capabilities, untraceable fault evolution, and low reliability of diagnostic results. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0016] Figure 1This is an overall flowchart of the computer vision-based method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) proposed in this invention. Figure 2 This is a flowchart illustrating the physical response vector of the components in the computer vision-based UAV intelligent device detection and fault diagnosis method proposed in this invention. Figure 3 This is a flowchart illustrating the evolution path generation for the optimal matching of the computer vision-based UAV intelligent device detection and fault diagnosis method proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figure 1-3 A computer vision-based method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) includes the following steps: Step 1: Acquire multimodal image data and synchronous sensor data of key components of the UAV, perform preprocessing on each, and generate multimodal image data and synchronous sensor data with unified structure; Step 2: Based on the unified structure of multimodal image data and unified structure of synchronous sensor data, construct the component state topology diagram at the current observation time; Step 3: Divide the multimodal image data with a unified structure into layers according to the image region size, and perform local perturbation enhancement operations to generate a multi-scale perturbation image set; Step 4: Based on the synchronous sensor data and component material parameters of the unified structure, extract the physical response vector to form a set of component physical response vectors; Step 5: Based on the multi-scale perturbation image set and the physical response vector set, perform virtual fault evolution to form a set of topology evolution diagrams, and perform structural comparison between the component state topology diagram and the topology evolution diagram of each component based on the Gromov-Wasserstein distance to determine the optimal matching evolution path; Step 6: Perform fault inversion simulation on the optimal matching evolution path to generate a backtracked state image, and calculate the structural residual between the backtracked state image and the initial state image at the current observation time. Step 7: Generate a natural language diagnostic report based on the optimal matching evolution path.

[0019] This invention achieves deep perception of the status of key components of a UAV by fusing multimodal images and synchronous sensor data; it constructs a component status topology map and introduces multi-scale disturbance images and physical response vectors to perform virtual fault evolution, improving the accuracy of potential fault identification; it uses Gromov-Wasserstein distance to accurately match evolution paths, effectively simulating structural degradation trends; it avoids misdiagnosis through a structural residual verification mechanism, enhancing diagnostic robustness; and finally, it outputs a natural language diagnostic report, realizing intelligent identification and interpretable fault diagnosis of UAV equipment status, improving the system's automation and traceability.

[0020] In this embodiment, step one specifically includes: The key components of the UAV include the rotor assembly, motor, arm, battery compartment, flight control sensors, and GPS communication module area, and each key component is numbered. The multimodal image data includes RGB images, infrared images, and structured light images; The synchronous sensor data includes stress, acceleration, vibration frequency, and temperature information; The preprocessing steps include: Image size standardization, spatial coordinate registration, bilateral filtering for noise reduction, and timestamp alignment are performed on multimodal image data to generate multimodal image data with a unified structure. The synchronous sensor data is timestamped, outliers are removed, and normalized to generate synchronous sensor data with a unified structure.

[0021] This step achieves comprehensive perception of the equipment's multi-dimensional status by numbering and managing multiple key components of the UAV, combining multimodal image data (RGB, infrared, and structured light images) with synchronous sensor data such as stress, acceleration, vibration frequency, and temperature. The preprocessing stage unifies the structure and time reference of image and sensor data, improving the accuracy and efficiency of multi-source information fusion. It effectively ensures the quality of basic data for subsequent topology modeling and fault evolution analysis, providing structured, clean, and time-consistent high-quality input for the intelligent diagnostic model, significantly enhancing system stability and diagnostic accuracy.

[0022] In this embodiment, step two specifically includes: Visual feature vectors are extracted from the preprocessed multimodal image data with a unified structure using Sobel edge detection. The visual feature vectors include edge direction, texture distribution, surface reflectivity, and regional crack boundaries. Physical state vectors are extracted from preprocessed, unified structured synchronous sensor data. These physical state vectors include stress fluctuation values, acceleration vectors, temperature gradients, and vibration spectra. The visual feature vector and physical state vector corresponding to each key component are concatenated according to the numerical dimension, and Z-score standardization is performed to make the mean of each feature dimension zero and the variance one, generating a unified node state vector. Each key component is used as a node in the component state topology graph, and the node state vector is the attribute vector of the corresponding node. Based on the actual physical connection relationship of each key component of the UAV, a directed edge of the component state topology graph is constructed. The direction of the directed edge is from the force source component to the force receiving component. For each directed edge, the angle between the acceleration vectors of the two connected components at the current moment is calculated as a weight value.

[0023] This step significantly improves the comprehensiveness and accuracy of key component state modeling by integrating visual features extracted from Sobel edge detection with synchronous sensor data to construct a unified multi-source node state vector. Z-score normalization enhances the comparability of different modal features and ensures consistent weights for each dimension in topology analysis. Directed edges established based on force sources and force relationships reflect the real physical dependency logic, and edge weights calculated using acceleration angles further characterize the dynamic characteristics of the current structural response, effectively supporting the accuracy of structural rationality and confidence assessment in subsequent fault path reasoning.

[0024] In this embodiment, step three specifically includes: Multimodal image data with a unified structure is hierarchically divided according to image region size to construct image patch sets of three scales. The construction steps are as follows: Image regions smaller than 32×32 pixels are defined as microscale image blocks to characterize crack edges, hole initiation points, and micro-damage areas. Image regions with sizes between 32×32 and 96×96 pixels are defined as mesoscale image blocks to characterize local deformation, edge warping, and loosening of spiral connection components. Image regions with a size greater than 96×96 pixels are defined as macro-scale image blocks, used to characterize overall anomalies such as structural tilt, morphological deviation, and component detachment. Local perturbation enhancement operations are performed on image patches of each scale, specifically including: Perform affine transformations and Gaussian perturbations on microscale image patches to simulate fine-grained deformation and blurred boundaries; Brightness perturbation and contrast stretching are applied to mesoscale image patches to enhance adaptability to local illumination changes; Random rotation, cropping, and edge occlusion are performed on macroscopic image patches to enhance robustness to overall structural disturbances; Image patches processed at different scales and with different perturbation methods are aggregated to form a multi-scale perturbation image set containing multi-granularity structural information.

[0025] This step divides multimodal image data into micro, meso, and macro scale image blocks according to region size, enabling detailed differentiation of structural anomaly features at different levels and accurate perception of crack initiation, component loosening, and overall displacement. Combined with perturbation enhancement strategies designed for different scales, such as affine transformation, Gaussian noise, brightness variation, and shearing occlusion processing, the robustness and generalization ability of the model to image changes under diverse disturbance conditions are significantly improved. The resulting multi-scale perturbation image set provides a richer and more discriminative structural perception foundation for subsequent fault evolution modeling.

[0026] In this embodiment, step four specifically refers to: Based on a pre-set component material library, material parameter information corresponding to each key component of the UAV is obtained. The material parameter information includes elastic modulus, thermal conductivity, mass density, specific heat capacity and Poisson's ratio. The material parameters and synchronous sensor data of the unified structure are aligned with the timestamp to construct a joint data pair for each key component at the current observation time. The joint data pair includes stress-elastic modulus relationship, temperature-thermal conductivity relationship, acceleration-mass density relationship, and vibration frequency-elastic modulus and mass density relationship. Based on the stress-elastic modulus relationship, and combining the stress measurement value in the sensor with the elastic modulus of the component, the current strain value is calculated using Hooke's Law; By combining the temperature-thermal conductivity relationship, the change in heat flux density per unit time is calculated using Fourier's law of heat conduction. Based on the acceleration-mass density relationship, the impact load response value is calculated according to Newton's second law. The frequency shift coefficient is calculated using the natural frequency theory based on the relationship between vibration frequency, elastic modulus, and mass density. The strain value, heat flux density change value, impact load response value, and frequency offset coefficient are arranged in a preset order to form the physical response vector of the current key component; The physical property response vectors of all key components are aggregated to form a set of component physical response vectors at the current observation time.

[0027] This step establishes a physical correspondence between component material parameters and data from various sensors. By combining Hooke's law, Fourier's law, Newton's laws, and natural frequency theory, it enables multidimensional quantitative modeling of the strain, heat flow, impact, and vibration performance of key components without destructive testing. The generated physical response vector comprehensively reflects the mechanical and thermal state of the component under complex operating conditions, effectively improving the early detection capability of structural hazards and providing an accurate and interpretable physical basis for subsequent fault evolution modeling.

[0028] In this embodiment, step five specifically includes: The multi-scale perturbation image set and the component physical response vector set are combined according to the key component number and the observation time to form a joint input pair for each component; Based on the initial state images of each key component, combined with the strain value, heat flux density change value, impact response value and frequency offset coefficient in the corresponding physical response vector, image state samples at multiple times are generated step by step according to the preset evolution step size and virtual state evolution strategy. The initial state images of each key component are micro-scale image blocks, meso-scale image blocks and macro-scale image blocks of each key component. The virtual state evolution strategy includes: The cumulative increase in strain value is mapped to the extension and depth increase of crack path in the initial state image, generating a crack evolution image sequence; The heat flux density change value is mapped to the expansion rate of the thermal anomaly region and the edge temperature difference gradient in the initial state image to generate a hot spot evolution image sequence; The impact load response value and frequency offset coefficient are jointly mapped to the structural torsion amplitude and edge misalignment degree in the initial state image to generate a morphological deformation image sequence. The crack evolution image sequence, hot spot evolution image sequence, and morphological deformation image sequence are arranged in chronological order to form a virtual fault evolution image sequence. Each virtual fault evolution image represents the potential fault morphology of the corresponding component at each evolution moment. Each virtual fault evolution image in the virtual fault evolution image sequence is used as a node in the topological evolution graph. In accordance with the time generation order of the virtual fault evolution image sequence, a directed edge is established by pointing the node corresponding to the previous image frame to the node corresponding to the next image frame, thus forming a unidirectional evolution path of the fault state. The image frame is the virtual fault evolution image at the corresponding time step. Generate a topology evolution graph for each key component and aggregate them to construct a set of topology evolution graphs; The Gromov-Wasserstein distance was used to calculate the structural similarity between the component state topology graph generated at the current observation time and each topology evolution graph in the set of topology evolution graphs. Based on the minimum Gromov-Wasserstein distance, the topology evolution graph most similar to the current component state topology graph is determined. The most similar topology evolution graph is the optimal matching evolution path, and the optimal matching evolution path is the most likely failure evolution process of the component. Extract the fault morphology information of the terminal node of the optimal matching evolution path. The fault morphology information includes crack path, expansion rate and edge temperature gradient of thermal anomaly region, structural distortion amplitude and edge misalignment degree. Finally, the evolution path number of the optimal match and the corresponding match confidence score are output. The match confidence score is the match similarity score obtained after Gromov-Wasserstein distance normalization.

[0029] This step constructs a virtual fault evolution sequence that integrates image perturbation and physical response. Based on various morphological changes such as crack propagation, hot spot expansion, and structural distortion, a topological evolution map is generated. The Gromov-Wasserstein distance is then applied for structural similarity comparison to accurately identify potential fault evolution paths and final morphologies of critical UAV components. This method not only significantly improves the accuracy and interpretability of fault inference but also provides high-confidence support for component pre-failure prediction under complex operating conditions, enhancing the intelligent diagnostic capabilities of the detection system.

[0030] In this embodiment, step six specifically includes: Obtain the topological evolution graph node information and physical response vector corresponding to each evolution stage on the optimal matching evolution path, and perform reverse state reconstruction according to the reverse time order. Generate a back-pull state image sequence using a simulation method opposite to the virtual fault evolution. The reverse reconstruction steps include: Perform path contraction and boundary reconstruction on the crack propagation path to simulate the microcrack retreat process; Temperature field reduction and thermal gradient smoothing are performed on the thermal anomaly region to simulate the thermal diffusion retreat process; Structural correction and edge alignment operations are performed on the deformed areas to simulate the macroscopic offset recovery process; The last frame of the back-pull state image in the back-pull state image sequence is selected as the back-pull state image, and structural residual analysis is performed with the initial state image at the current observation time. Compare the structural residual values ​​with a preset residual threshold: If the structural residual is less than the preset residual threshold, the optimal matching evolution path is considered to have high credibility, and the fault judgment is valid. If the structural residual is greater than the preset threshold, the current evolution path is considered to have a deviation, the current evolution path is excluded, and the process returns to step five to recalculate the Gromov-Wasserstein distance until the structural residual is less than the preset residual threshold.

[0031] This step introduces a graph-based state inverse reconstruction mechanism to perform operations such as crack regression, thermal field reduction, and structural correction on the optimally matched fault evolution path, generating a back-projected state image. This image is then compared with the current observed state using structural residuals to construct a closed-loop verification mechanism. This method effectively avoids one-way misjudgment in fault diagnosis, improves the robustness and reliability of the diagnostic process, ensures that the final output fault evolution path more closely matches the actual evolution process, and enhances the interpretability and adaptability of the UAV component fault detection system.

[0032] In this embodiment, step seven specifically includes: Based on the optimal matching evolution path, a natural language diagnostic report is generated, which includes: Name of the faulty component and its location within the UAV structure; The identified fault types include crack evolution, hot spot evolution, and morphological deformation; The matching similarity score obtained after Gromov-Wasserstein distance normalization; Based on the type of faulty component, recommended handling suggestions are generated, including suggestions for replacement, suggestions for shutdown and inspection, and suggestions for continued observation. Meanwhile, the initial state image, the backward state image, and the virtual fault evolution image are visualized in chronological order.

[0033] This step utilizes natural language generation technology to transform the diagnostic results of the optimal matching evolution path into a structured natural language diagnostic report, and simultaneously generates an image visualization atlas. This visually presents the faulty component, type, location, evolution process, and handling recommendations, significantly improving the interpretability and engineering usability of the diagnostic results. By introducing a matching similarity scoring mechanism and a visualized evolution comparison chart, it helps maintenance personnel quickly understand the fault mechanism, make scientific decisions on maintenance plans, and enhance the system's application value and intelligence level in actual UAV inspection and maintenance.

[0034] Example 1: To verify the feasibility of this invention in practice, it was applied to a large UAV inspection company during the inspection and maintenance of a batch of multi-rotor industrial UAVs. On-site engineers discovered that some UAVs exhibited slight vibrations, abnormal thermal imaging, and delayed control response during flight. Traditional detection methods mainly rely on experience-based manual inspection and sensor-based early warning models, which have weak capabilities for accurately locating faulty components and provide only a rough assessment of fault types. Especially when facing structural hazards such as microcrack propagation or potential thermal runaway trends, they lack effective forward-looking modeling methods. To improve inspection accuracy and reduce false alarm rates, the company introduced the "Computer Vision-Based UAV Intelligent Equipment Detection and Fault Diagnosis Method" proposed in this invention, and conducted a systematic evaluation and comparative test on 20 UAVs of the same model.

[0035] During implementation, RGB cameras, infrared thermal imagers, and structured light scanning equipment were uniformly deployed for six key components of each UAV: ​​rotor assembly, motor, arm, battery compartment, flight control sensors, and GPS communication module. Stress gauges, thermistors, MEMS accelerometers, and exciters were also provided to acquire corresponding multimodal images and synchronized sensor data. All image data underwent spatial coordinate alignment, image size normalization, and filtering for noise reduction. Sensor data underwent timestamp synchronization, outlier removal, and normalization to generate a standardized structured data format. Subsequently, the system applied the Sobel edge detection algorithm to extract visual features of the components from the processed image data and performed physical response extraction based on Hooke's law, Fourier's law, and vibration feature modeling on the sensor data to construct node state vectors and topology graphs.

[0036] To characterize the evolution trend of potential faults, the system fuses multi-scale image patches with physical response vectors, generating a multi-time-scale virtual fault image sequence under the control of a diffusion model. It then constructs a time-directed evolutionary topology graph and uses the Gromov-Wasserstein distance algorithm to compare the current state with historical evolution paths, achieving evolutionary path identification and fault type confirmation. The system automatically performs a reverse simulation verification process on the judgment results, constructing a backward state image and calculating structural residuals with the original observation images to improve the reliability of the diagnostic results. Finally, the system outputs a natural language diagnostic report containing component number, fault type, confidence score, fault severity score, and recommended handling strategy, simultaneously displaying the initial image, backward image, and virtual evolutionary map.

[0037] For example, during the inspection of a UAV with the serial number A13, the system identified a micro-crack evolution trend in its arm. The crack started as a high-contrast 27×24 pixel edge, and after evolution prediction, its path extended by approximately 62%, and the crack depth increased by 43%. Simultaneously, infrared images of the area showed an abnormally high heat flux density of approximately 2.1 W / m², suggesting possible material fatigue degradation. During the fault inversion simulation, the structural residual value was 0.034, lower than the system's preset threshold of 0.05, confirming the fault diagnosis. The output diagnostic report recommended immediate replacement of the component to prevent flight accidents.

[0038] To evaluate the performance difference between this method and traditional methods, the testing team selected three common diagnostic methods for comparison: Method A is a sensor alarm model based on threshold judgment, Method B is an image anomaly recognition system based on a rule base, and Method C is a traditional thermal imaging manual analysis mechanism. Statistical analysis was performed on the method of this invention across multiple dimensions, and the results are shown in the table below: Table 1. Performance Comparison of Typical UAV Fault Diagnosis Methods

[0039] As shown in Table 1, the method of this invention significantly outperforms traditional methods in fault identification accuracy, particularly in microcrack identification and thermal runaway trend judgment. By leveraging multi-scale image perturbation enhancement and physical response simulation mechanisms, the model's ability to perceive and proactively predict potential fault paths is effectively improved. Simultaneously, the Gromov-Wasserstein structural comparison and structural residual verification mechanisms significantly reduce the false positive rate. Regarding fault location, the method of this invention achieves centimeter-level positioning accuracy through topology graph construction and node state mapping, far superior to traditional manual thermal imaging analysis methods. While the generation time for natural language diagnostic reports is slightly longer, the output results are highly interpretable and combine text and graphics, greatly improving the practical experience and decision-making efficiency for engineers.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computer vision-based method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Step 1: Acquire multimodal image data and synchronous sensor data of key components of the UAV, perform preprocessing on each, and generate multimodal image data and synchronous sensor data with unified structure; Step 2: Based on the unified structure of multimodal image data and unified structure of synchronous sensor data, construct the component state topology diagram at the current observation time; Step 3: Divide the multimodal image data with a unified structure into layers according to the image region size, and perform local perturbation enhancement operations to generate a multi-scale perturbation image set; Step 4: Based on the synchronous sensor data and component material parameters of the unified structure, extract the physical response vector to form a set of component physical response vectors; Step 5: Based on the multi-scale perturbation image set and the physical response vector set, perform virtual fault evolution to form a set of topology evolution diagrams, and perform structural comparison between the component state topology diagram and the topology evolution diagram of each component based on the Gromov-Wasserstein distance to determine the optimal matching evolution path; Step 6: Perform fault inversion simulation on the optimal matching evolution path to generate a backtracked state image, and calculate the structural residual between the backtracked state image and the initial state image at the current observation time. Step 7: Generate a natural language diagnostic report based on the optimal matching evolution path.

2. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step one specifically includes: The key components of the UAV include the rotor assembly, motor, arm, battery compartment, flight control sensors, and GPS communication module area, and each key component is numbered. The multimodal image data includes RGB images, infrared images, and structured light images; The synchronous sensor data includes stress, acceleration, vibration frequency, and temperature information; The preprocessing steps include: Image size standardization, spatial coordinate registration, bilateral filtering for noise reduction, and timestamp alignment are performed on multimodal image data to generate multimodal image data with a unified structure. The synchronous sensor data is timestamped, outliers are removed, and normalized to generate synchronous sensor data with a unified structure.

3. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step two specifically involves: Visual feature vectors are extracted from the preprocessed multimodal image data with a unified structure using Sobel edge detection. The visual feature vectors include edge direction, texture distribution, surface reflectivity, and regional crack boundaries. Physical state vectors are extracted from preprocessed, unified structured synchronous sensor data. These physical state vectors include stress fluctuation values, acceleration vectors, temperature gradients, and vibration spectra. The visual feature vector and physical state vector corresponding to each key component are concatenated according to the numerical dimension, and Z-score normalization is performed to generate a unified node state vector. Each key component is used as a node in the component state topology graph, and the node state vector is the attribute vector of the corresponding node. Based on the actual physical connection relationship of each key component of the UAV, a directed edge of the component state topology graph is constructed. The direction of the directed edge is from the force source component to the force receiving component. For each directed edge, the angle between the acceleration vectors of the two connected components at the current moment is calculated as a weight value.

4. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step three specifically involves: Multimodal image data with a unified structure is hierarchically divided according to image region size to construct image patch sets of three scales. The construction steps are as follows: Image regions smaller than 32×32 pixels are defined as microscale image blocks to characterize crack edges, hole initiation points, and micro-damage areas. Image regions with sizes between 32×32 and 96×96 pixels are defined as mesoscale image blocks to characterize local deformation, edge warping, and loosening of spiral connection components. Image regions with a size greater than 96×96 pixels are defined as macro-scale image blocks, used to characterize overall anomalies such as structural tilt, morphological deviation, and component detachment. Local perturbation enhancement operations are performed on image patches of each scale, specifically including: Perform affine transformation and Gaussian perturbation on microscale image patches; Perform brightness perturbation and contrast stretching on mesoscale image patches; Perform random rotation, cropping, and edge occlusion processing on macroscopic image patches; Image patches processed at different scales and with different perturbation methods are combined to form a multi-scale perturbation image set.

5. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step four specifically refers to: Based on a pre-set component material library, material parameter information corresponding to each key component of the UAV is obtained. The material parameter information includes elastic modulus, thermal conductivity, mass density, specific heat capacity and Poisson's ratio. The material parameters and synchronous sensor data of the unified structure are aligned with the timestamp to construct a joint data pair for each key component at the current observation time. The joint data pair includes stress-elastic modulus relationship, temperature-thermal conductivity relationship, acceleration-mass density relationship, and vibration frequency-elastic modulus and mass density relationship. Based on the stress-elastic modulus relationship, and combining the stress measurement value in the sensor with the elastic modulus of the component, the current strain value is calculated using Hooke's Law; By combining the temperature-thermal conductivity relationship, the change in heat flux density per unit time is calculated using Fourier's law of heat conduction. Based on the acceleration-mass density relationship, the impact load response value is calculated according to Newton's second law. The frequency shift coefficient is calculated using the natural frequency theory based on the relationship between vibration frequency, elastic modulus, and mass density. The strain value, heat flux density change value, impact load response value, and frequency offset coefficient are arranged in a preset order to form the physical response vector of the current key component; The physical property response vectors of all key components are aggregated to form a set of component physical response vectors at the current observation time.

6. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step five specifically involves: The multi-scale perturbation image set and the component physical response vector set are combined according to the key component number and the observation time to form a joint input pair for each component; Based on the initial state images of each key component, combined with the strain value, heat flux density change value, impact response value and frequency offset coefficient in the corresponding physical response vector, image state samples at multiple times are generated step by step according to the preset evolution step size and virtual state evolution strategy. The initial state images of each key component are micro-scale image blocks, meso-scale image blocks and macro-scale image blocks of each key component. The virtual state evolution strategy includes: The cumulative increase in strain value is mapped to the extension and depth increase of crack path in the initial state image, generating a crack evolution image sequence; The heat flux density change value is mapped to the expansion rate of the thermal anomaly region and the edge temperature difference gradient in the initial state image to generate a hot spot evolution image sequence; The impact load response value and frequency offset coefficient are jointly mapped to the structural torsion amplitude and edge misalignment degree in the initial state image to generate a morphological deformation image sequence. The crack evolution image sequence, hot spot evolution image sequence, and morphological deformation image sequence are arranged in chronological order to form a virtual fault evolution image sequence. Each virtual fault evolution image represents the potential fault morphology of the corresponding component at each evolution moment. Each virtual fault evolution image in the virtual fault evolution image sequence is used as a node in the topological evolution graph. In accordance with the time generation order of the virtual fault evolution image sequence, a directed edge is established by pointing the node corresponding to the previous image frame to the node corresponding to the next image frame, thus forming a unidirectional evolution path of the fault state. The image frame is the virtual fault evolution image at the corresponding time step. Generate a topology evolution graph for each key component and aggregate them to construct a set of topology evolution graphs; The Gromov-Wasserstein distance was used to calculate the structural similarity between the component state topology graph generated at the current observation time and each topology evolution graph in the set of topology evolution graphs. Based on the minimum Gromov-Wasserstein distance, the topology evolution graph most similar to the current component state topology graph is determined. The most similar topology evolution graph is the optimal matching evolution path, and the optimal matching evolution path is the most likely failure evolution process of the component. Extract the fault morphology information of the terminal node of the optimal matching evolution path. The fault morphology information includes crack path, expansion rate and edge temperature gradient of thermal anomaly region, structural distortion amplitude and edge misalignment degree. Finally, the evolution path number of the optimal match and the corresponding match confidence score are output. The match confidence score is the match similarity score obtained after Gromov-Wasserstein distance normalization.

7. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step six specifically includes: Obtain the topological evolution graph node information and physical response vector corresponding to each evolution stage on the optimal matching evolution path, and perform reverse state reconstruction according to the reverse time order. Generate a back-pull state image sequence using a simulation method opposite to the virtual fault evolution. The reverse reconstruction steps include: Perform path contraction and boundary reconstruction on the crack propagation path to simulate the microcrack retreat process; Temperature field reduction and thermal gradient smoothing are performed on the thermal anomaly region to simulate the thermal diffusion retreat process; Structural correction and edge alignment operations are performed on the deformed areas to simulate the macroscopic offset recovery process; The last frame of the back-pull state image in the back-pull state image sequence is selected as the back-pull state image, and structural residual analysis is performed with the initial state image at the current observation time. Compare the structural residual values ​​with a preset residual threshold: If the structural residual is less than the preset residual threshold, the optimal matching evolution path is considered to have high credibility, and the fault judgment is valid. If the structural residual is greater than the preset threshold, the current evolution path is considered to have a deviation, the current evolution path is excluded, and the process returns to step five to recalculate the Gromov-Wasserstein distance until the structural residual is less than the preset residual threshold.

8. The method for detecting and diagnosing faults in unmanned aerial vehicles (UAVs) based on computer vision according to claim 1, characterized in that, Step seven specifically includes: Based on the optimal matching evolution path, a natural language diagnostic report is generated, which includes: Name of the faulty component and its location within the UAV structure; The identified fault types include crack evolution, hot spot evolution, and morphological deformation; The matching similarity score obtained after Gromov-Wasserstein distance normalization; Based on the type of faulty component, recommended handling suggestions are generated, including suggestions for replacement, suggestions for shutdown and inspection, and suggestions for continued observation. Meanwhile, the initial state image, the backward state image, and the virtual fault evolution image are visualized in chronological order.

Images