Robot power equipment defect detection system based on multi-modal image processing

Abstract

The application provides a robot power equipment defect detection system based on multi-modal image processing, which improves the accuracy and reliability of power equipment defect identification. Infrared and visible light images are jointly collected, and a multi-source feature and equipment structure prior registration algorithm is used to realize the spatio-temporal alignment of multi-modal images. Then, a dynamic weighted fusion strategy is used to generate more discriminative fusion features, and a double branch mechanism is used to extract abnormal features and thermal physical consistency for verification. Finally, a neural network model with fusion physical prior is constructed to classify and locate the verified feature data. The application combines structure and thermal information, introduces a physical constraint mechanism and a joint training strategy, significantly improves the robustness and engineering interpretability of the system under complex working conditions, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of power equipment defect detection, specifically a robotic power equipment defect detection system based on multimodal image processing. Background Technology

[0002] With the continuous expansion of my country's power system, the types of power equipment are becoming increasingly diverse and their structures increasingly complex. Their operating conditions and external environments are also becoming more variable, which places higher demands on equipment condition monitoring and defect diagnosis technologies. Once power equipment experiences problems such as insulation aging, poor contact, or partial discharge, it may lead to hidden defects such as thermal anomalies, structural deformation, and discharge damage, which in turn can induce serious power failures or even safety accidents. Traditional defect detection technologies mainly rely on manual inspection combined with single-modal images collected by robots for identification. However, these methods have limited information sources, are easily affected by environmental interference, and lack intelligent mechanisms.

[0003] In existing technologies, some studies have attempted to improve detection performance through intelligent image processing. Convolutional neural networks are used to process infrared images and extract areas with abnormal surface temperatures to determine whether there are potential defects in power equipment. The advantage lies in utilizing the non-contact and high sensitivity of infrared imaging and leveraging depth models to enhance automatic recognition capabilities. However, existing technologies still rely on single-modal input, resulting in a lack of structural information. Modal registration and fusion are not performed, leading to low information utilization efficiency. Furthermore, there is a lack of physical consistency verification for thermal anomalies, resulting in the problem of false anomalies. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention proposes a robot power equipment defect detection system based on multimodal image processing.

[0005] Therefore, the technical solution adopted by the present invention is as follows:

[0006] A robot-based power equipment defect detection system based on multimodal image processing, comprising:

[0007] The acquisition module includes multiple sensors and a registration device. The multiple sensors respectively acquire multimodal image data of the same power equipment, and the registration device performs an image registration algorithm on the multimodal image data and outputs the registered multimodal image data.

[0008] The constraint decision module includes a dynamic weighted fusion unit and a dual-branch analysis unit. The dynamic weighted fusion unit performs weighted fusion on the registered multimodal image data. Based on the fusion result, the dual-branch analysis unit extracts abnormal correlation features from the fusion result in the first branch, and performs physical constraint verification based on the abnormal correlation features in the second branch, and outputs the verified feature data.

[0009] The defect detection module constructs a neural network model and trains it using a joint loss function that includes physical constraint priors. After training, the model identifies defects in the verified feature data and outputs the defect type, location, and confidence level of the corresponding power equipment.

[0010] Furthermore, the multimodal image data is divided into infrared thermal image data. and visible light image data ,

[0011] The specific execution steps of the image registration algorithm are as follows:

[0012] 1) Extract features from infrared thermal image data and visible light image data, represented as:

[0013]

[0014] in, This represents an infrared image feature extraction network; This represents a network for extracting features from visible light images. and These represent the thermal feature map and the structural feature map, respectively.

[0015] 2) Extract the keypoint coordinate pairs from the thermal feature map and the structural feature map, represented as:

[0016]

[0017] in, Represents the first in the infrared image Coordinates of one feature point; Represents the first in a visible light image Coordinates of one feature point; This indicates the number of feature points involved in the registration optimization.

[0018] 3) Define the two-dimensional affine transformation matrix Furthermore, a structural similarity regularization term is introduced as a registration constraint to construct an optimization objective function, expressed as:

[0019]

[0020] in, This represents the optimized two-dimensional affine transformation matrix; Represents the first in the infrared image Homogeneous coordinates of the feature points; This indicates that a transformation is applied to the infrared image. The image after; This represents the image structure similarity loss function; Indicates the structural constraint weighting factor;

[0021] 4) The optimal transformation matrix is ​​obtained through optimization. Register the infrared thermal image to .

[0022] Furthermore, the weighted fusion is performed through a cross-attention mechanism, first extracting the registered infrared thermal image. thermal feature map ,

[0023] right and Perform a linear mapping, expressed as:

[0024]

[0025] in, To query the vector matrix; The key vector matrix; It is a value vector matrix; , and These represent the query vector mapping matrix, the key vector mapping matrix, and the value vector mapping matrix, respectively.

[0026] Based on the result of the linear mapping, the attention weights are calculated using the following formula:

[0027]

[0028] in, The first character in the structural feature diagram The feature pairs of the nth pixel in the thermal feature map Attention weights for features of each pixel; The first character in the structural feature diagram The query vector extracted from the features of each pixel; In the thermal feature map, the first Transpose of the feature key vector of pixels; and These are the height and width of the thermal feature map, respectively. This represents the transpose of the feature key vector of each pixel in the heat map; This is the scaling factor;

[0029] The attention weights are weighted and aggregated using the following formula:

[0030]

[0031] in, Represents the fused feature map; In the thermal feature map, the first A vector of values ​​for each pixel.

[0032] Furthermore, the first branch extracts anomaly correlation features through a deep residual network, represented as:

[0033]

[0034] in, This represents a convolutional network that includes residual connections; This indicates abnormal correlation characteristics.

[0035] Furthermore, the specific steps for the physical constraint verification are as follows:

[0036] 1) Through linear transformation, from abnormal correlation features Temperature map obtained , is represented as:

[0037]

[0038] in, The transpose of the linear transformation weight vector for extracting temperature estimates from anomaly correlation features.

[0039] 2) Divide the temperature map into several small grids, discretize the two-dimensional steady-state heat conduction equation on each grid, and construct a thermophysical residual evaluation function, expressed as:

[0040]

[0041] in, Represents a grid Thermal conductivity residual; , and Representing grids respectively The average thermal conductivity, temperature, and estimated heat source values;

[0042] 3) Set the thermal residual threshold and generate a physical consistency mask. , is represented as:

[0043]

[0044] in, Represents a grid Physical consistency mask;

[0045] The abnormal correlation features are multiplied point by point with the physical consistency mask to obtain the verified feature data.

[0046] Furthermore, the neural network model consists of a backbone network. With three-branch prediction head The overall mapping process is represented as follows:

[0047]

[0048] in, This represents the probability distribution of the output defect type; Indicates the two-dimensional location of the center of the output defect; This represents the confidence score of the output recognition result; This represents the verified feature data.

[0049] Furthermore, the joint loss function is defined as:

[0050]

[0051] in, Represents the classification loss term; Indicates the location loss term; This represents the physical consistency loss term; , and These represent the weights of the classification loss term, the location loss term, and the physical consistency loss term, respectively.

[0052] Compared with the prior art, the advantages of the present invention are as follows:

[0053] This invention introduces a registration and fusion mechanism between infrared thermal images and visible light images to achieve collaborative perception of equipment structure and thermal anomalies, overcoming the limitations of single-modal recognition in complex lighting and occlusion environments, and effectively improving the accuracy and adaptability of defect identification.

[0054] This invention combines the heat conduction equation to construct a physical residual evaluation model, performs thermodynamic consistency verification on abnormal feature regions, effectively eliminates false anomaly regions, reduces the risk of false positives, and enhances the engineering interpretability of the system's identification results.

[0055] 3. This invention trains a neural network model by introducing a joint loss function that includes prior physical constraints, enabling the system to not only identify multimodal features but also to have adaptive learning capabilities for different equipment types and operating environments, making it suitable for deployment in multiple scenarios. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1This is a flowchart of the power equipment defect detection system of the present invention;

[0058] Figure 2 This is a flowchart of the data acquisition module of the present invention;

[0059] Figure 3 This is a flowchart of the constraint decision module of the present invention;

[0060] Figure 4 This is a schematic diagram of infrared thermal and visible light images of the present invention. Detailed Implementation

[0061] To achieve the above objectives, the present invention provides a robot-based electrical equipment defect detection system based on multimodal image processing. Please refer to [link to relevant documentation]. Figure 1-4 The system includes:

[0062] M1, the acquisition module, includes multiple sensors and a registration device. The multiple sensors acquire multimodal image data of the same power equipment, and the registration device performs an image registration algorithm on the multimodal image data and outputs the registered multimodal image data.

[0063] The acquisition module is used to acquire multimodal image data of the target power equipment and perform high-precision spatial and temporal synchronization registration. It includes multiple sensors and a registration device.

[0064] Infrared thermal image data of power equipment are collected using multiple sensors. and visible light image data Infrared thermal image data is mainly used to detect the non-uniformity of heat distribution on the surface of power equipment; visible light image data is used to observe and record the structural appearance of power equipment, including information such as color, texture, cracks, corrosion, and foreign object obstruction.

[0065] In the embodiments, when electrical equipment experiences defects such as poor contact, partial discharge, or insulation aging, its surface usually exhibits thermal anomalies, which may be accompanied by structural deformation, cracks, erosion, or foreign object obstruction. Therefore, infrared thermal image data and visible light image data provide complementary information from both the thermal field and structural aspects, respectively, providing an engineering basis for joint defect detection.

[0066] Because infrared and visible light images originate from different sensors, their viewing angles, focal points, and mounting postures differ, resulting in geometric differences. A deep learning-driven image registration algorithm is used to align the two types of images at the pixel level. The specific steps are as follows:

[0067] 1) Two dedicated feature extraction networks (ResNet, U-Net, etc.) are used to extract multi-level feature tensors from infrared thermal image data and visible light image data, respectively, as follows:

[0068]

[0069] in, This represents an infrared image feature extraction network; This represents a network for extracting features from visible light images. and These represent the thermal feature map and the structural feature map, respectively.

[0070] 2) Extract several pairs of corresponding keypoint coordinates from the two feature maps, represented as follows:

[0071]

[0072] in, Represents the first in the infrared image Coordinates of one feature point; Represents the first in a visible light image Coordinates of one feature point; This indicates the number of feature points involved in the registration optimization.

[0073] 3) Define a two-dimensional affine transformation matrix. This is used to map any coordinate point in an infrared image to new coordinates, in order to optimize... To minimize the reprojection error of all paired feature points, and to introduce a structural similarity regularization term as a registration constraint, an optimization objective function is constructed, expressed as:

[0074]

[0075] in, This represents the optimized two-dimensional affine transformation matrix; Represents the first in the infrared image Homogeneous coordinates of the feature points; This indicates that a transformation is applied to the infrared image. The image after; This represents the image structure similarity loss function; Indicates the structural constraint weighting factor;

[0076] 4) After optimization, the optimal transformation matrix is ​​obtained. Used to register infrared images to Finally, the registered image pairs are obtained. It has achieved pixel-level alignment in the coordinate system and has the ability to express thermal and structural information synchronously, which can serve as the input basis for subsequent constraint decision modules.

[0077] In this embodiment, the image resolution is uniformly set to 640×480 pixels, the infrared image thermal sensitivity is set to ±0.1°C, and the visible light image supports the RGB three-channel standard structure representation. When extracting feature points, the number of point pairs selected is 300~800 pairs, adaptively adjusted according to the image texture complexity, and a two-dimensional affine transformation matrix is ​​used. Initialize as an identity matrix and optimize the initial values ​​by randomly perturbing them within ±0.05 to improve convergence stability.

[0078] M2, the constraint decision module, includes a dynamic weighted fusion unit and a dual-branch analysis unit. The dynamic weighted fusion unit performs weighted fusion on the registered multimodal image data. Based on the fusion result, the dual-branch analysis unit performs physical constraint verification based on the first branch, and outputs the verified feature data based on the second branch.

[0079] The constraint decision module is used to fuse the registered multimodal image data and introduces physical constraints to improve feature reliability and reduce false defect misjudgments. This module includes a dynamic weighted fusion unit and a dual-branch analysis unit.

[0080] The dynamic weighted fusion unit introduces a structure-thermal coupled attention fusion mechanism (STCA) to fully consider the dynamic correlation between infrared and visible light images in terms of spatial location, channel semantics, and local response, thereby achieving multimodal deep fusion. The specific steps of the fusion are as follows:

[0081] 1) Extract the features of the registered multimodal image data, represented as:

[0082]

[0083] in, and These are the registered thermal feature map and structural feature map, respectively.

[0084] 2) Employing a cross-attention mechanism, the features are first linearly mapped, as follows:

[0085]

[0086] in, The query vector matrix represents the information requirement in the structural feature map; Let be the key vector matrix, representing the semantic labels in the heat map; This is a value vector matrix, representing the original information in the heat map; , and These represent the query vector mapping matrix, the key vector mapping matrix, and the value vector mapping matrix, respectively.

[0087] Then, the degree of attention that each pixel in the structural feature map has to the features of each pixel in the thermal feature map is calculated using the following formula:

[0088]

[0089] in, The first character in the structural feature diagram The feature pairs of the nth pixel in the thermal feature map Attention weights for features of each pixel; The first character in the structural feature diagram The query vector extracted from the features of each pixel; In the thermal feature map, the first Transpose of the feature key vector of pixels; and These are the height and width of the thermal feature map, respectively. This represents the transpose of the feature key vector of each pixel in the heat map; This is a scaling factor to avoid gradient vanishing or exploding;

[0090] The attention weights are weighted and aggregated using the following formula:

[0091]

[0092] in, Represents the fused feature map; In the thermal feature map, the first A vector of values ​​for 1 pixel;

[0093] The final fused feature map contains spatial matching, temperature semantic information, and structural clarity, and can be used for subsequent anomaly analysis.

[0094] The dual-branch analysis unit, based on a fused feature map, employs a combination of visual feature extraction and physical constraint verification to accurately extract reliable defect candidate regions for power equipment, comprising a first branch and a second branch.

[0095] The first branch extracts anomaly correlation features related to power equipment defects from the fused feature map using a deep residual network, represented as follows:

[0096]

[0097] in, This indicates a convolutional network containing residual connections, used to capture potential defect features such as structural disorder, thermal anomalies, or texture abrupt changes in the fused feature map; This indicates abnormal correlation characteristics.

[0098] Because images of power equipment often contain complex factors such as uneven lighting, background interference, and surface impurities, relying solely on the abnormal correlation features captured by the first branch can easily lead to false positives. To further improve the reliability of defect localization, the second branch uses the heat conduction equation to make a consistency judgment on the thermal behavior of the abnormal correlation feature region, thereby achieving physical-level authenticity verification on the basis of semantic feature recognition, thus forming a complementary dual-branch reliable detection mechanism.

[0099] A false positive refers to an area that the system mistakenly identifies as a defect when it is not actually a defect. In the embodiment, in the image of power equipment, some areas that appear abnormal may actually be simply the thermal inertia of the material itself, or non-defective structures such as surface contamination, shadows, bolts, or signs. Sensor errors or environmental interference can also cause anomalies in the thermal or visible image. The first branch easily misidentifies these non-defective areas as defects, which is a false positive. The second branch judges whether the temperature change in these areas is reasonable. If the heat distribution conforms to the laws of physics, it indicates that there is no real defect and the area is removed, thereby improving the accuracy and reliability of the detection results.

[0100] The second branch, based on the heat conduction equation, performs physical constraint verification on the abnormal correlation characteristics to determine whether the regions corresponding to the abnormal correlation characteristics violate thermophysical laws. Specifically,

[0101] Through linear transformation, from abnormal correlation features Temperature map obtained The formula represents the local temperature distribution on the surface of electrical equipment:

[0102]

[0103] in, This represents the transpose of the linear transformation weight vector used to extract temperature estimates from anomaly correlation features;

[0104] Under ideal steady-state conditions, the temperature at any point on the surface of electrical equipment should satisfy the two-dimensional steady-state heat conduction equation, expressed as:

[0105]

[0106] in, Represents a two-dimensional gradient operator; Indicates the location of electrical equipment The actual temperature;

[0107] Indicates position The thermal conductivity depends on the type of material and the operating conditions, and can be estimated through a priori configuration or spectral analysis. The term represents the heat source, taking into account factors such as electric current heating and external radiation. When there is structural damage or abnormal conductivity in electrical equipment, heat may not be effectively dissipated in local areas or additional heat sources may accumulate, causing the actual temperature distribution to deviate from the two-dimensional steady-state heat conduction equation.

[0108] The temperature map is divided into several small grids, and the two-dimensional steady-state heat conduction equation on each grid is discretized to construct a thermophysical residual evaluation function, expressed as:

[0109]

[0110] in, Represents a grid Thermal conductivity residual; , and Representing grids respectively The estimated values ​​of average thermal conductivity, temperature, and heat source term;

[0111] Set thermal residual threshold Generate a physical consistency mask based on the threshold. , is represented as:

[0112]

[0113] in, Represents a grid The physical consistency mask; only when thermal behavior significantly violates physical laws is it considered a physical anomaly and a candidate for a credible defect;

[0114] The abnormal correlation features extracted from the first branch are multiplied point by point with the physical consistency mask to obtain the verified feature data.

[0115] The dual-branch analysis unit ensures that regions that simultaneously satisfy semantic and physical anomalies constitute high-confidence defect regions, effectively eliminating false positive regions caused by surface impurities, markings, uneven lighting, etc.

[0116] In the embodiment, the range of thermal conductivity values ​​in the two-dimensional steady-state heat conduction equation is as follows: Based on different material properties, the estimated value of the heat source term is... The thermal residual threshold is set to 0.1–0.5°C based on the type of power equipment, and the grid size is 16×16 pixel cells to ensure that each cell has sufficient resolution to reflect the temperature gradient change.

[0117] M3, the defect detection module, constructs a neural network model and trains it using a joint loss function that includes prior physical constraints. After training, the model identifies defects in the verified feature data and outputs the defect type, location, and confidence level of the corresponding power equipment.

[0118] The defect detection module is used to intelligently identify feature data after physical constraint verification, and output the defect type, location, and confidence level of the power equipment. This module mainly includes neural network model construction, joint loss function design, and training strategy.

[0119] A multi-layer convolutional neural network (CNN) is constructed as a defect recognition model. The input is the verified feature data output from the second branch, and the model consists of a backbone network. With three-branch prediction head The overall mapping process is represented as follows:

[0120]

[0121] in, This represents the probability distribution of the output defect type; Indicates the two-dimensional location of the center of the output defect; This represents the confidence score of the output recognition result; This represents the verified feature data;

[0122] To guide the model to simultaneously learn defect semantic recognition ability and physical consistency constraint ability, a joint loss function is constructed, expressed as:

[0123]

[0124] in, Represents the classification loss term; Indicates the location loss term; This represents the physical consistency loss term; , and These represent the weights of the classification loss term, the location loss term, and the physical consistency loss term, respectively.

[0125] The classification loss term measures the difference between the model's predicted defect type probability and the true label, and uses standard cross-entropy loss, expressed as:

[0126]

[0127] in, Indicates the number of actual labels; Indicates the first in the real label Class marker; express

[0128] Model for the first The predicted probability of a true label;

[0129] The location loss term is calculated by smoothing the difference between the defect location predicted by the L1 loss penalty model and the actual location.

[0130] The physical consistency loss term is masked by the physical consistency mask. By matching the corresponding thermal conductivity residual value, the defect region is ensured to conform to thermodynamic physical constraints. Penalties are imposed on the physical mismatch behavior of abnormal regions, and the model is encouraged to output defect regions with physical rationality.

[0131] The model is trained using a historical feature dataset containing multiple types of real defects. During the training process, the joint loss function is continuously minimized to obtain the optimal neural network model. The validated feature data is then input into the neural network model to obtain the final power equipment defect type, location, and confidence level.

[0132] In an embodiment, , and The ratios were set to 1, 0.5, and 0.2 respectively. This ratio was the optimal combination obtained through five sets of cross-validation during the training phase, which can balance recognition accuracy and reliability evaluation. During model training, the initial learning rate was 0.001, the Adam optimizer was used, and the maximum number of iterations was 100.

[0133] The proposed invention provides a robot power equipment defect detection system based on multimodal image processing, comprising an acquisition module, a constraint decision module, and a defect detection module. The acquisition module integrates infrared and visible light sensors and utilizes a registration algorithm jointly optimized by equipment structure priors and multimodal features to achieve spatial alignment of multi-source images. The constraint decision module includes a dynamic weighted fusion and bi-branch analysis structure, introducing physical thermal balance constraints to verify the authenticity of abnormal regions. The defect detection module trains a deep neural network model based on a joint loss function, embedding structural information, thermal features, and physical consistency into the learning process, ultimately achieving accurate output of defect type, location, and confidence level.

[0134] In summary, this invention overcomes the problems of insufficient information and low reliability in traditional single-modal detection. By integrating data-driven and physical modeling concepts, it constructs a defect detection system with multimodal perception, physical constraint verification, and intelligent recognition capabilities, which has strong engineering practical value and promising prospects for promotion.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robot-based power equipment defect detection system based on multimodal image processing, characterized in that, The system includes: The acquisition module includes multiple sensors and a registration device. The multiple sensors respectively acquire multimodal image data of the same power equipment, and the registration device performs an image registration algorithm on the multimodal image data and outputs the registered multimodal image data. The constraint decision module includes a dynamic weighted fusion unit and a dual-branch analysis unit. The dynamic weighted fusion unit performs weighted fusion on the registered multimodal image data. Based on the fusion result, the dual-branch analysis unit extracts abnormal correlation features from the fusion result in the first branch, and performs physical constraint verification based on the abnormal correlation features in the second branch, and outputs the verified feature data. The defect detection module constructs a neural network model and trains it using a joint loss function that includes prior physical constraints. After training, the model identifies defects in the verified feature data and outputs the defect type, location, and confidence level of the corresponding power equipment. The multimodal image data is divided into infrared thermal image data. and visible light image data , The specific execution steps of the image registration algorithm are as follows: 1) Extract features from infrared thermal image data and visible light image data, represented as: ； in, This represents an infrared image feature extraction network; This represents a network for extracting features from visible light images. and These represent the thermal feature map and the structural feature map, respectively. 2) Extract the keypoint coordinate pairs from the thermal feature map and the structural feature map, represented as: ； in, Represents the first in the infrared image Coordinates of one feature point; Represents the first in a visible light image Coordinates of one feature point; This indicates the number of feature points involved in the registration optimization. 3) Define the two-dimensional affine transformation matrix Furthermore, a structural similarity regularization term is introduced as a registration constraint to construct an optimization objective function, expressed as: ； in, This represents the optimized two-dimensional affine transformation matrix; Represents the first in the infrared image Homogeneous coordinates of the feature points; This indicates that a transformation is applied to the infrared image. The image after; This represents the image structure similarity loss function; Indicates the structural constraint weighting factor; 4) The optimal transformation matrix obtained through optimization Register the infrared thermal image to ; The specific steps for verifying the physical constraints are as follows: 1) Through linear transformation, from abnormal correlation features Temperature map obtained , represented as: ； in, This represents the transpose of the linear transformation weight vector used to extract temperature estimates from anomaly correlation features; 2) Divide the temperature map into several small grids, discretize the two-dimensional steady-state heat conduction equation on each grid, and construct a thermophysical residual evaluation function, expressed as: ； in, Represents a grid Thermal conductivity residual; , and Representing grids respectively The average thermal conductivity, temperature, and estimated heat source values; 3) Set the thermal residual threshold and generate a physical consistency mask. , represented as: ； in, Represents a grid Physical consistency mask; The abnormal correlation features are multiplied point by point with the physical consistency mask to obtain the verified feature data.

2. The robot power equipment defect detection system based on multimodal image processing according to claim 1, characterized in that, The weighted fusion obtains the fused feature map through a cross-attention mechanism. ; The first branch extracts anomaly correlation features using a deep residual network. .

3. The robot power equipment defect detection system based on multimodal image processing according to claim 2, characterized in that, The neural network model consists of a backbone network. With three-branch prediction head The overall mapping process is represented as follows: ； in, This represents the probability distribution of the output defect type; Indicates the two-dimensional location of the output defect center; This represents the confidence score of the output recognition result; This represents the verified feature data.

4. The robot power equipment defect detection system based on multimodal image processing according to claim 3, characterized in that, The joint loss function is defined as follows: ； in, Represents the classification loss term; Indicates the location loss term; This represents the physical consistency loss term; , and These represent the weights of the classification loss term, the location loss term, and the physical consistency loss term, respectively.

Images