Defect detection method and system based on joint distribution optimization and structural knowledge guidance

Abstract

This invention discloses a defect detection method and system based on joint distribution optimization and structural knowledge guidance. The method includes: parsing device structured documents to construct a device physical topology diagram containing physical connection relationships; using a graph convolutional network to process the topology diagram to generate structured feature vectors, and extracting perceptual feature vectors of multimodal perception data; constructing a knowledge-enhanced attention model, using the structured feature vectors as keys and values ​​to guide the cross-modal fusion of perceptual feature vectors; and training the model through a combined optimization objective including joint distribution divergence loss and Lipschitz stability constraints to ensure the consistency and physical authenticity of multimodal features in a shared semantic space. The invention can significantly improve the accuracy of defect detection and generate a physically interpretable root cause diagnosis report.

Description

Technical Field

[0001] The present invention relates to defect detection, and in particular to a defect detection method and system based on joint distribution optimization and structural knowledge guidance. Background Art

[0002] Industrial visual defect detection is the core technology cornerstone for ensuring the yield rate and production safety of intelligent manufacturing products. The accuracy and intelligence of its detection directly determine the upper limit of quality control in the high-end equipment manufacturing industry. In precision industries such as semiconductor manufacturing, aeroengines, and new energy, equipment defects often manifest themselves in complex forms spanning multiple physical dimensions. For example, a micron-scale crack in a circuit may be accompanied by subtle local temperature rises and abnormal fluctuations in operating parameters. Therefore, it is of great significance to develop defect detection methods that can deeply integrate multi-source information, possess physical interpretability, and provide early and accurate warnings.

[0003] Currently, research on industrial defect detection primarily focuses on the integration of deep learning and various sensing technologies. The mainstream approach primarily relies on the analysis of single-modal visible light images, employing object detection frameworks such as YOLO and Faster R-CNN, or image segmentation networks such as U-Net, to identify and locate defects within the images. To overcome the limitations of single-modal information, some research has begun exploring multimodal data fusion strategies. Initial attempts included simple pixel-level weighted overlay or channel merging of infrared thermal images and visible light images to supplement temperature information with visual features. Subsequently, attention-based methods were proposed, which, by constructing feature interaction modules, mutually enhance infrared and visible light features within deep networks. Furthermore, some work has attempted to utilize optical character recognition (OCR) to extract text from equipment maintenance reports and use this as auxiliary information to classify or corroborate defect categories identified by image analysis. In recent years, cross-modal pre-training models, such as CLIP, have made significant progress in image-text association, but these models are primarily general-purpose and are not specifically designed to address the structural characteristics of specific industrial equipment.

[0004] However, existing technologies still face several challenges when addressing complex industrial scenarios. These challenges stem from their inability to effectively establish the intrinsic connection between heterogeneous data and the physical entities of devices. For example, multimodal features suffer from pseudo-alignment and shallow correlation, and a lack of unified, physically interpretable metrics across cross-modal feature spaces. Specifically, at the modal collaboration level, traditional feature alignment methods (such as CCA and adversarial training) focus solely on matching statistical distributions, ignoring the constraints imposed by the physical structure of the device on multimodal associations. This results in the ineffective exploitation of essential connections between heat maps and key circuit nodes, and between text descriptions and 3D model regions. At the knowledge fusion level, existing methods simplify domain knowledge into rule bases or label expansions, failing to incorporate structured knowledge such as device topology maps as spatial constraints for feature learning. This results in defect localization deviating from physical reality. This stems from a modeling gap between the heterogeneous nature of industrial multimodal data (e.g., the local nature of images and the global nature of text) and the graph-structured nature of domain knowledge. Existing deep learning architectures struggle to uniformly address the contradiction between knowledge representation in non-Euclidean spaces and feature extraction in Euclidean spaces. Summary of the Invention

[0005] The purpose of the invention is to provide a defect detection method and system based on joint distribution optimization and structural knowledge guidance in order to solve the problems of inaccurate feature alignment and poor interpretability caused by the lack of physical topological constraints in existing industrial multimodal detection methods.

[0006] The technical solution is a defect detection method based on joint distribution optimization and structural knowledge guidance, including:

[0007] Parse device structured documents and build device physical topology diagrams;

[0008] Use graph neural networks to process the physical topology of the device and generate structured feature vectors;

[0009] Collect multimodal perception data from industrial equipment and extract perception feature vectors from the multimodal perception data;

[0010] In the attention model, the structured feature vector is used to guide the alignment of the perceptual feature vector, perform cross-modal feature fusion, and obtain the fused feature representation;

[0011] Based on the fused feature representation, defect detection results are generated.

[0012] Preferably, in the attention model, the steps of using the structured feature vector to guide the alignment of the perceptual feature vector to perform cross-modal feature fusion and obtain the fused feature representation include:

[0013] Mapping the structured feature vectors into a set of key vectors and a set of value vectors, respectively. The set of key vectors and value vectors are used to describe the structural information of the physical topology of the device;

[0014] Transform the perception feature vector into a query vector, which is used to represent the real-time status collected from the device;

[0015] The key vector is explored through the query vector to determine the correlation between the perception feature vector and the device physical topology, and the value vector is dynamically weighted based on the correlation to form a fused feature representation.

[0016] Preferably, the optimization step of the attention model includes:

[0017] Within the attention model, a corresponding modal feature distribution is generated for each of the multiple modalities, thereby obtaining a modal feature distribution set;

[0018] Calculate the arithmetic mean of all distributions in the modal feature distribution set and construct the mixed center distribution;

[0019] Measure the Kullback-Leibler divergence of each modal feature distribution in the modal feature distribution set toward the mixed center distribution, and integrate all the measured divergences into the joint distribution divergence loss;

[0020] The attention model is optimized based on the joint distribution divergence loss to minimize the topological differences between the feature distributions of each modality.

[0021] Preferably, the optimization step further comprises:

[0022] Obtain the gradient norm of the feature transformation function in the attention model, and obtain the stability constraint loss based on the gradient norm and the preset Lipschitz constant;

[0023] The stability constraint loss and the joint distribution divergence loss are weightedly combined to form a combined optimization objective;

[0024] The attention model is finally optimized based on the combined optimization objective to ensure that the feature transformation function meets the smoothness and continuity requirements of physical laws.

[0025] Preferably, the step of probing the key vector by the query vector to determine the relevance further comprises:

[0026] Generate a spatial constraint matrix based on the device physical topology. The spatial constraint matrix is ​​used to calibrate the physical adjacency relationships that are allowed to occur between device nodes.

[0027] When calculating the association between the query vector and the key vector, the spatial constraint matrix is ​​applied to shield the invalid associations between physically non-adjacent nodes and obtain the association.

[0028] Preferably, generating defect detection results based on the fused feature representation includes:

[0029] Decoding the fused feature representation to generate a defect localization mask that indicates the spatial location of the defect;

[0030] The features corresponding to the defect location in the fused feature representation are reasoned about in relation to the node attributes in the device physical topology diagram, tracing the physical root cause of the defect and generating a root cause diagnosis report that includes the confidence level of the faulty component.

[0031] Integrate defect location masks and root cause diagnosis reports to form component defect detection results.

[0032] According to another aspect of the present application, a defect detection system based on joint distribution optimization and structural knowledge guidance is provided, comprising:

[0033] A knowledge parsing module is configured to parse a device structured document to construct a device physical topology map;

[0034] a data acquisition module configured to collect multimodal sensing data from industrial equipment;

[0035] The feature processing module is connected to the knowledge analysis module and the data acquisition module.

[0036] A result generation module, communicating with the feature processing module, for generating defect detection results based on the fused feature representation;

[0037] The feature processing module includes:

[0038] Graph embedding unit, used to process the physical topology graph of the device and generate a structured feature vector;

[0039] A perceptual feature extraction unit, configured to extract a perceptual feature vector from multimodal perceptual data;

[0040] The feature fusion unit, with a built-in attention model, is configured to use the structured feature vector to guide the alignment of the perceptual feature vector, perform cross-modal feature fusion, and obtain a fused feature representation.

[0041] Preferably, the feature fusion unit is further configured to:

[0042] Mapping the structured feature vectors received from the graph embedding unit into a set of key vectors and a set of value vectors, respectively, to describe the structural information of the physical topology of the device;

[0043] Transforming the perception feature vectors received from the perception feature extraction unit into a set of query vectors for representing the real-time status collected from the device;

[0044] Among them, its built-in attention model is configured to: drive the query vector to explore the key vector to determine the correlation between the perception feature vector and the device physical topology, and dynamically weight the value vector based on the correlation to aggregate and form a fused feature representation.

[0045] Preferably, the feature processing module is further configured to be trained by optimizing objectives, wherein the optimization objectives include:

[0046] For each of the multiple modes, a corresponding modal feature distribution is generated within the feature processing module, thereby obtaining a modal feature distribution set;

[0047] Construct a mixed center distribution, which is the arithmetic mean of all distributions in the set of modal feature distributions;

[0048] The Kullback-Leibler divergence of each modal feature distribution tends to the mixed center distribution to form a joint distribution divergence loss, and optimization is performed based on this loss.

[0049] Preferably, the optimization objective further includes a stability constraint loss, wherein the feature processing module is further configured to:

[0050] Obtain the gradient norm from its internal feature transformation function, and obtain the stability constraint loss based on the gradient norm and the preset Lipschitz constant;

[0051] The stability constraint loss and the joint distribution divergence loss are weightedly combined to form a combined optimization objective, and training is completed based on the combined optimization objective.

[0052] Preferably, the feature fusion unit is further configured to:

[0053] Based on the device physical topology map received from the knowledge parsing module, a spatial constraint matrix is ​​generated to calibrate the physical adjacency relationships that are allowed to be associated between device nodes;

[0054] Among them, its built-in attention model applies a spatial constraint matrix to shield invalid associations between physically non-adjacent nodes when determining the association degree to obtain the association degree.

[0055] Preferably, the result generation module is further configured to:

[0056] Decoding the fused feature representation received from the feature processing module to generate a defect localization mask indicating the spatial location of the defect;

[0057] Perform associative reasoning between the fused feature representation and the device's physical topology to trace the physical root cause of the defect and generate a root cause diagnosis report that includes the confidence level of the faulty component;

[0058] Integrate the defect location mask and root cause diagnosis report to obtain the defect detection results.

[0059] Beneficial effects: By using the physical topology of the device as a priori knowledge and constructing an attention mechanism with structured features as keys and values ​​and perceptual features as queries, the pseudo-alignment problem is solved. Semantic-level alignment is performed under physical constraints, allowing for the association of thermal anomalies with specific circuit nodes, thereby improving the accuracy of defect location. By introducing a joint distribution optimization objective based on the Jensen-Shannon divergence and a Lipschitz stability constraint, the problem of metric unification and physical authenticity in the feature space is solved. This combined optimization objective ensures that heterogeneous modal features can be stably and smoothly mapped and aligned in the shared space, freeing the model from black-box characteristics and enhancing robustness. It can significantly improve the accuracy of defect detection and generate a physically interpretable root cause diagnosis report. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 It is a flow chart of the present invention for obtaining fusion feature representation.

[0062] Figure 3 It is a flowchart of the present invention for optimizing the attention model.

[0063] Figure 4 It is a flowchart of another embodiment of the present invention for optimizing the attention model. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following Figures 1 to 4 The present invention will be described in further detail. It should be noted that the specific embodiments herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention. The following embodiments and the technical features therein may be combined with each other without conflict.

[0065] The applicant has found that existing fusion methods, whether based on statistical correlation (such as CCA) or distribution alignment based on adversarial training, essentially remain at the distribution matching level of the data, while ignoring the hard constraints of the inherent physical topology of the device on the association of multimodal information. For example, the physical root of the abnormally high temperature area on the infrared thermal map must correspond to one or several specific electronic components on the circuit topology map. Due to the lack of guidance from this topological knowledge, the existing methods may mistakenly associate the thermal anomaly with physically unrelated adjacent areas, resulting in a feature fusion that is superficially consistent. This alignment method cannot go deep into the physical semantic level, causing the understanding and positioning of defects to deviate from physical reality. Especially when faced with minor defects or coupled faults, it is very easy to make misjudgments and missed detections.

[0066] Furthermore, industrial multimodal data (such as the local pixel grid of images, the global abstract semantics of text, and the non-Euclidean graph structure of device topology) is inherently heterogeneous. Traditional metrics such as Euclidean distance often fail when forcibly projecting them into the same feature space. More critically, existing deep learning fusion models act like black boxes, their internal feature transformations lacking physical constraints. For example, the model may learn a non-smooth mapping, causing small, continuous changes in input temperature to cause sharp, discontinuous jumps in the feature space, completely contradicting the physical reality of thermodynamic conduction. This lack of physical stability and interpretability in feature representations not only reduces the model's generalization and robustness, but also makes its diagnostic results difficult for domain experts to trust and adopt, limiting its practical application in safety-critical industrial scenarios. To address this issue, the following method was developed.

[0067] Example 1 describes the construction and operation process of a defect detection system guided by structural knowledge. Taking general industrial equipment as the object, the processing flow of the present invention is explained. Specifically, it includes the following steps:

[0068] Step S100: Parse the device structured document to construct a device physical topology map; and use a graph neural network to process the device physical topology map to generate a structured feature vector.

[0069] In this embodiment, the steps are used to convert prior knowledge representing the physical structure of the device into digital features that can be processed by the deep learning model.

[0070] In this embodiment, the device structured document refers to a standardized engineering document that can describe device components and their relationships, such as a computer-aided design model, a circuit schematic, a bill of materials (BOM), and the like.

[0071] A physical topology graph (i.e., a device physical topology graph) is a mathematical graph structure in which nodes represent physical components of a device and edges represent physical or functional connections between components.

[0072] The structured feature vector is a low-dimensional dense vector generated by a graph neural network that contains topological graph structure information.

[0073] In some embodiments, the process is as follows:

[0074] Obtain structured documentation for the device to be tested from the device manufacturer or operator. Use software tools to automatically or semi-automatically parse these documents, extract key components as nodes, and the connections between components as edges to construct a physical topology diagram of the device.

[0075] Graph neural networks (GNNs), such as graph convolutional networks (GCNs) or graph attention networks (GATs), are used to process the constructed physical topology graph. By aggregating information about neighboring nodes layer by layer, GNNs generate a structured feature vector for each node in the graph that represents its properties and position within the entire topology.

[0076] The physical structure of a device is objective and extremely valuable prior knowledge, determining the physical laws governing the occurrence and transmission of faults. Converting this structured knowledge in non-Euclidean space into vector form through GNNs is a prerequisite for enabling subsequent neural networks to understand and utilize these physical laws.

[0077] Step S200: Collect multimodal perception data from industrial equipment and extract perception feature vectors from the multimodal perception data.

[0078] In this embodiment, this step aims to acquire raw data from the external world that represents the real-time operating status of the device and extract its high-level semantic features. Multimodal sensory data refers to data from different sources or types that collectively describe the same detection object. Examples include visible light images, infrared thermal images, and operation log text. A sensory feature vector is a high-dimensional abstract vector representing the core content of the data, obtained by processing the raw sensory data using a feature extraction network.

[0079] In this embodiment, data is collected synchronously by various sensors deployed at the industrial site (such as industrial cameras and thermal imagers), and text data is obtained through technologies such as optical character recognition (OCR).

[0080] For different types of data, appropriate deep learning feature extractors are used. For example, pre-trained convolutional neural networks (CNNs), such as ResNet, are used to extract visual features from images and heatmaps; pre-trained language models, such as BERT, are used to extract semantic features from text records.

[0081] In some embodiments, different feature extraction networks can be selected according to the specific detection task and data characteristics, such as EfficientNet for high-resolution images, or long short-term memory network LSTM for sequence data.

[0082] Step S300: In the attention model, the structured feature vector is used to guide the alignment of the perceptual feature vector to perform cross-modal feature fusion, thereby producing a fused feature representation.

[0083] In this embodiment, this step is the core bridge connecting physical knowledge and perception information, which realizes a knowledge-guided feature fusion mechanism.

[0084] The attention model can dynamically and selectively focus on the most relevant parts of the input information. In this paper, it specifically refers to a fusion model with the Transformer architecture as its core.

[0085] The fusion feature representation is used to effectively combine feature vectors from multiple modalities and structured feature vectors to produce a unified and more information-rich feature vector.

[0086] The structured feature vector generated in step S100 and the perceptual feature vector generated in step S200 are input into the Transformer-based attention model.

[0087] Within this model, through its core cross-attention mechanism, the structural feature vector serves as a knowledge reference system that provides contextual background. Perceptual feature vectors are then weighted and aggregated under this reference system. This allows the model to prioritize associations that align with physical structure when fusing perceptual features.

[0088] This step aims to address the blindness of traditional fusion methods. Rather than simply concatenating or averaging features from different sources, it establishes a more granular fusion model. Perceptual information (for example, an anomalous hotspot) must be guided by physical topological knowledge to find its most plausible location, ensuring the logic and physical interpretability of the fusion process.

[0089] Step S400: generating defect detection results based on the fused feature representation.

[0090] This step of the present embodiment is used to convert the abstract fusion feature representation within the model into a specific, actionable detection conclusion for the end user.

[0091] Defect detection results refer to the final judgment on whether a device has defects, where the defects are located, and what type they are. They can be in the form of classification labels, bounding boxes, segmentation masks, etc.

[0092] The fused feature representation produced in step S300 is fed into one or more task head networks.

[0093] For example, the classification head (usually composed of fully connected layers) can be used to determine the category of defects; the segmentation head (usually composed of upsampling and convolutional layers) can be used to generate pixel-level defect location masks on the original image.

[0094] Since the input fusion feature representation already contains physical topology knowledge and multimodal perception information, the defect detection results decoded by it have improved accuracy and reliability compared with single modality or traditional fusion methods.

[0095] Example 2: This example uses electroluminescent (EL) image defect detection during photovoltaic (PV) module production as a specific application scenario and details the core fusion architecture. Building on the overall framework disclosed in Example 1 (the preferred embodiment), this example focuses on leveraging knowledge of PV module circuit topology to guide the deep fusion of EL image and infrared thermal image features, enabling accurate identification of defects such as dark spots and cold solder joints.

[0096] Reference Figure 1 The method disclosed in this embodiment specifically includes the following steps:

[0097] Step S100: Parse the device structured document to construct a device physical topology map; and use a graph neural network to process the device physical topology map to generate a structured feature vector.

[0098] The main purpose of this step is to convert the electrical connection relationship within the photovoltaic module into a graph data structure and extract its topological features. In this embodiment, the physical topology diagram of the device is the circuit connection diagram of the photovoltaic module. Each solar cell is regarded as a node, and the series or parallel relationship between the cells is represented by edges. The attributes of the node may include its coordinates in the component, nominal power, material batch, etc. The attribute of the edge may be the connection resistance value. The knowledge source is the layout file LayoutFile and the electrical wiring diagram of the photovoltaic module of this model. The process of graph construction is to automatically identify the position of each cell and create nodes by parsing the layout file. By parsing the electrical wiring diagram, the connection relationship between cells and between cells and bus bars is determined, and corresponding edges are created to construct a physical topology graph G representing the electrical topology of the entire component. The process of knowledge embedding is to use the graph attention network GAT to process the graph G to generate a 512-dimensional structured feature vector for each cell node.

[0099] Step S200: Collect multimodal perception data from industrial equipment and extract perception feature vectors from the multimodal perception data.

[0100] In this embodiment, the multimodal sensing data mainly includes electroluminescence (EL) images and infrared (IR) thermal images.

[0101] On the production line, a forward bias is applied to the photovoltaic module, and a dedicated near-infrared camera is used to capture its EL image (IEL). This image can directly reflect internal defects in the cell (such as dark spots, black cores, and cracks). Simultaneously, an infrared thermal imager is used to capture the module's temperature distribution (TIR) ​​during operation.

[0102] Extracting EL image features,Considering that the texture features of EL images are relatively simple, a lightweight convolutional neural network (CNN), such as MobileNetV3 or a customized shallow CNN, can be used to process the EL image and extract its perceptual feature vector.

[0103] Extract infrared image features, use lightweight CNN to process infrared thermal images, and extract perceptual feature vectors related to their temperature distribution.

[0104] Step S300: In an attention model, the structured feature vector is used to guide the alignment of the perceptual feature vector to perform cross-modal feature fusion, thereby generating a fused feature representation.

[0105] Through a novel attention mechanism configuration, visual perception information (EL and IR images) is fused under the guidance of circuit topology knowledge.

[0106] Construct a Transformer fusion unit. Its core cross-attention mechanism is configured as follows:

[0107] The photovoltaic module structured feature vector generated in step S100 (representing the position and connection relationship of each cell in the circuit) is used as the key (Key, K) and value (Value, V) of the attention mechanism.

[0108] The EL image perception features and the IR image perception features extracted in step S200 are concatenated or added to serve as a query (Query, Q).

[0109] The motivation for this configuration is that the cause and severity of dark spots (defects) on the EL image are closely related to their location in the circuit. For example, a cell failure at the end of a branch will have different impacts on the performance of the entire module than a cell failure in the middle of the branch.

[0110] By using a query representing image features to query a key representing the circuit topology, the model can calculate which specific cell node in the circuit has the highest correlation with the features of a certain area in the image. For example, when the query comes from a dark area in the image, the attention mechanism is directed to the key of the cell node corresponding to that area and obtains the value of that node (i.e., its structured feature vector).

[0111] In this way, the obtained fused feature representation contains both information about what the defect looks like (from Q) and where the defect is in the circuit (from K and V), achieving semantic-level alignment rather than simple pixel superposition.

[0112] Step S400: Generate a defect detection result based on the fused feature representation. In this embodiment, the final defect detection result not only indicates the defect location, but also associates it with the circuit attributes.

[0113] The fused feature representation generated in step S300 is input into a fully connected classifier and regressor.

[0114] The classifier outputs the defect type (e.g., cell breakage, solder joint failure, bypass diode failure).

[0115] The regressor outputs an accurate bounding box for the defect.

[0116] The system detected a dark electroluminescent spot on a cell. It first located the circuit node corresponding to that spot in the 3D module model (for example, cell 5 in series branch 3). It then simultaneously analyzed the thermal imaging data (no obvious temperature anomalies) and the EL image features (the presence of an irregular dark area). The knowledge graph automatically correlated the circuit characteristics at that location. The Transformer architecture then integrated this information, not only noting the defective area but also outputting a diagnostic conclusion: microcracks were present in cell C3-5, with a 96% confidence level.

[0117] Through the above steps, this embodiment can effectively distinguish whether it is a problem with a single cell or a series connection problem caused by the welding ribbon or bus bar, greatly improving the accuracy and automation level of photovoltaic module defect detection.

[0118] Example 3: Taking the multimodal nondestructive testing of aircraft engine turbine blades as the background, the complete optimization process for training the detection model is described in detail.

[0119] Steps S100-S200, knowledge graph construction and multimodal feature extraction. In this embodiment, steps similar to those in the previous embodiment are first performed.

[0120] Build a knowledge graph: Based on the blade's finite element analysis (FEA) model and 3D CAD file, a topology graph is constructed. Nodes represent finite element mesh elements on the blade, and edge weights represent the stress conduction relationships between elements. Node attributes can include design stress and temperature distribution at that location.

[0121] Extracting multimodal data and features: Synchronously collect X-ray images of the blade, visible light images of the surface coating, and data from the spindle vibration sensor during operation. Use appropriate feature extractors (e.g., CNN for images, time series network for vibration signals) to extract the respective perceptual feature vectors.

[0122] Step S300, (training phase) optimizes the attention model using a combined optimization objective.

[0123] In this embodiment, the detailed systematic optimization process is key to the model's ability to learn a unified and physically consistent feature representation. The combined optimization objective is a weighted combination of multiple loss functions with different objectives, resulting in an overall optimization goal. In this invention, it drives the update of model parameters toward low reconstruction error, consistent distribution across modalities, and stable transformations.

[0124] In each training iteration, the forward propagation process of the model is similar to that of Example 2, and the feature representation of each modality is obtained. θ L recon +λ1L JSD +λ2L Lip to calculate the total loss and perform backpropagation.

[0125] Calculate the joint distribution divergence loss LJSD:

[0126] First, the characteristic distribution F of the three modes of X-ray, visible light and vibration signal is X ,F V ,F Vib , constructed into a modal feature distribution set within the model.

[0127] Then, the arithmetic mean of all distributions in the set is calculated to construct the mixed center distribution F*=(1 / 3)(F X +F V +F Vib ).

[0128] Finally, the KL divergence of each modal feature distribution toward the central distribution of the mixture is measured, and all KL divergence values ​​are integrated (for example, averaged) to obtain the joint distribution divergence loss LJSD. This loss term serves the purpose of pulling heterogeneous data from different physical sources into a unified, shared semantic space at the information theory level, so that, for example, a tiny crack in an X-ray image can be mapped to a specific harmonic in the vibration signal at the feature level.

[0129] Calculate the stability constraint loss L Lip : Identify the key feature transformation functions in the model, such as the mapping function f(•) from the original X-ray image to its feature vector.

[0130] During the training process, the gradient norm of the function f(•) is constrained by techniques such as spectral normalization || gradxf(x) || 2 Not exceed a preset Lipschitz constant L (e.g., L=1.5).

[0131] Based on the difference between the gradient norm and L, the stability constraint loss L is derived Lip The motivation for this loss term is that the cracks on the blade are physically continuous, and a small shift in their position on the X-ray image should not cause drastic, unrelated jumps in their feature representation. Imposing the Lipschitz constraint injects this physical continuity prior into the model's learning process, ensuring the model's stability and robustness to small changes.

[0132] The above two core loss terms are combined with the standard reconstruction loss term L recon Perform weighted combination to form the final combined optimization target, and use Adam and other optimizers to update the parameters θ of the entire network based on this target.

[0133] Through the above training steps, the model obtained in this embodiment can not only accurately fuse multi-source data to locate blade microcracks, but more importantly, its internal feature representation is highly ordered and conforms to physical intuition, thereby enabling more reliable predictions of the future expansion path of cracks, reducing the missed detection rate to below 0.3%, and meeting the stringent requirements of predictive maintenance of aircraft engines.

[0134] In this embodiment, the step of optimizing the attention model specifically includes:

[0135] Within the attention model, a corresponding modality feature distribution is generated for each of the multiple modalities;

[0136] Presetting an ordered circular alignment path between multiple modal feature distributions;

[0137] Along the circular alignment path, the Kullback-Leibler divergence of the feature distribution of the subsequent modality relative to the feature distribution of the previous modality is measured in turn, and the divergences of all measurements are accumulated to form a chain alignment loss;

[0138] The attention model is optimized based on the chain alignment loss to drive the distribution of each modality features on the path to form local and global consistency.

[0139] Specifically, an ordered circular alignment path refers to a predefined, one-way, closed-loop alignment sequence between modal feature distributions. For example, for the three modalities of vision (V), thermal (H), and text (T), a valid alignment path might be V→H→T→V. The chained alignment loss is the total loss obtained by summing the KL divergence values ​​between all adjacent modalities along the circular alignment path.

[0140] Before model training begins, a predefined alignment path is established based on domain knowledge or experiments. For example, in circuit breaker diagnosis, it's generally assumed that visible light anomalies are often indicative of thermal anomalies, while textual records summarize the phenomena. Therefore, an alignment path of V→H→T→V can be defined. This means that engineers expect the following: the thermal feature distribution FH should be aligned with the visual feature distribution FV; the text feature distribution FT should be aligned with the thermal feature distribution FH; and, to ensure a global closed-loop, the visual feature distribution FV should also be aligned with the text feature distribution FT.

[0141] In each training iteration, the loss function is calculated as: L chain =KL(F H ∣∣F V )+KL(F T ∣∣F H )+KL(F V ∣∣F T );

[0142] The chain alignment loss L chain (can be combined with other loss terms such as reconstruction loss, Lipschitz loss, etc.) as the overall optimization goal, and backpropagate and update the parameters of the entire attention model.

[0143] This approach establishes an asymmetric, ordered dependency structure for the alignment between modalities, which may be more consistent with physical reality in some industrial scenarios. For example, the evolution of physical events is inherently sequential.

[0144] Through this chain-like constraint, not only is the local feature consistency between adjacent modes guaranteed, but through the final closed-loop constraint, it is ensured that the entire feature space will not produce information drift due to the chain being too long, thus achieving global topological consistency.

[0145] Compared with the center-and-spoke alignment, this ordered ring alignment provides a different inductive bias for model optimization, which may lead to better convergence characteristics and performance on specific tasks.

[0146] In some embodiments, the alignment path can be non-circular. For example, for scenarios where the dominant modality is very clear, a linear, self-contained alignment path (e.g., V→H→T) can be defined. In this case, the loss function will not include the last term.

[0147] The order of the paths can be adjusted and optimized as a hyperparameter. For example, two different path orders, V→H→T→V and V→T→H→V, may lead to different model performance. The optimal one can be selected based on experimental results.

[0148] Example 4, using multimodal intelligent diagnosis of high-voltage circuit breakers in power substations as a specific application scenario, details the complete implementation process of the defect detection method and system disclosed in this invention, based on joint distribution optimization and structural knowledge guidance. This method aims to accurately locate and explainably trace the root causes of early-stage circuit breaker faults (such as internal insulation degradation and minor mechanical deformation). The method specifically includes the following steps:

[0149] Step S401: parse the device structured document to construct a device physical topology map; and use a graph neural network to process the device physical topology map to generate a structured feature vector.

[0150] In this embodiment, this step aims to convert the domain knowledge describing the relationship between the circuit breaker's physical structure and electrical connections into graph-structured data that can be understood and processed by computers, and use graph neural networks (GNNs) to extract its deep structural features as the knowledge skeleton for the subsequent fusion process.

[0151] In this embodiment, the device physical topology graph, consisting of nodes (V) and edges (E), is a graph G = (V, E). This graph mathematically represents the physical composition of the device and the connections between its components. Nodes represent device components (such as main contacts and insulators), while edges represent the physical or electrical connections between them.

[0152] Knowledge extraction is performed based on the 3D CAD assembly model (e.g., STEP format file) and the primary and secondary wiring schematics (e.g., DWG format files) of this high-voltage circuit breaker model. Specifically, by parsing the CAD model and employing an octree spatial partitioning algorithm, the circuit breaker is broken down into hierarchical components. Each component is a node, with attributes such as 3D spatial coordinates and material. Simultaneously, by parsing the electrical wiring diagram, electrical components are extracted as nodes, and connection relationships as edges, with attributes such as rated voltage and current. Finally, the nodes and edges from these two sources are merged to construct a unified physical topology graph G.

[0153] A four-layer graph attention network (GAT) is used to process the constructed physical topology graph G. At layer l, the feature hi(l+1) of node i is a weighted aggregation of the features of its neighboring nodes Ni. The output structured feature vector is preferably set to 512 dimensions to meet the processing requirements of the subsequent attention model.

[0154] Traditional deep learning models cannot directly process device topology knowledge in non-Euclidean space. Constructing this valuable prior knowledge into a knowledge graph and embedding it into vector space using GAT is a critical bridge for infusing physical constraints and domain knowledge into the model. GAT uses an attention mechanism to adaptively learn the importance of different neighboring nodes, generating a structured feature vector that not only incorporates the node's own attributes but also aggregates its context within the entire device topology.

[0155] Step S402 collects multimodal sensory data from industrial equipment and extracts sensory feature vectors from this multimodal sensory data. This data is synchronously acquired from a sensor array, using multi-source, heterogeneous condition monitoring data from circuit breakers. Standardized preprocessing is performed, and high-level semantic features are extracted using a deep neural network designed for the characteristics of different modal data.

[0156] In this embodiment, the multimodal perception data specifically includes: visible light inspection images, infrared thermal imaging videos, equipment operation logs and historical maintenance record texts.

[0157] During operation, a high-resolution industrial camera and a high-precision infrared thermal imager simultaneously capture visible light images (Iraw) and thermal imaging videos (Traw) of key circuit breaker components. The OCR engine parses and associates the corresponding text documents.

[0158] The collected raw data is corrected, calibrated and aligned in time and space to eliminate sensor noise and environmental interference, forming a regular data triple {It, Tt, Textt} as the input of the feature extractor.

[0159] Extracting visual features: The improved ResNet-50 network is used to process the visible light image Icorrected. To enhance the ability to capture irregular shaped defects such as surface microcracks, a deformable convolution layer is inserted after the third residual block of ResNet-50. The calculation process is y(p)=∑ k=1 K w k •x(p+p k +Δp k ).

[0160] y(p) is the output feature at position p. Wk is the weight of the kth sample point in the deformable convolution. P kis the predefined sampling offset. Δp k is the additional offset learned by the network.

[0161] Extracting thermal features includes: using discrete wavelet transform (DWT) and db4 wavelet basis to decompose the calibrated thermal map Tcalib into coefficients of different frequency bands to separate the low-frequency components representing steady-state temperature and the high-frequency components representing transient anomalies, which together constitute the thermal perception features.

[0162] Extract text features, specifically: use the RoBERTa model to process the aligned text records and extract text-aware features containing information such as fault descriptions and operation records.

[0163] In some embodiments, the visual feature extractor can also use other advanced network architectures such as EfficientNet and SwinTransformer. The text feature extractor can also be replaced with BERT or its variants. The preprocessing stage can also introduce more data augmentation strategies, such as random rotation and color jitter, to improve the model's generalization ability.

[0164] Step S403: In the attention model, the structured feature vector is used to guide the alignment of the perceptual feature vector to perform cross-modal feature fusion, thereby generating a fused feature representation.

[0165] This step constructs a knowledge-enhanced attention architecture, using the extracted structured knowledge to guide the deep fusion of the multimodal perception features extracted in step S402.

[0166] Knowledge-enhanced crisscross attention architecture: A multi-head (preferably, h=8) Transformer architecture is constructed as the feature fusion unit. The core crisscross attention calculation process is configured as follows:

[0167] The structured feature vector generated in step S401 is used as the key (Key, K) and value (Value, V) of the attention mechanism.

[0168] The perceptual feature vector generated in step S402 is used as the query (Query, Q).

[0169] The attention calculation formula is CrossAttention(Q,K,V)=softmax(QK T / sqrt(d k )+M mask )V.

[0170] d k is the dimension of the key vector; M mask is the spatial constraint matrix (attention mask) generated based on the physical topology.

[0171] The spatial constraint matrix Mmask is generated based on the physical topology map generated in step S401. If nodes i and j in the topology map are not physically adjacent or have no direct electrical connection, the corresponding values ​​in the matrix are set to large negative numbers. This forces the model to mask physically unreasonable connections when calculating attention.

[0172] Optimization goal of model training: The combined optimization objective function used to train the attention model is min θ L recon +λ1L JSD +λ2L Lip λ1 and λ2 are weight hyperparameters used to balance different loss terms. θ is the total trainable parameter of the entire model. L is the preset Lipschitz constant.

[0173] Among them, the joint distribution divergence loss LJSD is obtained by calculating the KL divergence between the feature distribution of each modality and the mixed center distribution of all modalities, aiming to drive the features of different modalities to be semantically aligned.

[0174] The stability constraint loss LLip is obtained by imposing Lipschitz constraints on the characteristic transformation function in the model, aiming to ensure that the calculation process within the model conforms to the smoothness and continuity of the physical world.

[0175] Preferably, the weight coefficients are set to λ1=0.7, λ2=0.3.

[0176] To address the two major challenges of false alignment and the lack of a unified physical metric in existing technologies, K / V / Q directly materializes the prior knowledge of how perceptual information should query physical structures into a network architecture, achieving deep knowledge guidance. The joint optimization objective provides a solid mathematical foundation for fusion from the perspectives of information theory and differential geometry.

[0177] Step S404: generating a defect detection result based on the fused feature representation.

[0178] In this embodiment, this step is used to convert the fusion feature representation output by the model, which contains rich information, into a diagnosis result that can be directly understood and used by the user and includes specific locations and possible causes.

[0179] A differentiable clustering head is designed at the end of the model to decode the final fused feature representation into a pixel-level defect probability map Mdefect, highlighting potential defect areas to form a defect location mask.

[0180] Extract the fused feature vector hdj corresponding to the defect area.

[0181] The cosine similarity between the defect feature vector and the embedding vector hek of all possible faulty components in the knowledge graph is calculated.

[0182] The similarity score is converted into the posterior probability distribution P(ei|dj) through the Softmax function, which represents the possibility that the root cause of defect dj is component ei under the condition that defect dj is detected.

[0183] Finally, a structured root cause diagnosis report is output.

[0184] For example, in a substation circuit breaker application scenario, when an abnormality is detected in the phase A bushing, the system outputs the following defect detection results:

[0185] Defect location mask: Generates a red highlighted area at the lower end skirt of the phase A bushing in the circuit breaker image.

[0186] The root cause diagnosis report indicated that discharge traces were detected on the phase A bushing, and the associated thermal map showed an abnormal temperature increase of 1.2°C in this area. The root cause tracing results indicated that partial discharge was caused by internal insulation deterioration, with a confidence level of 92%. The associated historical text records indicated that the equipment had previously experienced excessive acetylene content in the oil chromatography.

[0187] Since the feature representation of the present invention is deeply bound to the physical topology map from the beginning of learning, it contains physical semantics, making it possible to translate abstract features back to the root cause of specific physical components.

[0188] Through the above steps, the solution disclosed in this embodiment successfully implements multimodal, explainable fault diagnosis for complex industrial equipment. Test data shows that compared to methods using only visual information, this method improves the comprehensive detection F1-score by approximately 27.6%. Furthermore, due to the effective guidance of knowledge, the number of training samples required for model convergence is reduced by approximately 83%. Its diagnostic results, due to their physical interpretability, are more easily adopted by domain experts, demonstrating significant industrial application value.

[0189] In another embodiment of the present application, the nodes of the device physical topology map include at least one of electrical parameters and material properties;

[0190] The generation process of structured feature vectors includes: using a graph attention network to process the physical topology graph, where the attention coefficient between nodes depends on both the node feature similarity and the physical distance between nodes.

[0191] In another embodiment of the present application, the step of performing cross-modal feature fusion further includes:

[0192] A spatial constraint matrix is ​​generated based on the physical topology of the device. When calculating the association between the query and the key, the spatial constraint matrix is ​​applied to shield invalid associations between physically non-adjacent nodes.

[0193] In another embodiment of the present application, the process of calculating the association degree is specifically as follows: generating a spatial constraint matrix based on the device physical topology diagram, where the spatial constraint matrix is ​​used to calibrate the physical adjacency relationships between device nodes that are allowed to be associated;

[0194] When calculating the association between the query vector and the key vector, the spatial constraint matrix is ​​applied to shield the invalid associations between physically non-adjacent nodes and obtain the association.

[0195] In another embodiment of the present application, the step of driving the query vector to explore the key vector to determine the relevance further includes:

[0196] Generate a physical attenuation attention matrix based on the physical properties of the nodes in the device physical topology. Each element in the physical attenuation attention matrix is ​​a continuous value between 0 and 1, which is used to represent the attenuation degree of the mutual influence between the corresponding node pairs.

[0197] After computing the raw relevance score between the query vector and the key vector, the score is modulated by element-wise multiplication of the raw relevance score with the physical attenuation attention matrix to obtain the final relevance.

[0198] In this embodiment, a soft-constrained physical attenuation attention mechanism is adopted to more finely integrate physical prior knowledge into the attention calculation.

[0199] The physical attenuation attention matrix is ​​a matrix of the same dimension as the attention score matrix. Its elements (Matten)ij are continuous values ​​between [0, 1], which are inversely proportional to the degree of physical influence attenuation between nodes i and j. Values ​​closer to 1 indicate less influence attenuation, while values ​​closer to 0 indicate greater influence attenuation.

[0200] Element-by-element multiplication refers to multiplying the elements of corresponding positions of two matrices of the same dimension to obtain a new matrix of the same dimension, namely the Hadamard product.

[0201] The attenuation matrix generation process is as follows:

[0202] From the physical topology graph, the physical properties between all node pairs (i, j) are extracted. In this embodiment, the spatial distance dij between the nodes in the three-dimensional CAD model is preferably used.

[0203] The value of each element of the physical attenuation attention matrix is ​​calculated by the preset Gaussian attenuation function: (Matten)ij=exp(-d ij 2 / (2σ 2 )); where σ is a hyperparameter that controls the attenuation range. When two nodes physically overlap (dij = 0), the attenuation value is 1; as the distance increases, the attenuation value smoothly approaches 0.

[0204] The attention modulation process is as follows: First, according to the standard attention calculation process, the original, unconstrained relevance score matrix S = QK is obtained T / sqrt(d k ). Then, the original score matrix S is multiplied element-by-element with the physical attenuation attention matrix Matten we generated to obtain the modulated score matrix S′=M atten OS. O represents the Hadamard product. Finally, the modulated score matrix S′ is fed into the Softmax function to calculate the final attention weight.

[0205] This approach is more in line with physical reality. Since the impact varies continuously, it allows the model to focus on nodes that are not directly adjacent but still have weak physical connections. This is crucial for capturing complex cascading failures that require consideration of global stress or temperature fields.

[0206] By changing the constraint from adding a large negative number to multiplying by a decay coefficient from 0 to 1, the attention modulation process becomes smoother and more differentiable, which may be more conducive to the gradient propagation and stable convergence of the model.

[0207] In some embodiments for electrical devices, the attenuation level may be calculated based on the equivalent resistance or impedance between nodes, for example (M atten ) ij =1 / (1+R ij ), where R ij is the equivalent resistance between nodes i and j.

[0208] The attenuation function may also be in the form of other functions, such as an exponential attenuation function or an inverse proportional function, as long as it can reflect the tendency of the influence to attenuate as physical properties (such as distance, resistance) change.

[0209] Embodiment 5: Describe the data processing process of another embodiment.

[0210] Step 1: Multimodal data collection and structured preprocessing

[0211] This step uses an industrial-grade sensor array to achieve synchronous acquisition and spatiotemporal alignment of multi-source data. The visible light vision channel uses a high-resolution camera equipped with a telecentric lens to capture surface images at a rate of 15 frames per second under the control of a trigger signal. The pixel resolution must meet Δx≤5μm / pixel. The thermodynamic channel uses the FLIRA8580 infrared thermal imager. The acquisition frequency is strictly synchronized with the visible light frame rate, and the temperature measurement accuracy must reach ±0.3℃. The text modal data is parsed in real time by the OCR engine to parse the equipment operation log and extract the key fields of the maintenance record containing the timestamp. In the data preprocessing stage, the RGB image is first subjected to dark current correction and nonlinear illumination compensation: I corrected =(I raw -B) / GOM illu , B is the dark field reference, G is the gain matrix, M illu is the light intensity distribution mask, and O is the Hadamard product. The thermal data is unified to the same spatial resolution as the visual data through bicubic interpolation, and temperature drift compensation is performed: T calib =T raw +α(t)•Δt, where α(t) is the rate of change of ambient temperature. After extracting word embeddings from the text data using the BERT model, a time alignment module is used to establish a mapping relationship between the text data and the image sequence. The multimodal tensor output from this step must satisfy the normalization constraint |I_corrected|_L2∈[0.2,0.8].

[0212] Step 2: Device topology knowledge graph construction and embedding

[0213] Based on the equipment CAD model and process documents, a knowledge graph containing physical connection relationships and functional semantics is constructed. The AutoCAD Electrical plug-in is used to parse the electrical schematic diagram and generate a topological graph G = (V, E) with node attributes including position coordinates, rated current, and material conductivity, where the edge weight w ij ∈E represents the connector impedance value. The three-dimensional mechanical structure is imported through the STEP file, and the hierarchical assembly relationship is established using the octree space segmentation algorithm. The knowledge embedding module uses a two-layer graph attention network: h i l+1 =σ(∑ j∈N(i)) α ij W l h j l ), attention coefficient α ij By the node spacing d ij Determined together with functional similarity:

[0214] α ij =(exp(LeakyReLU(a T [Wh i ||Wh j]))) / (∑ k∈N(i) exp(LeakyReLU(a T [Wh i ||Wh k ])))•e^(-βd ij ), the output dimension is set to 512 to match the subsequent Transformer architecture, and optimal topological coverage is achieved when the number of layers L = 4. The knowledge graph update cycle is synchronized with equipment maintenance records to ensure that node attributes reflect the latest status.

[0215] Step 3: Deep multimodal feature extraction and representation learning

[0216] This step uses a heterogeneous network to achieve feature encoding of each modality. The visual branch uses an improved ResNet-50 architecture and inserts a deformable convolution layer after the third residual block to adapt to small defect deformations: y(p)=∑ k=1 K w k •x(p+p k +Δp k ), Δp k Prediction is done through the offset network. The thermal feature extraction uses the frequency domain decomposition method to separate the steady-state and transient components through wavelet transform: T feature =DWT(T calib ,ψ db4 )•W thermal After extracting word vectors using the RoBERTa model for text modality, key description information is aggregated through temporal convolution. All feature output dimensions are unified to d=256 and distribution normalized through LayerNorm. The feature extractor adopts a contrastive learning strategy in the pre-training stage to construct positive sample pairs (x i ,x j +) and negative sample pairs (x i ,x j -), the optimization goal is: L cont =-log[(exp(s(x i ,x j + ) / τ)) / (∑ k=1 N exp(s(x i ,x k - ) / τ))], the best modal discrimination is achieved when the temperature coefficient τ=0.07.

[0217] Step 4: Knowledge-guided cross-modal attention alignment

[0218] Construct a multimodal interaction architecture based on Transformer, the core of which is the knowledge-enhanced cross-attention mechanism.k As Key and Value, each modal feature F_m is used as Query, and the hierarchical attention weight CrossAttention(Q,F,K,H)=softmax((Q(F)K(H) T ) / sqrt(d k )+M mask )V(H),M mask The spatial constraint matrix generated based on the device topology prevents the incorrect association of non-adjacent nodes. A two-way mutual information maximization module is designed: Jensen-Shannon divergence is used to constrain feature distribution consistency within the modality, and InfoNCE loss is used between modalities to enhance semantic alignment: L JSD =1 / 3∑ m=1 3 KL(F m ||1 / 3∑F m ),

[0219] L info NCE=-E[log[(exp(f(x) T f(x + ))) / (exp(f(x) T f(x + ))+∑exp(f(x) T f(x - ))]], the hyperparameters at this stage include the number of attention heads h=8, the hidden layer dimension d k =512, drop rate p=0.1.

[0220] Step 5: Joint distribution optimization and stability control

[0221] The manifold alignment of multimodal features is achieved through differential equation constraints. Constructing joint optimization objective function: min θ L recon +λ1L jSD +λ2L Lip ,

[0222] Reconstruction loss L recon Contains the self-encoding error of each mode, and the Lipschitz constraint term is: L Lip = max(|gradxf(x)|2-L, 0), and spectral normalization techniques are used to ensure that the network meets the stability requirement of L ≤ 1.5. The optimization process uses an alternating training strategy: the knowledge encoder parameters are fixed, the feature extractor is updated, and then the entire network is jointly optimized. Ablation experiments on a semiconductor defect dataset show that when λ1 = 0.7 and λ2 = 0.3, the model achieves the optimal balance between precision and robustness.

[0223] Step 6: Explain defect location and root cause tracing

[0224] The final stage generates pixel-level defect heatmaps and associates them with physical causes. Designing a differentiable clustering head: M defect =∑ c=1 C π c N(μ c ,Σ c ), mixing coefficient π c Dynamically generated by knowledge features. The visual attention area is extracted by Grad-CAM++ algorithm and spatially intersected with the thermal anomaly area. final =GeLU(W g •(A vis OA thermal The root cause tracing module performs probabilistic reasoning based on the knowledge graph:

[0225] P(e i |d j )=(exp(sim(h_e i ,h_(d j )))) / (∑exp(sim(h_e k , h_d j ))), the output results include positioning mask, defect type confidence and associated component failure probability, which meets the interpretability requirements of industrial inspection.

[0226] d j represents the specific defect detected by the model, the jth defect example. i Represents the physical entity that may cause the defect for the i-th component. j Indicates the defect d j The characteristic vector of the region. i Represents the i-th component e i The generated embedding vector.

[0227] In the intelligent operation and maintenance of power equipment, the present invention can solve the problem of multi-source monitoring data fusion of substation equipment. Defects of high-voltage circuit breakers and transformers are often manifested as surface cracks in visible light images, local overheating in infrared images, and abnormal mechanical vibration in ultrasonic detection. By constructing the three-dimensional CAD model of the equipment and the electrical wiring diagram into a knowledge graph, and using the Transformer architecture to synchronously analyze visible light inspection images, infrared thermal imaging videos, and voiceprint signals, cross-modal causal reasoning of mechanical structural anomalies and electrical parameter fluctuations can be achieved. For example, when discharge traces are detected on the surface of the bushing, the system can automatically associate the excessive acetylene content in the oil chromatography analysis data, and generate an interpretable diagnostic report of partial discharge caused by internal insulation degradation, providing a basis for condition-based maintenance decisions and avoiding misjudgments caused by traditional single-sensor detection.

[0228] In terms of engine health management, the present invention can achieve multi-dimensional collaborative diagnosis of blade damage. Blade defects involve multi-modal characterizations such as microcracks (X-ray imaging), coating peeling (visible light images), and aerodynamic performance degradation (pressure sensor data). By using the blade finite element analysis model as a domain knowledge constraint and adopting the mutual information maximization theory to fuse multi-source detection data, a mapping relationship between material stress concentration areas and surface temperature field anomalies is established. When a vibration signal of a specific frequency is detected, the system can automatically associate the microcrack image features at the corresponding position, predict the crack propagation path, and evaluate the remaining life. Compared with the single-modal detection method, the missed detection rate is reduced to below 0.3%, meeting the stringent requirements for engine predictive maintenance.

[0229] The breakthrough of this invention lies in constructing a cross-modal joint optimization framework under the constraints of domain knowledge: 1) Converting the equipment structure topology map into the adjacency matrix of a graph convolutional network to establish an explicit mapping between multimodal features and physical structure; 2) Designing a two-stream Transformer architecture based on mutual information maximization, achieving semantic alignment of text descriptions, thermal anomaly areas, and 3D model components through Jensen-Shannon divergence optimization; 3) Introducing Lipschitz continuity conditions to constrain the feature projection process, ensuring that multimodal associations conform to the physical laws of the equipment's actual operating conditions. This method theoretically solves the problem of mathematical unification for the collaborative representation of heterogeneous data and structured knowledge, and achieves simultaneous improvements in defect location accuracy and annotation efficiency in engineering, providing a new path for building self-explanatory industrial visual inspection systems.

[0230] In complex industrial scenarios, equipment visual defect detection faces the core technical difficulties of scarce labeled data and fragmented cross-modal information. Traditional single-modal image analysis methods rely on manually labeled defect samples, which have the problems of high labeling costs and weak guidance. In particular, for tiny defects or anomalies in non-visible light bands (such as thermodynamic distribution anomalies), a single visual modality is difficult to provide sufficient representation. Although existing cross-modal methods attempt to fuse multi-source data, they lack the embedding of prior knowledge of the physical structure of the equipment, resulting in insufficient semantic consistency in the feature alignment process and false detections and missed detections. The deeper problem is that the heterogeneity of multimodal data makes the traditional Euclidean space metric criterion invalid, and the interpretability requirements of defect positioning in the industrial field limit the direct application of black box models. The present invention addresses the three major technical bottlenecks of multimodal representation mismatch, insufficient utilization of domain knowledge, and lack of physical interpretability, and proposes a knowledge-guided collaborative representation framework.

[0231] The technical solution of this invention constructs a three-stage processing flow of knowledge embedding, cross-modal alignment, and joint optimization. First, through the domain knowledge graph construction module, structured knowledge such as device circuit diagrams and 3D CAD models are converted into topological graphs. Graph convolutional networks (GCNs) are used to extract hierarchical features to form node embedding vectors with physical meaning. Then, a multimodal Transformer architecture is established, the core of which is the improved mutual information maximization module: let the visual modality feature be V∈R d×m , the text modality feature is T∈R d×n , the heat map mode is H∈R d×p The Jensen-Shannon divergence is used to construct a joint distribution constraint, JSD(V,T,H)=1 / 3[KL(V||M)+KL(T||M)+KL(H||M)], where M=(V+T+H) / 3 is the mixture distribution and KL is the Kullback-Leibler divergence. This mathematical model enforces topological consistency among the modalities in a shared subspace while controlling the stability of feature transformations through the Lipschitz continuity condition ‖f(x)-f(y)‖≤L‖xy‖. Key parameters include the number of graph convolutional layers K (typically 3-5), the number of Transformer heads h (recommended 8-12), and the Lipschitz constant L (constrained to be in the range of 1.2-1.8). The optimal configuration of these parameters is determined through domain knowledge validation experiments. During training, the algorithm alternately optimizes the intra-modal reconstruction loss and the cross-modal alignment loss, ultimately outputting a physically interpretable probability map of the defect area.

[0232] The overall technical solution of the present invention follows: After system startup, the equipment CAD model and process documentation are first loaded. A knowledge parsing engine is then used to generate a topological map containing node attributes (such as electrical parameters and material properties) and edge relationships (such as connection methods and spatial orientation). Next, the multimodal acquisition module simultaneously acquires visible light images, infrared thermal images, and maintenance record text, extracting raw features using ResNet, a specific band decomposition algorithm, and the BERT model, respectively. In the core processing phase, a graph convolutional network hierarchically aggregates topological knowledge, and its output serves as the Transformer's key-value pairs, interacting with the multimodal feature query. During this process, the mutual information maximization module dynamically adjusts the contribution weights of each modality. Finally, a defect heat map is generated through differentiable clustering, and combined with the knowledge graph, defect root cause tracing is achieved. The entire process uses end-to-end training, while retaining the editability of the knowledge graph to accommodate different equipment models.

[0233] Taking EL inspection on photovoltaic panels as an example, the implementation process is explained in detail: When the system detects a dark electroluminescent spot on a cell, it first locates the circuit node corresponding to that area (e.g., string branch 3) within the 3D module model. It then simultaneously analyzes the thermal imaging data (an abnormal temperature increase of 0.8°C) and the maintenance record text (historical record of a solder ribbon detachment) at that point. The knowledge graph automatically correlates the current path characteristics upstream and downstream of that location. The Transformer architecture integrates this information, not only marking the defective area but also outputting a probabilistic diagnosis (a loose solder ribbon connection leads to increased local resistance, with a 92% confidence level). During this process, the number of GCN layers is set to 4 to cover the three-level topology of cell, string ribbon, and busbar. A Lipschitz constant constraint of 1.5 ensures a stable mapping of thermodynamic and electrical characteristics.

[0234] Improvements of this solution include: 1) a method of integrating the device structure topology map as a trainable parameter into the multimodal learning framework; 2) a multimodal joint distribution optimization algorithm based on Jensen-Shannon divergence and its Lipschitz constraint implementation; 3) a dynamic mapping mechanism between knowledge graph node attributes and deep learning feature vectors; 4) an end-to-end interpretable feature alignment system architecture for industrial defect detection; 5) a cross-modal attention weight allocation strategy that integrates physical priors.

[0235] In this embodiment, multimodal collaborative representation improves detection performance by 27.6% compared to purely visual methods, as verified by a semiconductor device dataset. Specifically, it improves recall by 41.3% for micron-level crack detection. At the engineering implementation level, the introduction of structured knowledge guidance reduces the number of training samples required for model convergence by 83%, and GPU memory usage during inference is reduced by 62%, meeting the real-time requirements of industrial sites (single-frame processing time <50ms). Furthermore, the constructed knowledge anchors support automated reasoning of defect attributes (e.g., transistor Q2 overheating leading to a loose solder joint), providing semantic annotation for subsequent data closure. These advantages stem directly from the newly added domain knowledge embedding step and joint distribution optimization module. The former encodes device physical constraints through a graph structure, while the latter leverages information theory principles to unify metrics across heterogeneous modalities. Their synergistic effect transcends the traditional approach's reliance on manual annotation.

[0236] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the scope of protection of the present invention.

Claims

1. A defect detection method based on joint distribution optimization and structural knowledge guidance, characterized by: include: Parse device structured documents and build device physical topology diagrams; Use graph neural networks to process the physical topology of the device and generate structured feature vectors; Collect multimodal perception data from industrial equipment and extract perception feature vectors from the multimodal perception data; In the attention model, the structured feature vector is used to guide the alignment of the perceptual feature vector, and cross-modal feature fusion is performed to obtain the fused feature representation; Generate defect detection results based on fused feature representation; Among them, the structured feature vector is used to guide the alignment of the perceptual feature vector to perform cross-modal feature fusion and obtain the fused feature representation, including: Mapping the structured feature vectors into a set of key vectors and a set of value vectors, respectively. The set of key vectors and value vectors are used to describe the structural information of the physical topology of the device; Transform the perception feature vector into a query vector, which is used to represent the real-time status collected from the device; The key vector is explored through the query vector to determine the correlation between the perception feature vector and the device physical topology, and the value vector is dynamically weighted based on the correlation to form a fused feature representation.

2. The method according to claim 1, characterized in that The method also includes the following steps to optimize the attention model: Within the attention model, a corresponding modal feature distribution is generated for each of the multiple modalities, thereby obtaining a modal feature distribution set; Calculate the arithmetic mean of all distributions in the modal feature distribution set and construct the mixed center distribution; Measure the Kullback-Leibler divergence of each modal feature distribution in the modal feature distribution set toward the mixed center distribution, and integrate all the measured divergences into the joint distribution divergence loss; The attention model is optimized based on the joint distribution divergence loss to minimize the topological differences between the feature distributions of each modality.

3. The method according to claim 2, characterized in that The optimization steps also include: Obtain the gradient norm of the feature transformation function in the attention model, and obtain the stability constraint loss based on the gradient norm and the preset Lipschitz constant; The stability constraint loss and the joint distribution divergence loss are weightedly combined to form a combined optimization objective; The attention model is finally optimized based on the combined optimization objective so that the feature transformation function meets the smoothness and continuity requirements of physical laws.

4. The method according to claim 1, wherein The step of probing the key vector with the query vector to determine the relevance further comprises: Generate a spatial constraint matrix based on the device physical topology. The spatial constraint matrix is ​​used to calibrate the physical adjacency relationships that are allowed to occur between device nodes. When calculating the correlation between the query vector and the key vector, the spatial constraint matrix is ​​applied to shield the invalid correlation between physically non-adjacent nodes and obtain the correlation; Based on the fused feature representation, defect detection results are generated, including: Decoding the fused feature representation to generate a defect localization mask that indicates the spatial location of the defect; The features corresponding to the defect location in the fused feature representation are reasoned about in relation to the node attributes in the device physical topology diagram, tracing the physical root cause of the defect and generating a root cause diagnosis report that includes the confidence level of the faulty component. Integrate defect location masks and root cause diagnosis reports to form component defect detection results.

5. A defect detection system based on joint distribution optimization and structural knowledge guidance, characterized in that: include: A knowledge parsing module is configured to parse a device structured document to construct a device physical topology map; a data acquisition module configured to collect multimodal sensing data from industrial equipment; The feature processing module is connected to the knowledge analysis module and the data acquisition module. A result generation module, communicating with the feature processing module, for generating defect detection results based on the fused feature representation; The feature processing module includes: Graph embedding unit, used to process the physical topology graph of the device and generate a structured feature vector; A perceptual feature extraction unit, configured to extract a perceptual feature vector from multimodal perceptual data; The feature fusion unit has a built-in attention model and is configured to use the structured feature vector to guide the alignment of the perceptual feature vector, perform cross-modal feature fusion, and obtain a fused feature representation; The feature fusion unit is further configured as follows: Mapping the structured feature vectors received from the graph embedding unit into a set of key vectors and a set of value vectors, respectively, to describe the structural information of the physical topology of the device; Transforming the perception feature vectors received from the perception feature extraction unit into a set of query vectors for representing the real-time status collected from the device; Among them, its built-in attention model is configured to: drive the query vector to explore the key vector to determine the correlation between the perception feature vector and the device physical topology, and dynamically weight the value vector based on the correlation to aggregate it into a fused feature representation.

6. The system according to claim 5, characterized in that The feature processing module is further configured to be trained by optimizing the following objectives: For each of the multiple modes, a corresponding modal feature distribution is generated within the feature processing module, thereby obtaining a modal feature distribution set; Construct a mixed center distribution, which is the arithmetic mean of all distributions in the set of modal feature distributions; The Kullback-Leibler divergence of each modal feature distribution tends to the mixed center distribution to form a joint distribution divergence loss, and optimization is performed based on this loss.

7. The system according to claim 6, characterized in that The optimization objective also includes a stability constraint loss, wherein the feature processing module is further configured as follows: Obtain the gradient norm from its internal feature transformation function, and obtain the stability constraint loss based on the gradient norm and the preset Lipschitz constant; The stability constraint loss and the joint distribution divergence loss are weightedly combined to form a combined optimization objective, and training is completed based on the combined optimization objective.

8. The system according to claim 7, characterized in that The feature fusion unit is further configured as follows: Based on the device physical topology map received from the knowledge parsing module, a spatial constraint matrix is ​​generated to calibrate the physical adjacency relationships that are allowed to be associated between device nodes; Among them, its built-in attention model uses a spatial constraint matrix to mask invalid associations between physically non-adjacent nodes when determining the association degree, and obtains the association degree; The result generation module is further configured to: Decoding the fused feature representation received from the feature processing module to generate a defect localization mask indicating the spatial location of the defect; Perform associative reasoning between the fused feature representation and the device's physical topology to trace the physical root cause of the defect and generate a root cause diagnosis report that includes the confidence level of the faulty component; Integrate the defect location mask and root cause diagnosis report to obtain the defect detection results.

Images