Industrial equipment fault detection method, device and equipment based on vertical domain large model

Abstract

The application discloses an industrial equipment fault detection method and device based on a vertical domain large model, and relates to the field of artificial intelligence. The method comprises the following steps: obtaining multi-modal data to be detected and preprocessing the multi-modal data into processed data; processing the processed data through a fault detection model to obtain a fault category result; the fault detection model comprises a structure perception enhancement module and a multi-modal fusion module which are connected with each other; the fault detection model is trained based on fault sample data and target pseudo samples; part of the fault sample data is labeled with a fault label result; the target pseudo samples are obtained by taking acquired domain prior data as constraint parameters and according to the fault label result and the fault sample data; the structure perception enhancement module is used for performing feature enhancement processing on structure data to obtain a structure feature vector; and the multi-modal fusion module is used for performing semantic fusion processing on the structure feature vector, time sequence data, image data and text data. The application improves fault recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and equipment for industrial equipment fault detection based on a large vertical domain model. Background Technology

[0002] With the rapid development of industrial technology, industrial equipment, as the core carrier of the manufacturing production system, directly determines production efficiency, product quality, and operational safety. In the process of deepening industrialization and intelligent manufacturing, key equipment such as CNC machine tools, heavy-duty motors, wind power equipment, and rail transit traction systems are developing towards higher integration, higher automation, and higher complexity. Once a malfunction occurs, it can not only lead to production line shutdowns and huge economic losses, but also cause safety accidents such as equipment damage and personnel injuries. Therefore, in order to ensure the continuous and stable operation of industrial production, accurate, efficient, and early diagnosis of industrial equipment malfunctions is particularly important.

[0003] Currently, in industrial production scenarios, the relevant technologies use general large models to diagnose faults in industrial data. However, this model is difficult to adapt to complex equipment structures, variable industrial operating conditions, and complex data modalities in vertical scenarios. Especially when the number of equipment fault data samples is small, using a general large model to diagnose faults results in a high false alarm rate and poor fault identification accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, and device for industrial equipment fault detection based on a large vertical domain model.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a method for industrial equipment fault detection based on a large vertical domain model, including:

[0007] Acquire multimodal data to be detected during the operation of the target industrial equipment;

[0008] The multimodal data to be detected is preprocessed to obtain processed data; the processed data includes: structural data, time series data, image data, and text data.

[0009] The processed data is processed through a fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The fault detection model is trained based on fault sample data and target pseudo-samples. Some of the fault sample data is labeled with fault tags. The target pseudo-samples are obtained by using acquired domain prior data as constraint parameters and based on the fault tag results and fault sample data. The structure-aware enhancement module is used to perform feature enhancement processing on the structure data to obtain a structure feature vector. The multimodal fusion module is used to perform semantic alignment and fusion processing on the structure feature vector, the time-series data, the image data, and the text data.

[0010] Secondly, this application provides an industrial equipment fault detection device based on a vertical domain large model, the device comprising:

[0011] The acquisition module is used to acquire the multimodal data to be detected during the operation of the target industrial equipment;

[0012] The preprocessing module is used to preprocess the multimodal data to be detected to obtain processed data; the processed data includes: structural data, time series data, image data and text data;

[0013] A fault detection module is used to process the processed data through a fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The fault detection model is trained based on fault sample data and target pseudo-samples. Some of the fault sample data is labeled with fault tags. The target pseudo-samples are obtained by using acquired domain prior data as constraint parameters and based on the fault tag results and fault sample data. The structure-aware enhancement module is used to perform feature enhancement processing on the structure data to obtain a structure feature vector. The multimodal fusion module is used to perform semantic alignment and fusion processing on the structure feature vector, the time-series data, the image data, and the text data.

[0014] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the industrial equipment fault detection method based on a vertical domain large model as described above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0016] This application provides a method, apparatus, and device for industrial equipment fault detection based on a vertical domain large model. The method includes: acquiring multimodal data to be detected during the operation of the target industrial equipment; preprocessing the multimodal data to be detected to obtain processed data; the processed data includes structural data, time-series data, image data, and text data; processing the processed data through a fault detection model to obtain fault category results; the fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module; the fault detection model is trained based on fault sample data and target pseudo-samples, where some fault label results are annotated in the fault sample data, and the target pseudo-samples are obtained by using acquired domain prior data as constraint parameters and based on the fault label results and fault sample data; the structure-aware enhancement module is used to perform feature enhancement processing on the structural data to obtain structural feature vectors; the multimodal fusion module is used to perform semantic alignment and fusion processing on the structural feature vectors, time-series data, image data, and text data.

[0017] Compared with existing technologies, this solution acquires and preprocesses multi-dimensional, multi-modal data of the target industrial equipment, including structural, temporal, image, and text data. This accurately covers the complex data modalities in industrial scenarios, providing a comprehensive data source that closely matches the actual operating status of the equipment for subsequent diagnosis. Furthermore, during the model training phase, prior domain data is introduced as constraint parameters, and target pseudo-samples are generated by combining some fault sample data labeled with fault tags. This effectively compensates for the limited number of fault data samples in industrial scenarios, significantly reducing the risk of false alarms due to insufficient samples. Finally, the fault detection model incorporates interconnected structural elements. The system comprises a structure perception enhancement module and a multimodal fusion module. The structure perception enhancement module enhances the structural data and generates structural feature vectors, thereby deeply mining the structural characteristics of the target industrial equipment in the image. This effectively adapts to the structural characteristics of complex equipment and overcomes the limitations of traditional general models in matching complex equipment structures. The multimodal fusion module performs semantic alignment and fusion processing on the structural feature vectors with three other types of data, achieving deep correlation and information complementarity of multi-dimensional data. This avoids information bias in single-modal data and significantly improves fault identification accuracy, providing more reliable fault diagnosis support for the stable operation of industrial equipment. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of the application environment of an industrial equipment fault detection method based on a vertical domain large model in one embodiment of this application;

[0020] Figure 2 A flowchart illustrating an industrial equipment fault detection method based on a large vertical domain model provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating a method for processing processed data through a fault detection model to obtain fault category results, provided as an embodiment of this application;

[0022] Figure 4 A flowchart illustrating a method for obtaining target pseudo-samples according to an embodiment of this application;

[0023] Figure 5 A schematic diagram of the functional modules of an industrial equipment fault detection device based on a large vertical domain model provided in this application embodiment;

[0024] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] In related technologies, equipment fault diagnosis generally relies on a large number of historical fault samples and high-quality labeled data. However, in actual industrial production scenarios, equipment fault data suffers from significant issues such as small sample sizes, class imbalance, and modal heterogeneity. This results in weak generalization ability and high deployment difficulty for traditional supervised learning-based or scenario-specific models. While general-purpose large models are typically used for fault diagnosis of industrial data, these models struggle to adapt to complex equipment structures, variable industrial operating conditions, and complex data modalities. Especially when the number of equipment fault data samples is limited, using general-purpose large models for fault diagnosis leads to a high false alarm rate and poor fault identification accuracy.

[0028] To address the aforementioned shortcomings, this application provides an industrial equipment fault detection method based on a large vertical domain model. Compared with existing technologies, this solution acquires and preprocesses multi-dimensional, multi-modal data of the target industrial equipment, including structural, temporal, image, and textual data. This accurately covers the complex data modalities in industrial scenarios, providing a comprehensive data source that closely matches the actual operating state of the equipment for subsequent diagnosis. Furthermore, during the model training phase, prior domain data is introduced as constraint parameters, and target pseudo-samples are generated by combining some fault sample data labeled with fault tags. This effectively compensates for the limited number of fault data samples in industrial scenarios, significantly reducing the risk of false alarms due to insufficient samples. Finally, the fault detection model incorporates interconnected structural elements. The system comprises a structure perception enhancement module and a multimodal fusion module. The structure perception enhancement module enhances the structural data and generates structural feature vectors, thereby deeply mining the structural characteristics of the target industrial equipment in the image. This effectively adapts to the structural characteristics of complex equipment and overcomes the limitations of traditional general models in matching complex equipment structures. The multimodal fusion module performs semantic alignment and fusion processing on the structural feature vectors with three other types of data, achieving deep correlation and information complementarity of multi-dimensional data. This avoids information bias in single-modal data and significantly improves fault identification accuracy, providing more reliable fault diagnosis support for the stable operation of industrial equipment.

[0029] This application provides an industrial equipment fault detection method based on a large vertical domain model, which can be applied to, for example... Figure 1 The application environment of the industrial equipment fault detection method based on a vertical domain large model is shown. This application environment includes a terminal 102, a server 104, and a data storage system. The terminal 102 communicates with the server 104 via a network. The data storage system stores the multimodal data to be detected during the operation of the target industrial equipment acquired by the server 104. The data storage system can be set up independently, integrated into the server 104, or placed in the cloud or on another server. The terminal 102 can send the acquired multimodal data to be detected to the server 104. After receiving the multimodal data, the server 104 performs preprocessing and then processes it through the fault detection model to obtain the fault category result. Furthermore, in some embodiments, the industrial equipment fault detection method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform preprocessing and then process it through the fault detection model to obtain the fault category result. This fault detection model can be understood as a vertical domain large model, adaptable to vertical domain scenarios such as complex equipment structures, variable industrial operating conditions, and complex data modalities.

[0030] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0031] In one exemplary embodiment, such as Figure 2 As shown, a method for industrial equipment fault detection based on a large vertical domain model is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S201 to S203. Wherein:

[0032] Step S201: Obtain the multimodal data to be detected during the operation of the target industrial equipment;

[0033] It should be noted that multimodal data covering the equipment status is collected from the target industrial equipment, including raw structural data, raw time-series data, raw image data, and raw text data. This multimodal data can be, for example, time-series signals, equipment images, text descriptions, operation and maintenance logs, etc. It can also include other industrial modal information such as audio data, video data, radar maps, etc. This embodiment does not impose any limitations on the various industrial modalities of the multimodal data.

[0034] The aforementioned structural data may include an equipment Bill of Materials (BOM) and a topology diagram. This topology diagram records the hierarchical relationship of "equipment → component → sub-component" and the functional connections between components. For example, the hierarchical relationship could be "CNC machine tool → spindle system → bearing," and the functional connections between components could be, for example, the transmission connection between a bearing and a gear. The data format of the structural data may include graphical formats or structured table formats, used for subsequent modeling of the equipment's physical structure. Time-series data is obtained in real-time from sensors, collecting dynamic signals during the operation of the target industrial equipment, including temperature, current, vibration, and pressure. The data format can be represented as "timestamp + numerical sequence," for example, collecting one vibration value every 10ms to form a 1-hour time-series sequence.

[0035] Image data refers to visual information about the appearance and internal components of target industrial equipment acquired through industrial cameras, infrared thermal imagers, etc. This can include surface defect images and infrared thermal images. Surface defect images include optical images of gear tooth wear, and infrared thermal images include the distribution of heating areas in motors. The data format can be represented by a pixel matrix, such as a 256×256 pixel grayscale / color image, used to capture visually visible fault features of the target industrial equipment, such as cracks, deformation, and areas of abnormal temperature. Text data refers to unstructured text collected during equipment operation and maintenance, which can include maintenance records, abnormal alarm logs, etc. Its data format can be string text, containing semantic information such as fault phenomena, handling methods, and fault codes, used to supplement empirical features that mechanistic data cannot cover.

[0036] By collecting multi-dimensional data in this step, we can obtain more comprehensive fault characteristics of the target industrial equipment, providing good data guidance for subsequent multimodal fusion.

[0037] Step S202: Preprocess the multimodal data to be detected to obtain processed data; the processed data includes: structural data, time series data, image data and text data.

[0038] Understandably, the original multimodal data to be detected has problems such as heterogeneous format, asynchronous timing, and missing data. For example, vibration data timestamps are at 10ms intervals, while temperature data is at 100ms intervals; text data has typos, and time series data has null values. Therefore, it is necessary to preprocess the data to transform it into processed data. The processed data can be standardized data with a unified format, high quality, and can be directly input into the model.

[0039] In one embodiment, the multimodal data to be detected is preprocessed to obtain processed data, including: performing timestamp alignment processing on the multimodal data to be detected to obtain aligned data; performing null value imputation processing on the aligned data to obtain imputed data; the imputed data includes temporal imputed data and other imputed data; performing segmentation and slicing processing on the temporal imputed data to obtain multiple continuous segments of fixed length; and performing feature standardization processing based on the continuous segments and other imputed data to obtain processed data.

[0040] Specifically, after acquiring the multimodal data to be detected, data synchronization and format unification can be performed first. This involves aligning the timestamps of the multimodal data to be detected, i.e., time-series synchronization, to obtain aligned data. For example, using high-frequency time-series data as a reference, low-frequency data can be padded with the same timestamps using interpolation, thus ensuring that multimodal data at the same moment correspond to the same device state. High-frequency time-series data could be, for example, vibration data at 10ms intervals, and low-frequency data could be, for example, temperature data at 100ms intervals. The interpolation method can be linear interpolation.

[0041] The aligned data may contain missing values. After determining the aligned data, it undergoes null value imputation to obtain the imputed data. Different imputation methods can be used for different modalities. For missing values ​​in time-series and image data, mean imputation or interpolation imputation can be used. For example, missing values ​​could be 10 null values ​​caused by a temporary sensor malfunction. Mean imputation is suitable for stationary data, while interpolation imputation is suitable for time-series trending data. For ambiguous records in text data, they can be supplemented by associating them with fault codes. For example, if an ambiguous record in text data indicates a spindle abnormality without specifying the phenomenon, and the associated alarm code is F03, it can be supplemented to "Spindle abnormality: bearing wear". The imputed data can include time-series imputed data and other imputed data. Time-series imputed data refers to the imputed time-series data, while other imputed data refers to the imputed data from other modalities.

[0042] Interpolation or sliding window methods are used to segment and slice the time-series imputation data, resulting in multiple fixed-length segments. For example, a sliding window technique can be used to divide one hour of time-series data into 100 subsequences of "6 minutes / segment". After obtaining continuous segments, the continuous segments and other imputation data are subjected to feature normalization processing to obtain processed data. This can involve converting the imputation structured data into graph structured data, scaling the imputation image data to a uniform resolution (e.g., normalizing the pixel values ​​of a 256×256 image to the range of 0-1), and converting the imputation text data into vectors through word segmentation, word embedding, etc., to ensure that different modal data formats are compatible with subsequent encoders.

[0043] This step involves performing timestamp alignment, null value imputation, segmentation, and feature standardization on the multimodal data to be tested. This eliminates the "format barriers" and "quality defects" of the multimodal data, ensuring that the processed data meets the input requirements of the model. This allows the data to be effectively read and utilized by the model, while accurately covering the complex characteristics of data modes in industrial scenarios. This provides a comprehensive data source that closely matches the actual operating status of the equipment for subsequent diagnostics.

[0044] Step S203: The processed data is processed through a fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The fault detection model is trained based on fault sample data and target pseudo-samples. Some fault sample data is labeled with fault labels. The target pseudo-samples are obtained by using acquired domain prior data as constraint parameters and based on the fault label results and fault sample data. The structure-aware enhancement module is used to perform feature enhancement processing on the structure data to obtain structure feature vectors. The multimodal fusion module is used to perform semantic alignment and fusion processing on the structure feature vectors, time series data, image data, and text data.

[0045] It should be noted that the preprocessed structural data needs to be further transformed into vector features that the model can understand and correlated with other modal data to enhance the physical correlation between fault features. The multimodal fusion module is used to eliminate semantic barriers of multimodal data through modal encoding and cross-attention alignment, allowing multimodal features to collaboratively reflect the fault state and avoiding the limitations of single-modal features.

[0046] In one embodiment, the fault detection model includes a fault classification module connected to a multimodal fusion module. This application also provides a specific implementation method for processing the processed data through the fault detection model to obtain fault category results; please refer to [link to relevant documentation]. Figure 3 As shown, the method includes:

[0047] Step S301: The structural data is processed by the structure perception enhancement module to extract features and enhance component relationships, thereby obtaining a structural feature vector.

[0048] The processed data includes structural data, text data, image data, and time-series data. After obtaining the processed data, the structural data needs to undergo feature extraction processing through the structure-aware enhancement module, and the hierarchical structural data needs to undergo component relationship enhancement processing to obtain structural feature vectors.

[0049] Optionally, in the process of extracting features from structural data and strengthening component relationships through the structure-aware enhancement module to obtain structural feature vectors, the structural data is first topologically modeled to construct a topology graph. The topology graph includes nodes and edges, where nodes represent component information in the device and edges represent structural information between components. The structural information is encoded using a graph neural network to obtain a structural embedding vector. For hierarchical structural data in the structural data, the hierarchical information of the subsystem to which the component belongs and its position information in the topology graph are obtained. A hierarchical position code is generated based on the hierarchical and position information. The hierarchical position code is added to the component information and fused with the component information to obtain a structural fusion vector. The structural fusion vector and the structural embedding vector are then processed by an attention layer for attention allocation to obtain the structural feature vector.

[0050] Specifically, the processed data includes structural data, text data, image data, and time-series data. After obtaining the processed data, the structural data needs to be feature-encoded. First, the device topology in the structural data is converted into a graph structure G=(V,E), where V is a node representing the component information in the device, and E is a connecting edge representing the component information in the device. The component information can be the attribute features of each node. For example, for a bearing node, its component information can include model, rated speed, and material. Connection features can also be added to each edge, such as transmission connection, fixed connection, etc.

[0051] In this embodiment, by modeling the equipment topology as a graph structure, the physical dependencies of core components can be preserved, enabling subsequent models to more comprehensively capture the graph structure relationships of fault propagation and avoiding misjudgments caused by relying solely on the features of a single component. Furthermore, the use of graph neural networks (such as GCN / GAT) for node feature aggregation captures the dependencies and propagation path characteristics between various components of industrial equipment, strengthening the correlation between local and global features. The graph convolutional network (GCN) model can be used to achieve weighted aggregation of neighborhood node features, and the graph attention network (GAT) can be used to assign higher attention weights to critical paths, thereby effectively identifying component-related features and improving fault identification accuracy. Combining hierarchical positional encoding and attention fusion mechanisms, hierarchical positional encoding can distinguish the feature differences of components at different functional levels, and the attention fusion mechanism dynamically adjusts the global and local feature weights, focusing on fault-sensitive areas. This allows the model to consider the spatial hierarchy and functional coupling of the equipment while extracting features, thus obtaining a consistent structural representation from global to local. Compared to models based solely on single-mode signals, the structural perception enhancement module provided in this application can improve fault identification accuracy by approximately 8% to 15%, and significantly reduce the false positive rate, especially in complex equipment (such as multi-stage transmission and thermoelectric coupling systems), providing a precise structural feature basis for multimodal fusion.

[0052] It should be noted that the above-mentioned structural feature vectors refer to multi-dimensional vectors formed after graph network encoding. They are used to comprehensively reflect the topological connection relationship, functional hierarchy and signal interaction strength of equipment components. They are the core structural representation for the model to perform fault identification and causal inference.

[0053] After constructing the graph structure, a graph neural network can be used to encode the structural information in the graph structure. By calculating the association weights between nodes and their neighbors, the node's own attributes are fused with the global connectivity to generate a structural embedding vector S, where S∈R. d Here, d represents the dimension of the structural embedding vector, and R represents the set of real numbers. Each node's vector not only contains its own attributes but also its position and function within the overall structure of the device. For example, the vector of a spindle bearing assembly would reflect structural information such as "connected to the motor drive and linked with gears." This structural embedding vector is encoded using a graph neural network, ensuring that each node's features include the global structural context. This enhances the ability to identify faults in small samples. In subsequent feature extraction processes for time series, images, and text modalities, an attention fusion mechanism is used to calculate the correlation weights between the structural embedding vector and each modal feature, dynamically amplifying the modal information most useful for current fault identification and achieving selective interaction and enhancement of cross-modal features. For example, when a bearing fault occurs, the correlation weight between the bearing node's structural vector and vibration features increases. The graph neural network can include GAT or GCN, or it can be implemented using device path embedding, structural Transformer, or sparse representation methods based on topological constraints.

[0054] After constructing the structural embedding vector, for the hierarchical structural data in the hierarchical structure, a unique hierarchical position code is constructed for each component by first obtaining the hierarchical information of the subsystem to which the component belongs and its position information in the topology graph. This hierarchical position code can effectively represent the relative relationship and spatial layout of the component in the entire equipment structure system. Then, this code is fused with the original component information, and the resulting structural fusion vector not only retains the inherent attributes of the component, but also incorporates the structural context information in which it is located, making the feature expression more holistic and relevant. Through attention layer, attention allocation processing is performed on the structural fusion vector and the structural embedding vector, which can adaptively focus on the structural features that are more discriminative for fault diagnosis and weaken the interference of irrelevant information. The final generated structural feature vector not only accurately captures the hierarchical topological relationship of the equipment, but also strengthens the weight of key structural information through the attention mechanism, providing a more targeted and discriminative structural feature representation for subsequent fault detection, and significantly improving the model's ability to perceive and understand complex equipment structures.

[0055] For example, for data with a hierarchical structure of "equipment → component → sub-component", a hierarchical position code is added to the node. For example, "spindle system" belongs to the second-level component. The hierarchical position code reflects its hierarchical relationship of "higher than bearing and lower than CNC machine tool". After being fused with the original node features, it is input into the model so that the model can perceive the hierarchical relationship between components and thus obtain the structural feature vector.

[0056] In this step, the structural data is encoded using a graph neural network, which can transform the abstract equipment structure into a numerical vector and link it with other modal features. This allows the model to combine the physical structure and hierarchical positional relationships of the equipment when extracting fault features, enhancing the module's ability to perceive the relative positions between components. This avoids the physical logic disconnect caused by relying solely on statistical data features, dynamically amplifies the modal information most useful for current fault identification, and achieves selective interaction and enhancement of cross-modal features.

[0057] Step S302: The multimodal fusion module performs feature extraction, modality missing repair, and semantic alignment on image data, time series data, and text data to obtain a semantically consistent joint representation vector.

[0058] Understandably, the multimodal fusion module is used to achieve cross-modal feature alignment and fusion. Preprocessed time-series data, image data, and text data need to have features extracted separately and combined with structural feature vectors to fuse them into joint features in a unified semantic space. This multimodal fusion module includes, in sequence, an encoder, a missing data repair unit, and a semantic alignment unit. The encoders for each modality of data are different.

[0059] In this embodiment, a multimodal fusion module performs feature extraction, modality missing data repair, and semantic alignment on image data, temporal data, and text data to obtain a semantically consistent joint representation vector. This includes: processing image data, temporal data, and text data respectively through their corresponding encoders to extract image feature vectors, temporal feature vectors, and text feature vectors; determining missing vectors from structural feature vectors, image feature vectors, temporal feature vectors, and text feature vectors through a missing data repair unit, and performing modality repair on the missing vectors to obtain target vectors; the target vectors include: target image vector, target temporal vector, target text vector, and target structural vector; and aligning the target image vector, target temporal vector, target text vector, and target structural vector in a common semantic space through a semantic alignment unit to extract a semantically consistent joint representation vector.

[0060] Specifically, each modality of data is assigned a corresponding encoder, which is used to extract modal features. The encoder for image data can be a CNN / ResNet encoder, the encoder for time-series data can be a Transformer / LSTM encoder, and the encoder for text data can be a BERT / GRU encoder. Image data is input into the CNN / ResNet encoder to extract visual features, resulting in image feature vectors, such as extracting high-temperature region feature vectors from infrared thermal images. Time-series data is input into the Transformer / LSTM encoder to capture temporal trend features, resulting in temporal feature vectors, such as the "2x frequency peak feature vector" from vibration data. Text data is input into the BERT / GRU encoder to extract semantic features, resulting in text feature vectors, such as extracting the keyword vector for "bearing wear" from maintenance records.

[0061] The multimodal fusion module extracts feature vectors for each modality through corresponding encoders, providing accurate foundational data for subsequent feature fusion. This is achieved through complementary enhancements using three types of encoders: ① Encoders for time-series data capture the dynamic trends of fault occurrence, correlate periodic faults, and improve the model's sensitivity to abnormal fluctuations; ② Encoders for image data extract equipment topology and inter-component relationships, enhancing the accuracy of locating conducted faults; ③ Encoders for text / semantic data incorporate expert experience and alarm semantics, filling knowledge gaps that are difficult to quantify. By extracting features from each modality, a more comprehensive understanding of these features can be obtained, improving overall detection accuracy.

[0062] Furthermore, the missing data repair unit identifies and repairs missing vectors through the feature vectors of each modality, effectively addressing the pain point of missing multimodal data in industrial scenarios. This allows for timely acquisition of missing vectors and modal repair, resulting in more complete and comprehensive target vectors. This facilitates subsequent fault detection based on more complete data, significantly improving fault detection accuracy. Subsequently, the target vectors of each modality are unified through attention fusion and semantic alignment mechanisms to reduce modal bias and noise interference, facilitating the acquisition of more effective and reliable modal feature information, further improving fault detection accuracy. The overall solution can improve fault identification accuracy by approximately 10% to 18% and significantly enhance robustness to multi-source heterogeneous data scenarios.

[0063] The aforementioned target vectors may include target feature vectors for each modality. This embodiment also includes a modality missing tolerance mechanism, embodied in a missing data repair unit. After obtaining image feature vectors, temporal feature vectors, and text feature vectors, modality data may be missing. Such missing data often occurs in industrial scenarios due to sensor malfunctions, data transmission interruptions, or other issues. The missing data repair unit determines whether any feature vectors are missing. If no missing data is found, the feature vector of that modality at that time is determined as the target feature vector. For example, if the temporal feature vector is not missing, it is determined as the target structural vector. If missing data is found, the missing vector is determined, and a repair strategy adapted to the characteristics of industrial data is used for modality repair processing to obtain the target feature vector. For example, if an infrared camera malfunction results in no image data, the image data is completed and repaired to obtain the target image vector.

[0064] After obtaining the target vector, a cross-modal attention mechanism is used through a semantic alignment unit to map the target feature vectors of each modality to a common semantic space for alignment, thereby extracting a semantically consistent joint representation vector. For example, the similarity between the vibration temporal feature vector and the text semantic feature vector is calculated, and their vector representations are adjusted to make the vibration peak feature and the bearing wear text description closer in the common space, achieving semantic consistency, and thus outputting a semantically consistent joint representation vector. This joint representation vector integrates the core features of the four modalities of temporal, image, text, and structure, fully reflecting the multi-dimensional features of the fault.

[0065] In this step, multimodal information such as images, time series, text, and structure is used for fault judgment, making the identified information more comprehensive. A dedicated modal encoder is used to extract features and semantic fusion is performed through a cross-modal attention mechanism, which effectively solves the problem of modal semantic inconsistency. A modal missing tolerance mechanism is set up to ensure that judgment can still be made in the case of modal loss or partial signal loss, thereby improving the robustness of the system.

[0066] Optionally, in addition to using cross-modal attention mechanisms for fusion, the multimodal fusion module can also use gated fusion, modal complementary encoders, universal modal mapping spaces, and other methods to achieve equivalent fusion of multimodal data.

[0067] In this embodiment, the collaborative operation of the missing data repair unit and the semantic alignment unit effectively improves the integrity and semantic consistency of multimodal data. The missing data repair unit first performs a comprehensive detection of each feature vector, accurately identifying any potentially missing vectors. For the identified missing vectors, the unit employs a repair strategy adapted to the characteristics of industrial data to perform modal repair, ultimately generating a complete set of target vectors. This ensures the integrity of the multimodal data and avoids information gaps caused by the loss of some modalities. Based on this, the semantic alignment unit maps the four types of target vectors to a unified common semantic space for alignment. By mining potential correlations between different modalities, such as the correspondence between equipment vibration time-series data and component wear images, and semantic matching between fault text descriptions and structural anomaly features, a semantically consistent joint representation vector is extracted. This process not only eliminates semantic bias caused by differences in the sources of multimodal data, but also achieves deep correlation and complementarity of cross-modal information. This allows the fused features to retain the unique information of each modality and form a comprehensive representation with a unified semantic dimension. This provides more comprehensive, consistent and robust input features for subsequent fault detection models, and significantly enhances the model's ability to understand and utilize multi-source heterogeneous data in complex industrial scenarios.

[0068] Step S303: The joint representation vector is processed by the fault classification module to obtain the fault category result.

[0069] After obtaining the joint representation vector, the joint representation vector fully integrates core information from multiple dimensions such as device structure, runtime sequence, appearance images and text descriptions. After semantic alignment, it has the same semantic dimension, which can comprehensively and accurately reflect the overall operating status and potential abnormal characteristics of the device.

[0070] Optionally, the fault classification module mentioned above may include: a fully connected layer, an attention layer, a regularization layer, and a classification layer. The joint representation vector is transformed nonlinearly by the fully connected layer to mine deep fault-related features. The attention layer focuses on key fault features with high discriminative power. A regularization layer (such as Dropout or Batch Normalization) suppresses overfitting and enhances the model's adaptability to data noise and distribution differences in industrial scenarios, resulting in enhanced features. Finally, a classification decision layer maps the enhanced features to specific fault categories based on an industrial fault type system. Multi-class discrimination logic outputs the final fault category result, achieving accurate transformation from fused features to diagnostic conclusions.

[0071] It should be noted that the above fault category results can include fault categories and corresponding confidence levels. Fault categories include, for example, mechanical wear, short circuits, and seal failures. Categories with confidence levels greater than a preset confidence threshold can be identified as the final target fault categories. These fault category results not only clearly pinpoint the equipment fault type, providing a direct and reliable basis for subsequent fault investigation and maintenance plan development, but also, thanks to the support of the previous multimodal data fusion and processing stages, ensure the accuracy and robustness of the classification results. This effectively solves the problems of poor adaptability and high false alarm rates of general models in industrial fault classification, further enhancing the practical value of the overall fault diagnosis system.

[0072] For example, in a small sample scenario with only 100 fault samples, fault detection is performed using both an existing general-purpose large model and the fault detection model in this application. The detection data from different models are then statistically analyzed to obtain the corresponding detection results. The general-purpose large model can include a general-purpose model and a general-purpose visual model. For example, the general-purpose model could be BERT-based, and the general-purpose visual model could be a ResNet50 multimodal fusion model. The detection results can include metrics such as accuracy, precision, recall, F1 score, and false positive rate, as shown in Table 1 below.

[0073] Table 1

[0074]

[0075] As shown in Table 1 above, compared with the existing general large model, the fault detection model in this application has higher accuracy, precision, recall, and F1 score, and a significantly lower false alarm rate. Therefore, compared with the existing general large model, the fault detection model provided in this application has a better fault detection effect.

[0076] This application provides a method for industrial equipment fault detection based on a large vertical domain model. Compared with existing technologies, this solution acquires and preprocesses multi-dimensional multimodal data of the target industrial equipment, including structural, temporal, image, and text data. This accurately covers the complex data modalities in industrial scenarios, providing a comprehensive data source that closely matches the actual operating state of the equipment for subsequent diagnosis. Furthermore, during the model training phase, prior domain data is introduced as constraint parameters. By combining partially labeled fault sample data with structural feature vectors, target pseudo-samples are generated, effectively compensating for the limited number of fault data samples in industrial scenarios and significantly reducing the risk of false alarms due to insufficient samples. The fault detection model incorporates an interconnected structure perception enhancement module and a multimodal fusion module. The structure perception enhancement module enhances structural data and generates structural feature vectors, thereby deeply mining the structural characteristics of the target industrial equipment in the image. This effectively adapts to the structural features of complex equipment, overcoming the limitations of traditional general models that struggle to match complex equipment structures. The multimodal fusion module performs semantic alignment and fusion processing on the structural feature vectors with three other types of data, achieving deep correlation and information complementarity among multi-dimensional data. This avoids information bias from single-modal data, significantly improving fault identification accuracy and providing more reliable fault diagnosis support for the stable operation of industrial equipment.

[0077] In one embodiment, a specific implementation method for constructing a fault detection model is also provided, the method comprising:

[0078] The process involves acquiring target pseudo-samples and original equipment data, preprocessing the original equipment data into fault sample data, including first data labeled with fault tags, second data without labels, and equipment operation data; acquiring equipment attribute data from a pre-defined industry knowledge base and vectorizing the equipment attribute data into knowledge vectors; dividing the knowledge vectors, fault sample data, and target pseudo-samples into training and validation sets according to pre-defined partitioning rules; inputting the training set into the initial model to obtain fault output results; constructing a loss function based on the fault output results and fault tag results; iteratively optimizing the parameters of each module in the initial model using a few-shot learning optimization strategy to minimize the loss function, resulting in a model to be validated; the loss function includes: combined classification loss, feature consistency loss, and prototype constraint loss; and inputting the validation set into the model to be validated for verification, resulting in a fault detection model.

[0079] Optionally, the aforementioned original device data includes data from various modes of the device, which can be imported from external devices, acquired through real-time parameter collection of the device, or obtained from a blockchain or database. In this embodiment, no limitation is placed on the method of acquiring fault sample data.

[0080] Specifically, in the process of acquiring fault sample data, the raw equipment data can be acquired first and preprocessed, including timestamp alignment, null value imputation, feature standardization, and segmentation of the time-series data in the raw equipment data to obtain processed sample data. Then, the processed sample data undergoes sample labeling and cleaning to obtain fault sample data. This processed sample data includes equipment fault data and normal operation data. The equipment fault data is divided into first data and second data. The first data is labeled to form fault labeling results, while the second data is not labeled. The normal operation data, excluding the first and second data, is then retained to form the fault sample data.

[0081] The aforementioned fault labeling results can include fault categories, such as bearing wear faults, and abnormal samples with contradictory labels and features are removed, thus forming a sample set of a very small number of first data with fault labels, a large number of unlabeled second data, and some normal equipment operation data. This provides a data foundation for subsequent semi-supervised pre-training and small sample fine-tuning, enabling the model to maintain high recognition accuracy and robustness even when there are very few fault samples.

[0082] In one embodiment, please refer to Figure 4 As shown, a specific implementation method for obtaining target pseudo-samples is also provided, which includes:

[0083] Step S401: Obtain domain prior data and convert the domain prior data into constraint parameters; the constraint parameters include: condition constraint parameters, latent variable distribution constraint parameters, and multimodal consistency rule parameters.

[0084] In step S402, using constraint parameters as generation conditions, the fault label results, fault sample data, and structural sample vectors are input into the autoencoder for processing to obtain pseudo sample data; the pseudo sample data is configured with weight values ​​and sample parameters.

[0085] Step S403: Calculate the confidence level according to the weight value and sample parameters, filter out pseudo sample data with confidence levels less than the preset threshold from all pseudo sample data, and obtain the target pseudo sample through contrastive learning constraint processing; contrastive learning constraint is used to narrow the semantic distance between pseudo sample data and fault label results.

[0086] Understandably, in small-sample scenarios of industrial equipment fault detection, there are very few real fault samples. Directly training the model will result in insufficient training and poor generalization. Pseudo-samples are needed to supplement the data. In order to avoid pseudo-samples from deviating from the actual industrial scenario, it is necessary to first obtain the domain prior data of equipment faults. This domain prior data can include the physical mechanism, fault mode and statistical law of equipment faults. Physical mechanism, for example, bearing wear is accompanied by vibration at a specific frequency. Fault mode, for example, gear tooth breakage has impact pulse. Statistical law, for example, the range of fault current fluctuation.

[0087] The aforementioned autoencoders can include variational autoencoders (CVAEs) or generative adversarial networks (GANs), and can also employ techniques such as graph augmentation, contrastive learning, random interpolation (e.g., Mixup), and few-shot synthesis methods (e.g., SMOTE) to enhance sample distribution. When constructing small-sample pseudo-samples based on domain prior data, this domain prior data is transformed into specific constraint parameters. These constraint parameters include conditional constraints, latent variable distribution constraints, and multimodal consistency rules. Conditional constraints correspond to fault types, and latent variable distribution constraints conform to statistical characteristics. These constraint parameters are used as generation conditions. Fault label results and fault sample data are input into the autoencoder for processing, guiding the autoencoder to generate pseudo-sample data that conforms to the physical and statistical characteristics of industrial realities. For example, conditional constraints require the generated pseudo-samples to correspond to a specific fault type; for example, motor overload samples must satisfy the simultaneous increase in current and temperature. Latent variable distribution constraints require pseudo-sample values ​​to conform to the statistical range of real faults; for example, vibration peak values ​​must not exceed equipment limits. Multimodal consistency rules include that the linkage of multiple signals such as vibration and temperature conforms to physical logic, avoiding contradictions such as high current but low temperature. These pseudo-samples can fill gaps in real samples, balance sample distribution, and provide AI models with sufficiently reliable learning materials, ultimately improving the accuracy and generalization ability of fault identification in small sample scenarios.

[0088] Taking a small-sample scenario for equipment fault diagnosis as an example, the input parameters of CVAE or GAN can include a small number of real fault samples, providing the generation tool with basic feature templates of real samples to ensure that the pseudo-samples are consistent with real samples in terms of data format and basic features. They can also include constraint parameters derived from domain priors, such as conditional constraint parameters corresponding to "bearing wear faults," latent variable distribution constraint parameters, and multimodal consistency rule parameters. These parameters are embedded in the model structure of the generation tool (such as the conditional layer of CVAE and the generator loss function of GAN), thereby limiting the range of pseudo-sample generation. Random noise or latent variables can also be included, such as the latent variable vector of CVAE and the initial noise vector of the GAN generator, providing randomness for generating diverse pseudo-samples and preventing complete duplication of generated pseudo-samples. Simultaneously, the random noise, guided by the constraint parameters, is transformed into feature signals conforming to industrial laws, outputting pseudo-synthesized fault sample feature vectors, i.e., pseudo-sample data. The output pseudo-samples have the same format and dimensions as the real fault sample data and satisfy the constraint parameters corresponding to the domain prior data. They possess the feature attributes of real fault samples and supplement the quantity gap of real samples, providing sufficient and reliable material for subsequent AI model training.

[0089] After generating pseudo-sample data, a pseudo-sample screening mechanism can be used to control the quality of the pseudo-samples and obtain target pseudo-samples. Each pseudo-sample corresponding to a device can be configured with a corresponding weight value and sample parameters. The weight value is used to characterize the correlation strength between the sample and the real fault mode, and the sample parameters are used to characterize key attributes such as fault occurrence duration and impact range. To ensure the quality of pseudo-samples, the confidence level of each pseudo-sample needs to be calculated based on the weight value and sample parameters to measure its matching degree with the real fault features, and low-quality samples with confidence levels below a preset threshold are filtered out. Subsequently, the remaining samples are further optimized through contrastive learning constraints: by constructing a loss function to narrow the distance between the pseudo-sample data and the corresponding fault label results in the semantic space, the correlation between the pseudo-samples and the real fault categories is strengthened, thereby obtaining target pseudo-samples. For example, this makes the pseudo-samples of the "bearing wear" category closer to the real bearing wear label features in the feature space. The final target pseudo-samples retain the typical features of industrial faults and have the same distribution characteristics as real samples, which can effectively expand the training dataset and improve the generalization ability and diagnostic accuracy of the fault detection model in small sample scenarios.

[0090] To improve model accuracy and generalization ability, industry knowledge can be injected during data training. For example, industry knowledge bases can be acquired, and structures such as rule engines, expert systems, domain ontology graphs, and fault reasoning trees can be used as prior representations to enhance model interpretability and practicality. Taking an industry knowledge base as an example, fault codes, maintenance rules, and causal relationship graphs between faults and components can be extracted from the industry knowledge base, aligned, and encoded into learnable knowledge vectors K∈R. d d represents the dimension of the knowledge vector, used to define the length of the feature in the mathematical space, and R is the set of real numbers. Among them, the fault code is, for example, F01 = current over-limit, the operation and maintenance rule is, for example, "temperature > 80℃ requires shutdown", and the fault and component causal relationship graph is, for example, "bearing wear → abnormal spindle vibration". The knowledge vector corresponding to the fault code F01 is [1,0,0,...].

[0091] Knowledge vectors, fault sample data, and target pseudo-samples are mixed and used as training data for the initial model to solve the problems of insufficient sample size and class imbalance. The data is then divided into training set and validation set according to a preset partitioning rule, such as 8:2.

[0092] The initial model includes: an initial structure-aware enhancement module, an initial multimodal fusion module, and an initial fault classification module connected in sequence. The training set is input into the initial model to obtain fault output results, including: performing structure enhancement processing on the structural sample data of the training set through the initial structure-aware enhancement module to obtain a structural sample vector; performing feature extraction, modality missing repair, and semantic alignment processing on other sample data and the structural sample vector in the training set through the initial multimodal fusion module to obtain a joint sample vector; and classifying the joint sample vector through the initial fault classification module to obtain the fault output results.

[0093] The training set mentioned above may include structural sample data, temporal sample data, text sample data, and image sample data. The structural sample data undergoes structural enhancement processing through an initial structural awareness enhancement module. This involves generating structural vectors and then interacting with modal features through an attention fusion mechanism. Hierarchical positional encoding is added to the nodes of the device topology graph and added to or concatenated with the original node features. This allows the model to simultaneously perceive the functional hierarchy and relative positional relationships of components during feature calculation, thereby outputting structural sample vectors. The temporal sample data, text sample data, and image sample data undergo feature extraction through an initial multimodal fusion module. After encoding by corresponding encoders and modality missing data repair, the results from different modalities are aligned in a common semantic space by a semantic alignment unit to obtain a joint sample vector. This joint sample vector is then processed by an initial fault classification module to classify the faults, resulting in the fault output.

[0094] Optionally, for each type of fault, a class center vector Ck is calculated based on real fault sample data and target pseudo-samples. For example, the class center vector of "bearing wear" can be the mean of all features of that type of sample. The fault category is determined by calculating the distance (Euclidean distance / cosine distance) between the sample to be detected and Ck, thereby improving the recognition accuracy of boundary samples.

[0095] A loss function is constructed based on the fault output and fault label results. This loss function can include classification loss, feature consistency loss, and prototype constraint loss. The classification loss optimizes the accuracy of fault category prediction, the feature consistency loss ensures semantic consistency after multimodal feature fusion, and the prototype constraint loss narrows the distance between samples and class centers, allowing the model to converge quickly and without bias in small sample sizes. Following the minimization of the loss function, a few-shot learning optimization strategy is used to iteratively optimize the parameters of the initial structure perception enhancement module, the initial multimodal fusion module, and the initial fault classification module in the initial model, resulting in a model to be validated. The validation set is then input into the model to be validated to obtain the fault detection model. This few-shot learning optimization strategy can also employ matching networks, relationship networks, class-specific comparison loss, TripletLoss, or attention-based metric networks to achieve similarity-driven learning.

[0096] Understandably, when training a corresponding fault detection model for new devices with similar structural features, a meta-learning module or adapter module can be used to allow the model to transfer and generalize without retraining. This enables the model to quickly adapt to and maintain high performance with minimal new data, thereby significantly reducing training costs and deployment time. Specifically, a meta-learning module, such as the MAML algorithm, can quickly fine-tune model parameters using a small number of new samples; an adapter module could be, for example, inserting a lightweight fully connected layer, fine-tuning only that layer to adapt to the new device.

[0097] This embodiment utilizes industry fault codes, maintenance experience, and knowledge graphs to construct knowledge vectors, which serve as guiding directions for feature extraction. This makes the model output more closely resemble expert experience, reducing spurious relevance and "biased learning," and improving the reliability of the results. Furthermore, domain prior constraints ensure that the generated pseudo-samples conform to industrial physical laws, preventing false samples from misleading the model. Simultaneously, supplementing the sample size ensures the main model has sufficient data to learn fault features. By incorporating knowledge vectors as part of the training data, the training data becomes closer to the actual attributes of the equipment. A few-shot optimization strategy allows the model to learn efficiently and transfer flexibly with limited data, addressing the problem of weak generalization ability in few-shot scenarios.

[0098] Furthermore, after the fault detection model is built through training, it can be lightweighted: through knowledge distillation, quantization to compress the model size, etc., it can be adapted to the embedded edge environment. This ensures that the lightweight model can be deployed on edge devices in the industrial field and run on the edge side, thereby achieving real-time fault diagnosis and avoiding the latency and security risks of data uploading to the cloud. Quantization can be converting 32-bit floating-point numbers to 16-bit, and model size compression can refer to compressing the size from the original 1GB to 200MB. Edge side devices can include PLCs, industrial gateways, edge servers, etc.

[0099] Optionally, edge devices can collect new data in real time, including new fault samples and new operating condition data, and feed the collected new data back to the cloud. The cloud model is periodically updated with the fed-back data to optimize model parameters, adapt to new operating conditions and new fault types, and achieve continuous iteration of the model.

[0100] This embodiment deploys a lightweight fault detection model to edge devices in industrial scenarios, meeting the real-time and hardware adaptation requirements of industrial environments. Continuous learning allows the model to optimize as the equipment operates, preventing a decrease in diagnostic accuracy due to changes in operating conditions. Furthermore, by constructing target pseudo-samples, the problem of insufficient real samples is addressed, expanding the diversity of training data. A class-centric prototype network is introduced to improve the model's discriminative power in categories with ambiguous boundaries. A multi-task loss function is constructed, which, guided by the multi-task loss function, significantly enhances the model's adaptability to small sample classes and its convergence speed. This results in a fault detection model with high accuracy and strong generalization ability.

[0101] Based on the same inventive concept, this application also provides an industrial equipment fault detection device for implementing the above-mentioned fault detection method. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more embodiments of the industrial equipment fault detection device provided below can be found in the limitations of the industrial equipment fault detection method described above, and will not be repeated here.

[0102] In one exemplary embodiment, such as Figure 5 As shown, an industrial equipment fault detection device based on a vertical domain large model is provided. The device includes:

[0103] The acquisition module 510 is used to acquire the multimodal data to be detected during the operation of the target industrial equipment;

[0104] The preprocessing module 520 is used to preprocess the multimodal data to be detected to obtain processed data; the processed data includes: structural data, time series data, image data and text data;

[0105] The fault detection module 530 is used to process the processed data through the fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The fault detection model is trained based on fault sample data and target pseudo-samples. Some fault sample data is labeled with fault labels. The target pseudo-samples are obtained by using acquired domain prior data as constraint parameters and based on the fault label results and fault sample data. The structure-aware enhancement module is used to perform feature enhancement processing on the structural data to obtain structural feature vectors. The multimodal fusion module is used to perform semantic alignment and fusion processing on the structural feature vectors, time series data, image data, and text data.

[0106] As an optional implementation, the fault detection module 530 is specifically used for:

[0107] The structural feature vector is obtained by performing feature extraction and component relationship enhancement on the structural data through the structure awareness enhancement module.

[0108] The multimodal fusion module performs feature extraction, modality missing repair, and semantic alignment on image data, time-series data, and text data to obtain a semantically consistent joint representation vector.

[0109] The joint representation vector is processed by the fault classification module to obtain the fault category result.

[0110] As an optional implementation, the fault detection module 530 is also used for:

[0111] Perform topological modeling on the structural data to construct a topological graph; the topological graph includes nodes and edges, where nodes represent component information in the device and edges represent structural information between components.

[0112] The structural information is encoded using a graph neural network to obtain the structural embedding vector;

[0113] For hierarchical structure data in the structural data, obtain the hierarchical information of the subsystem in which the component is located and its position information in the topology diagram;

[0114] Generate hierarchical location codes based on hierarchical and location information;

[0115] The hierarchical position code is added to the component information and fused with the component information to obtain the structure fusion vector;

[0116] The structural fusion vector and structural embedding vector are processed by attention layer to obtain structural feature vector.

[0117] As an optional implementation, the fault detection module 530 is also used for:

[0118] Image data, time-series data, and text data are processed by their respective encoders to extract image feature vectors, time-series feature vectors, and text feature vectors.

[0119] The missing vector is determined from the structural feature vector, image feature vector, temporal feature vector, and text feature vector by the missing vector repair unit, and modal repair processing is performed on the missing vector to obtain the target vector; the target vector includes: target image vector, target temporal vector, target text vector, and target structural vector;

[0120] The target image vector, target temporal vector, target text vector, and target structure vector are aligned in a common semantic space using a semantic alignment unit, and a semantically consistent joint representation vector is extracted.

[0121] As an optional implementation, the above-described apparatus is further used for:

[0122] Obtain target pseudo-samples and raw equipment data, and preprocess the raw equipment data into fault sample data; the fault sample data includes first data with fault label results, unlabeled second data, and equipment operation data;

[0123] Obtain equipment attribute data from a pre-set industry knowledge base, and vectorize the equipment attribute data into knowledge vectors;

[0124] The knowledge vectors, fault sample data, and target pseudo-samples are divided into training set and validation set according to a preset partitioning rule;

[0125] Input the training set into the initial model to obtain the fault output results;

[0126] Based on the fault output and fault label results, a loss function is constructed. Following the minimization of the loss function, a few-shot learning optimization strategy is used to iteratively optimize the parameters of each module in the initial model to obtain the model to be validated. The loss function includes: combined classification loss, feature consistency loss, and prototype constraint loss.

[0127] The validation set is input into the model to be validated for validation, and the fault detection model is obtained.

[0128] As an optional implementation, the above-described apparatus is further used for:

[0129] Acquire domain prior data and transform it into constraint parameters; the constraint parameters include: conditional constraint parameters, latent variable distribution constraint parameters, and multimodal consistency rule parameters.

[0130] Using constraint parameters as generation conditions, fault label results and fault sample data are input into an autoencoder for processing to obtain pseudo sample data; the pseudo sample data is configured with weight values ​​and sample parameters.

[0131] Based on the weight values ​​and sample parameters, the confidence level is calculated. Pseudo-sample data with confidence levels lower than a preset threshold are filtered out from all pseudo-sample data. The target pseudo-sample is obtained by processing it through contrastive learning constraints. The contrastive learning constraints are used to narrow the semantic distance between the pseudo-sample data and the fault label results.

[0132] As an optional implementation, the above-described apparatus is further used for:

[0133] The structural sample data of the training set is processed by the initial structure-aware enhancement module to obtain structural sample vectors.

[0134] The other sample data and structured sample vectors in the training set are processed by the initial multimodal fusion module for feature extraction, modality missing repair and semantic alignment to obtain a joint sample vector;

[0135] The joint sample vector is classified by the initial fault classification module to obtain the fault output result.

[0136] As an optional implementation, the preprocessing module 520 described above is specifically used for:

[0137] The multimodal data to be detected is timestamped to obtain the aligned data.

[0138] The aligned data is padded with null values ​​to obtain the padded data; the padded data includes time-series padded data and other padded data.

[0139] The time-series filling data is segmented and sliced ​​to obtain multiple continuous segments of fixed length;

[0140] The processed data is obtained by performing feature standardization on continuous segments and other imputed data.

[0141] The industrial equipment fault detection device provided in this application acquires and preprocesses multi-dimensional, multi-modal data of the target industrial equipment, including structural, temporal, image, and text data. This accurately covers the complex data modalities in industrial scenarios, providing a comprehensive data source that closely matches the actual operating state of the equipment for subsequent diagnosis. Furthermore, during the model training phase, prior domain data is introduced as constraint parameters, and target pseudo-samples are generated by combining some fault sample data labeled with fault tags. This effectively compensates for the limited number of fault data samples in industrial scenarios, significantly reducing the risk of false alarms due to insufficient samples. The fault detection model also incorporates... The interconnected structure perception enhancement module and multimodal fusion module enhance structural data and generate structural feature vectors, thereby deeply mining the structural characteristics of the target industrial equipment in the image. This effectively adapts to the structural features of complex equipment, overcoming the limitations of traditional general models in matching complex equipment structures. The multimodal fusion module performs semantic alignment and fusion processing on the structural feature vectors with three other types of data, achieving deep correlation and information complementarity of multi-dimensional data. This avoids information bias in single-modal data, significantly improving fault identification accuracy and providing more reliable fault diagnosis support for the stable operation of industrial equipment.

[0142] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores video tag processing data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a fault detection method for industrial equipment based on a large vertical domain model.

[0143] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0144] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0145] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0146] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0147] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0148] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0149] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0150] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0151] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for fault detection of industrial equipment based on a large vertical domain model, characterized in that, The industrial equipment fault detection method based on a vertical domain large model includes: Acquire multimodal data to be detected during the operation of the target industrial equipment; The multimodal data to be detected is preprocessed to obtain processed data; the processed data includes: structural data, time series data, image data, and text data. The processed data is then processed through a fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The model is trained based on fault sample data and target pseudo-samples. Some fault sample data contains fault label results, and the target pseudo-samples are obtained by using acquired domain prior data as constraint parameters, based on the fault label results and fault sample data. The structure-aware enhancement module performs feature enhancement processing on the structure data to generate structure feature vectors. The multimodal fusion module performs semantic alignment and fusion processing on the structure feature vectors, the time-series data, the image data, and the text data. The fault detection model further includes a fault classification module; the fault classification module is connected to the multimodal fusion module. The processed data is then processed through a fault detection model to obtain fault category results, including: The structural awareness enhancement module performs feature extraction and component relationship enhancement processing on the structural data to obtain a structural feature vector. The multimodal fusion module performs feature extraction, modality missing repair, and semantic alignment on the image data, time-series data, and text data to obtain a semantically consistent joint representation vector. The joint representation vector is processed by the fault classification module to obtain the fault category result; The structure-aware enhancement module performs feature extraction and component relationship enhancement on the structural data to obtain a structural feature vector, including: The structural data is used to perform topological modeling to construct a topological graph; the topological graph includes nodes and edges, the nodes are used to represent component information in the device, and the edges are used to represent structural information between the components; The structural information is encoded using a graph neural network to obtain a structural embedding vector; For the hierarchical structure data in the structural data, obtain the hierarchical information of the subsystem where the component is located and its position information in the topology diagram; Generate a hierarchical location code based on the hierarchical information and the location information; The hierarchical position code is added to the component information and fused with the component information to obtain a structural fusion vector; The structural fusion vector and the structural embedding vector are processed by attention layer for attention allocation to obtain the structural feature vector.

2. The industrial equipment fault detection method based on a large vertical domain model according to claim 1, characterized in that, The multimodal fusion module includes: an encoder, a missing component repair unit, and a semantic alignment unit connected in sequence; The multimodal fusion module performs feature extraction, modality missing repair, and semantic alignment on the image data, time-series data, and text data to obtain a semantically consistent joint representation vector, including: The image data, time-series data, and text data are processed by their respective encoders to extract image feature vectors, time-series feature vectors, and text feature vectors. The missing vector is determined from the structural feature vector, image feature vector, temporal feature vector, and text feature vector by the missing vector repair unit, and modal repair processing is performed on the missing vector to obtain the target vector; the target vector includes: target image vector, target temporal vector, target text vector, and target structural vector; The semantic alignment unit aligns the target image vector, target temporal vector, target text vector, and target structure vector in a common semantic space, and extracts a semantically consistent joint representation vector.

3. The industrial equipment fault detection method based on a large vertical domain model according to claim 1, characterized in that, The fault detection model is constructed through the following steps: Obtain target pseudo-samples and original equipment data, and preprocess the original equipment data into fault sample data; the fault sample data includes first data labeled with fault tags, second data without labels, and equipment operation data; Obtain equipment attribute data from a preset industry knowledge base, and vectorize the equipment attribute data into knowledge vectors; The knowledge vector, the fault sample data, and the target pseudo-sample are divided into a training set and a validation set according to a preset partitioning rule; The training set is input into the initial model to obtain the fault output result; Based on the fault output and fault label results, a loss function is constructed. By minimizing the loss function, a few-shot learning optimization strategy is used to iteratively optimize the parameters of each module in the initial model to obtain the model to be validated. The loss function includes: combined classification loss, feature consistency loss, and prototype constraint loss. The validation set is input into the model to be validated for validation to obtain the fault detection model.

4. The industrial equipment fault detection method based on a large vertical domain model according to claim 3, characterized in that, Obtaining target pseudo-samples includes: Acquire domain prior data and transform the domain prior data into constraint parameters; the constraint parameters include: conditional constraint parameters, latent variable distribution constraint parameters, and multimodal consistency rule parameters. Using the constraint parameters as generation conditions, the fault label results and fault sample data are input into the autoencoder for processing to obtain pseudo sample data; the pseudo sample data is configured with weight values ​​and sample parameters. Based on the weight values ​​and sample parameters, the confidence level is calculated, and pseudo-sample data with confidence levels less than a preset threshold are filtered out from all pseudo-sample data. The target pseudo-sample is obtained by processing it through contrastive learning constraints. The contrastive learning constraints are used to narrow the semantic distance between the pseudo-sample data and the fault label results.

5. The industrial equipment fault detection method based on a large vertical domain model according to claim 3, characterized in that, The initial model includes: an initial structure-aware enhancement module, an initial multimodal fusion module, and an initial fault classification module connected in sequence; the training set is input into the initial model to obtain fault output results, including: The structural sample data of the training set is processed by the initial structure-aware enhancement module to obtain a structural sample vector. The other sample data in the training set and the structured sample vector are processed by the initial multimodal fusion module for feature extraction, modality missing repair and semantic alignment to obtain a joint sample vector; The joint sample vector is classified by the initial fault classification module to obtain the fault output result.

6. The industrial equipment fault detection method based on a large vertical domain model according to claim 1, characterized in that, The multimodal data to be detected is preprocessed to obtain processed data, including: The multimodal data to be detected is timestamped to obtain aligned data. The aligned data is then padded with null values ​​to obtain padded data; the padded data includes time-series padded data and other padded data. The time-series filling data is segmented and sliced ​​to obtain multiple continuous segments of fixed length; The processed data is obtained by performing feature standardization processing based on the continuous segments and the other imputed data.

7. An industrial equipment fault detection device based on a large vertical domain model, characterized in that, The industrial equipment fault detection device based on a vertical domain large model includes: The acquisition module is used to acquire the multimodal data to be detected during the operation of the target industrial equipment; The preprocessing module is used to preprocess the multimodal data to be detected to obtain processed data; the processed data includes: structural data, time series data, image data and text data; A fault detection module is used to process the processed data through a fault detection model to obtain fault category results. The fault detection model includes an interconnected structure-aware enhancement module and a multimodal fusion module. The fault detection model is trained based on fault sample data and target pseudo-samples. Some of the fault sample data is labeled with fault tags. The target pseudo-samples are obtained by using acquired domain prior data as constraint parameters, and are based on the fault tag results and fault sample data. The structure-aware enhancement module performs feature enhancement processing on the structure data to generate structure feature vectors. The multimodal fusion module performs semantic alignment and fusion processing on the structure feature vectors, the time-series data, the image data, and the text data. The fault detection model also includes a fault classification module connected to the multimodal fusion module. The fault detection module is specifically used for: The structural awareness enhancement module performs feature extraction and component relationship enhancement processing on the structural data to obtain a structural feature vector. The multimodal fusion module performs feature extraction, modality missing repair, and semantic alignment on the image data, time-series data, and text data to obtain a semantically consistent joint representation vector. The joint representation vector is processed by the fault classification module to obtain the fault category result; The fault detection module is also used for: The structural data is used to perform topological modeling to construct a topological graph; the topological graph includes nodes and edges, the nodes are used to represent component information in the device, and the edges are used to represent structural information between the components; The structural information is encoded using a graph neural network to obtain a structural embedding vector; For the hierarchical structure data in the structural data, obtain the hierarchical information of the subsystem where the component is located and its position information in the topology diagram; Generate a hierarchical location code based on the hierarchical information and the location information; The hierarchical position code is added to the component information and fused with the component information to obtain a structural fusion vector; The structural fusion vector and the structural embedding vector are processed by attention layer for attention allocation to obtain the structural feature vector.

8. A computer device, comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the industrial equipment fault detection method based on a large vertical domain model as described in any one of claims 1-6.

Images