Fault diagnosis method, system and equipment based on multi-modal information fusion and medium

Abstract

The invention relates to a fault diagnosis method, system and device based on multi-modal information fusion and a medium, and the method comprises the steps: carrying out the feature extraction of pre-collected multi-source heterogeneous data, and obtaining the point cloud feature, the image feature and the vibration feature; calculating a basic probability distribution function for the point cloud features, the image features and the vibration features through a weighted evidence theory, and generating credibility weights of the corresponding features; and according to a federated learning framework, interactively fusing the point cloud features and the image features in a feature layer, and constructing a panoramic information framework model based on credibility weights. And performing fault prediction and diagnosis based on the panoramic information framework model, and outputting a fault type and a corresponding probability. The problem of low fault diagnosis accuracy of a fault diagnosis method based on multi-modal information fusion in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of fault diagnosis technology, and in particular to a fault diagnosis method, system, device and medium based on multimodal information fusion. Background Technology

[0002] Fault diagnosis is a key technology for ensuring the safe operation of intelligent devices. Its core lies in identifying potential or existing faults in the device through data analysis. Traditional methods usually rely on single-modal data (such as vibration signals or image data), but since device faults often exhibit multi-dimensional characteristics (such as structural deformation, abnormal vibration, etc.), the limitations of single-modal data lead to low diagnostic accuracy and poor robustness.

[0003] In recent years, multimodal information fusion technology has received widespread attention in the field of fault diagnosis. For example, weighted evidence theory can handle data uncertainty, but its support for the aggregation of multi-source heterogeneous data (such as point clouds and RGB images) is insufficient, affecting diagnostic accuracy. Although federated learning can achieve data fusion under privacy protection, its high computational complexity makes it difficult to meet real-time requirements. In addition, existing technologies mostly focus on a single stage (such as data quality assessment or model building), resulting in low information utilization and affecting diagnostic accuracy.

[0004] Currently, fault diagnosis using multimodal information fusion in related technologies suffers from low accuracy, and no effective solution has yet been proposed. Summary of the Invention

[0005] This application provides a fault diagnosis method, system, device, and medium based on multimodal information fusion, to at least address the problem of low accuracy in fault diagnosis based on multimodal information fusion in related technologies.

[0006] In a first aspect, embodiments of this application provide a fault diagnosis method based on multimodal information fusion, the method comprising:

[0007] Feature extraction is performed on pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features;

[0008] Using weighted evidence theory, a basic probability assignment function is calculated for the point cloud features, the image features, and the vibration features to generate the credibility weights of the corresponding features.

[0009] According to the federated learning framework, the point cloud features and the image features are interactively fused at the feature layer, and a panoramic information architecture model is constructed based on the confidence weights.

[0010] Based on the panoramic information architecture model, fault prediction and diagnosis are performed, and the fault type and corresponding probability are output.

[0011] In one embodiment, the multi-source heterogeneous data includes point cloud data, RGB image data, and sensor vibration data. The step of extracting features from the pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features includes:

[0012] The point cloud data is grouped according to a preset sampling threshold, and point cloud features are extracted using a PointNet network.

[0013] Image features are extracted from the RGB image data using the YOLOv3 algorithm;

[0014] Perform a Fourier transform operation on the vibration data from the sensor to extract frequency domain features as vibration features.

[0015] In one embodiment, the step of calculating a basic probability assignment function for the point cloud features, the image features, and the vibration features using weighted evidence theory to generate corresponding feature confidence weights includes:

[0016] Based on the historical fault dataset, initial probability values ​​are assigned to the point cloud features, the image features, and the vibration features, respectively.

[0017] The initial probability values ​​of the point cloud features and vibration features are fused by a preset combination rule.

[0018] The probability allocation weights of the image features are adjusted based on the conflict factor K value;

[0019] Based on the initial probability value and the probability allocation weight, the credibility weight of each modality feature is generated.

[0020] In one embodiment, the step of interactively fusing the point cloud features and the image features at the feature layer according to the federated learning framework, and constructing a panoramic information architecture model based on the confidence weights, includes:

[0021] Based on the aforementioned confidence weights, interaction weights between point cloud features and image features are assigned through an attention mechanism;

[0022] Based on the interaction weights, a cross-modal association model is adopted to generate a point cloud and image feature association matrix on the local client of federated learning.

[0023] By using the parameter aggregation mechanism of federated learning, the association matrices of multiple clients are merged on the server side to construct a panoramic information architecture model.

[0024] In one embodiment, the step of fusing the association matrices of multiple clients on the server side through the parameter aggregation mechanism of federated learning to construct a panoramic information architecture model includes:

[0025] Based on the data quality scores of each client, aggregation weights are assigned to the correlation matrix, wherein the data quality scores are obtained through the product of the signal-to-noise ratio and the resolution coefficient.

[0026] The association matrix after assigning aggregation weights is mapped to a three-dimensional topological structure of a unified semantic space using the OWL semantic model.

[0027] Based on the credibility weight of vibration characteristics, a spectral feature layer is embedded in the three-dimensional topology to construct a panoramic information framework model.

[0028] In one embodiment, the panoramic information framework model is based on multi-source heterogeneous data aggregation technology, integrating point cloud 3D structure and image color information to obtain a unified information framework.

[0029] In one embodiment, the step of performing fault prediction and diagnosis based on the panoramic information architecture model, and outputting the fault type and its corresponding probability, includes:

[0030] Based on the multidimensional fusion features of the panoramic information architecture model, temporal fault mode features are extracted through a pre-trained recurrent neural network model.

[0031] Based on the probability distribution of historical fault datasets, calculate the matching degree between the current fused features and each fault type, and generate initial probabilities;

[0032] By using a Bayesian inference model, and combining the initial probability with the confidence level of real-time sensor data, the fault probability output is corrected.

[0033] When the output failure probability exceeds the preset probability threshold, the failure type is determined.

[0034] Secondly, embodiments of this application provide a fault diagnosis system based on multimodal information fusion. The system includes: an extraction module, a credibility weighting module, a panoramic information architecture model module, and a fault diagnosis module, wherein:

[0035] The extraction module is used to extract features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features;

[0036] The credibility weight module is used to calculate the basic probability allocation function for the point cloud features, the image features and the vibration features using weighted evidence theory, and generate the credibility weight of the corresponding features.

[0037] The panoramic information architecture model module is used to interactively fuse the point cloud features and the image features at the feature layer according to the federated learning framework, and to construct a panoramic information architecture model based on the credibility weights.

[0038] The fault diagnosis module is used to predict and diagnose faults based on the panoramic information architecture model, and output the fault type and its corresponding probability.

[0039] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a fault diagnosis method based on multimodal information fusion as described in the first aspect above.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a fault diagnosis method based on multimodal information fusion as described in the first aspect above.

[0041] The fault diagnosis method, system, device, and medium based on multimodal information fusion provided in this application embodiment have at least the following technical effects.

[0042] Feature extraction is performed on pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features, solving the problem of incomplete information in single-modal data and providing a multi-dimensional feature foundation for subsequent fusion. Using weighted evidence theory, basic probability assignment functions are calculated for point cloud features, image features, and vibration features to generate corresponding feature confidence weights, quantifying the uncertainty of different modal data and effectively reducing the interference of noise and abnormal data. Based on the federated learning framework, point cloud features and image features are interactively fused at the feature layer, protecting data privacy while achieving cross-modal deep correlation, overcoming the difficulty of heterogeneous data fusion in traditional methods. Based on the confidence weights, a panoramic information architecture model is constructed, integrating three-dimensional structure, two-dimensional texture, and temporal vibration information to form a unified high-precision diagnostic framework. Fault prediction and diagnosis are performed based on the panoramic information architecture model, outputting the fault type and its corresponding probability. The synergistic effect of the above features improves the accuracy of fault diagnosis while ensuring real-time performance and reliability. This addresses the problem of low accuracy in fault diagnosis caused by multi-modal information fusion in related technologies.

[0043] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1This is a flowchart illustrating a fault diagnosis method based on multimodal information fusion according to an exemplary embodiment;

[0046] Figure 2 This is a system structure block diagram of a fault diagnosis system based on a multimodal information fusion according to an exemplary embodiment;

[0047] Figure 3 This is a structural block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0049] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0050] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] Firstly, embodiments of this application provide a fault diagnosis method based on multimodal information fusion. Figure 1 This is a flowchart of a fault diagnosis method based on multimodal information fusion, such as... Figure 1 As shown, the method includes:

[0053] Step S101: Extract features from the pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features.

[0054] Step S102: Using weighted evidence theory, calculate the basic probability assignment function for point cloud features, image features, and vibration features, and generate the credibility weights of the corresponding features.

[0055] Step S103: Based on the federated learning framework, point cloud features and image features are interactively fused at the feature layer, and a panoramic information architecture model is constructed based on the credibility weight.

[0056] Step S104: Perform fault prediction and diagnosis based on the panoramic information architecture model, and output the fault type and its corresponding probability.

[0057] In summary, this application provides a fault diagnosis method, system, device, and medium based on multimodal information fusion. It extracts features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features, solving the problem of incomplete information in single-modal data and providing a multi-dimensional feature foundation for subsequent fusion. Through weighted evidence theory, basic probability allocation functions are calculated for point cloud features, image features, and vibration features, generating corresponding feature confidence weights. This quantifies the uncertainty of different modal data and effectively reduces interference from noise and abnormal data. Based on the federated learning framework, point cloud features and image features are interactively fused at the feature layer, protecting data privacy while achieving deep cross-modal correlation. This overcomes the difficulty of heterogeneous data fusion in traditional methods. Based on the confidence weights, a panoramic information architecture model is constructed, integrating three-dimensional structure, two-dimensional texture, and temporal vibration information to form a unified high-precision diagnostic framework. Fault prediction and diagnosis are performed based on the panoramic information architecture model, outputting the fault type and its corresponding probability. The synergistic effect of the above features improves the accuracy of fault diagnosis while ensuring real-time performance and reliability. This addresses the issue of low accuracy in fault diagnosis using multimodal information fusion in related technologies.

[0058] In one embodiment, step S101 involves extracting features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features. Specifically, this includes:

[0059] The point cloud data is grouped according to a preset sampling threshold, and point cloud features are extracted through the PointNet network.

[0060] Image features are extracted from RGB image data using the YOLOv3 algorithm;

[0061] Perform a Fourier transform operation on the sensor vibration data to extract frequency domain features as vibration features.

[0062] Optionally, the raw point cloud data (1024×1024 resolution) acquired by LiDAR is first spatially grouped. A sampling threshold of 0.3-0.6 meters is set to divide adjacent point clouds into local clusters. The 3D geometric features (such as curvature and normal vectors) of each cluster are extracted using the PointNet network and aggregated into global point cloud features. Secondly, for the RGB image data (1920×1080 resolution), the YOLOv3 algorithm is used to extract multi-scale texture features (such as edges and color histograms) in the convolutional layer, and local and global information are integrated through spatial pyramid pooling. Finally, for the sensor vibration data (1000Hz sampling rate), the time-domain signal is converted into a frequency-domain energy spectrum using Fast Fourier Transform (FFT), and the dominant frequency amplitude and harmonic components are extracted as vibration features. All feature extraction processes are executed in parallel under GPU acceleration to ensure real-time performance.

[0063] Step S101 reduces computational complexity through spatial grouping while preserving key 3D structural information, avoiding noise interference, and improving the geometric accuracy of subsequent fusion. YOLOv3 multi-scale extraction balances local details and global context, making image features robust to occlusion and illumination changes. Environmental noise is removed, focusing on the inherent frequency characteristics of the equipment, thus enhancing the physical correlation between vibration signals and mechanical faults (such as bearing wear).

[0064] In one embodiment, before step S101, which involves extracting features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features, the method further includes:

[0065] Real-time acquisition of point cloud data (1024x1024 resolution), RGB images (1920x1080 resolution), and vibration signals (1000Hz sampling rate). An RGB camera and vibration sensor are connected to a computing unit (such as a GPU server). The software is implemented in Python, using the PyTorch framework to build a neural network.

[0066] Specifically, point cloud data acquisition utilizes a LiDAR sensor to acquire 3D point clouds of the device at a resolution of 1024×1024. Motion distortion is eliminated through spatial calibration, and the data is transmitted to a GPU server. RGB image capture is performed synchronously using an industrial-grade RGB camera (1920×1080 resolution, 30fps) to capture images of the device surface. OpenCV is used for illumination equalization and geometric correction. Vibration signal sampling is performed synchronously using a triaxial vibration sensor (1000Hz sampling rate, ±50g range). Digital signals are generated through anti-aliasing filtering and ADC conversion, and uploaded in real-time via TCP / IP protocol. This provides the foundation for subsequent feature fusion.

[0067] In one embodiment, step S102 involves calculating a basic probability assignment function for point cloud features, image features, and vibration features using weighted evidence theory to generate the corresponding feature confidence weights. Specifically, this includes:

[0068] Based on the historical fault dataset, initial probability values ​​are assigned to point cloud features, image features, and vibration features respectively;

[0069] The initial probability values ​​of point cloud features and vibration features are fused by pre-defined combination rules.

[0070] The probability allocation weights of image features are adjusted based on the conflict factor K value;

[0071] Based on the initial probability values ​​and probability allocation weights, the credibility weights of each modality feature are generated.

[0072] Optionally, initial probability values ​​are first assigned to each modal feature based on a historical fault dataset (containing multiple sets of labeled samples). For point cloud features, probabilities are assigned based on their 3D structural integrity score (e.g., 0.7 for a complete structure); for vibration features, weights are assigned based on the matching degree of spectral energy distribution (e.g., 0.8 for a matching fundamental frequency amplitude); and for image features, probabilities are initialized based on texture anomaly detection results (e.g., 0.9 for a detected crack). Subsequently, the probability values ​​of point cloud and vibration features are fused using the Dempster combination rule to calculate their joint confidence, and a conflict factor K (K = 1 - Σ(evidence intersection)) is introduced to quantify the degree of contradiction between modalities. When K > 0.5, the image feature weights are dynamically attenuated (e.g., image weight × 0.8 when K = 0.6). The final output is the confidence weight.

[0073] Step S102 improves the accuracy of initial weight allocation by establishing a probability mapping relationship through a large number of fault samples, and uses weighted evidence theory to calculate the basic probability allocation function to integrate uncertainty.

[0074] In one embodiment, step S103 involves fusing point cloud features and image features at the feature layer according to the federated learning framework, and constructing a panoramic information architecture model based on confidence weights. Specifically, this includes:

[0075] Based on the confidence weight, the interaction weights between point cloud features and image features are assigned through an attention mechanism;

[0076] A cross-modal association model is adopted, and a point cloud and image feature association matrix is ​​generated on the local client of federated learning according to the interaction weights.

[0077] By employing a parameter aggregation mechanism based on federated learning, association matrices from multiple clients are fused on the server side to construct a panoramic information architecture model. Specifically, aggregation weights are assigned to the association matrices based on the data quality scores of each client, where the data quality scores are obtained through the product of the signal-to-noise ratio and the resolution coefficient. The association matrices, after weighting, are mapped into a three-dimensional topological structure within a unified semantic space using the OWL semantic model. Based on the credibility weights of vibration features, a spectral feature layer is embedded in the three-dimensional topological structure to construct the panoramic information architecture model.

[0078] Optionally, based on the confidence weights generated in step S102 (e.g., point cloud weight 0.7, image weight 0.8), the interaction weights (e.g., point cloud → image weight 0.6, image → point cloud weight 0.4) are dynamically calculated through a multi-head attention mechanism to achieve adaptive association of the feature layer.

[0079] Subsequently, on the local clients of the federated learning (such as edge devices in industrial fields), a cross-modal association model (based on the Transformer architecture) is used to map the 3D coordinate features of the point cloud and the 2D texture features of the image into a sparse association matrix (size 1024×1024). After receiving the association matrices uploaded by each client, the server assigns aggregation weights according to the data quality score (formula: signal-to-noise ratio (dB) / 10×resolution coefficient (0-1)) (e.g., client A score 0.9, weight 0.4, client B score 0.7, weight 0.3), and fuses the matrices using the FedAvg algorithm. Finally, the OWL semantic model is used to parse the fused matrix into a 3D topological structure (e.g., device skeleton point cloud plus surface texture map), and a spectral energy layer (FFT amplitude heatmap) is embedded based on vibration feature weights (e.g., 0.75), forming a panoramic model containing geometric, visual, and vibration information.

[0080] Step S103 uses federated learning to share model parameters across multiple devices without sharing data. Deep fusion uses an attention mechanism to interact point cloud and image features in the feature Dempster layer to construct a panoramic model with OWL semantics, achieving multimodal deep fusion under privacy protection and improving the information integrity of the panoramic model.

[0081] In one embodiment, the panoramic information architecture model is based on multi-source heterogeneous data aggregation technology, integrating point cloud 3D structure and image color information to obtain a unified information framework. Specifically, it includes:

[0082] Optionally, using multi-source heterogeneous data aggregation technology, the three-dimensional spatial coordinates of LiDAR point cloud data (1024×1024 resolution) are first spatiotemporally aligned with the pixel-level color information of RGB images (1920×1080 resolution), and a point-to-pixel correspondence is established using a feature descriptor-based matching algorithm. Subsequently, an adaptive weighted fusion algorithm is used to integrate the geometric accuracy of the point cloud with the texture details of the image into a unified semantic three-dimensional mesh model.

[0083] In one embodiment, step S104 involves performing fault prediction and diagnosis based on a panoramic information architecture model, and outputting the fault type and its corresponding probability. Specifically, this includes:

[0084] Based on the multidimensional fusion features of the panoramic information architecture model, temporal fault mode features are extracted through a pre-trained recurrent neural network model;

[0085] Based on the probability distribution of historical fault datasets, calculate the matching degree between the current fused features and each fault type, and generate initial probabilities;

[0086] By using a Bayesian inference model, combining the initial probability with the confidence level of real-time sensor data, the fault probability output is corrected.

[0087] When the output failure probability exceeds the preset probability threshold, the failure type is determined.

[0088] Optionally, the multi-dimensional fusion features (including 3D structure, texture, and spectral information) output from the panoramic information architecture model are first input into a pre-trained RNN model. An LSTM unit is then used to capture the temporal evolution of the equipment state (e.g., the increasing pattern of vibration amplitude over time). Subsequently, based on a probability distribution table constructed from historical fault datasets, the cosine similarity between the current feature and various fault types is calculated to generate initial probabilities (e.g., fault A: 0.75, fault B: 0.20). Next, a Bayesian inference model (prior probability = initial probability, likelihood function = real-time sensor data confidence level) dynamically corrects the probabilities. For example, when the confidence level of newly added vibration sensor data is 0.9, the probability of fault A is updated from 0.75 to 0.82. Finally, the system monitors the probability output. When the probability of any fault type exceeds a 95% threshold (configurable), an alarm is triggered and the specific fault type is marked (e.g., "bearing wear - probability 96%)". In this process, a data-driven RNN model is used to predict faults, and the fusion features are used to output probabilities. Training was performed using data similar to the KITTI dataset, with 1000 iterations and a learning rate of 0.001.

[0089] For example:

[0090] First, it is applied to generator fault diagnosis. Image and vibration data are acquired. After feature extraction, the fusion module calculates weighted evidence, with a probability allocation of fault A: 0.8 and uncertainty: 0.2. The diagnostic module outputs "Abnormal generator vibration," with an accuracy of 96%.

[0091] Secondly, it is applied to industrial equipment. It collects point clouds and images of the equipment, processes them in groups, and then fuses them to construct a panoramic model. Federated learning protects data from multiple devices, diagnoses "structural damage," and achieves real-time performance of 225ms / frame.

[0092] Step S104 improves the diagnostic robustness in complex scenarios and achieves breakthroughs in multimodal diagnostic accuracy by combining spatiotemporal feature collaboration (RNN temporal modeling + panoramic model spatial fusion) and probability calibration (historical data prior distribution + real-time confidence posterior correction).

[0093] In summary, this application provides a fault diagnosis method, system, device, and medium based on multimodal information fusion. It extracts features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features, solving the problem of incomplete information in single-modal data and providing a multi-dimensional feature foundation for subsequent fusion. Through weighted evidence theory, basic probability allocation functions are calculated for point cloud features, image features, and vibration features, generating corresponding feature confidence weights. This quantifies the uncertainty of different modal data and effectively reduces interference from noise and abnormal data. Based on the federated learning framework, point cloud features and image features are interactively fused at the feature layer, protecting data privacy while achieving deep cross-modal correlation. This overcomes the difficulty of heterogeneous data fusion in traditional methods. Based on the confidence weights, a panoramic information architecture model is constructed, integrating three-dimensional structure, two-dimensional texture, and temporal vibration information to form a unified high-precision diagnostic framework. Fault prediction and diagnosis are performed based on the panoramic information architecture model, outputting the fault type and its corresponding probability. The synergistic effect of the above features improves the accuracy of fault diagnosis while ensuring real-time performance and reliability. This addresses the issue of low accuracy in fault diagnosis using multimodal information fusion in related technologies.

[0094] Secondly, embodiments of this application provide a fault diagnosis system based on multimodal information fusion. Figure 2 This is a system architecture block diagram illustrating a multimodal information fusion fault diagnosis system according to an exemplary embodiment. For example... Figure 2 As shown, the system includes: an extraction module 210, a credibility weight module 220, a panoramic information architecture model module 230, and a fault diagnosis module 240, wherein:

[0095] Extraction module 210 is used to extract features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features and vibration features;

[0096] The credibility weight module 220 is used to calculate the basic probability assignment function for point cloud features, image features and vibration features through weighted evidence theory, and generate the credibility weight of the corresponding features.

[0097] The panoramic information architecture model module 230 is used to interactively fuse point cloud features and image features at the feature layer according to the federated learning framework, and to construct a panoramic information architecture model based on the confidence weight.

[0098] The fault diagnosis module 240 is used to predict and diagnose faults based on the panoramic information architecture model, and output the fault type and the corresponding probability.

[0099] In summary, the multimodal information fusion-based fault diagnosis system provided in this application improves the accuracy of fault diagnosis through the synergistic effect of the extraction module 210, the confidence weight module 220, the panoramic information architecture model module 230, and the fault diagnosis module 240, while ensuring real-time performance and reliability. This addresses the problem of low accuracy in multimodal information fusion-based fault diagnosis in related technologies.

[0100] Specifically, feature extraction is performed on pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features, solving the problem of incomplete information in single-modal data and providing a multi-dimensional feature foundation for subsequent fusion. Using weighted evidence theory, basic probability assignment functions are calculated for point cloud features, image features, and vibration features to generate corresponding feature confidence weights, quantifying the uncertainty of different modal data and effectively reducing the interference of noise and abnormal data. Based on the federated learning framework, point cloud features and image features are interactively fused at the feature layer, protecting data privacy while achieving cross-modal deep correlation, overcoming the difficulty of heterogeneous data fusion in traditional methods. Based on the confidence weights, a panoramic information architecture model is constructed, integrating three-dimensional structure, two-dimensional texture, and temporal vibration information to form a unified high-precision diagnostic framework. Fault prediction and diagnosis are performed based on the panoramic information architecture model, outputting the fault type and its corresponding probability. The synergistic effect of the above features improves the accuracy of fault diagnosis while ensuring real-time performance and reliability. This addresses the problem of low accuracy in fault diagnosis caused by multi-modal information fusion in related technologies.

[0101] It should be noted that the multimodal information fusion fault diagnosis system provided in this embodiment is used to implement the above-described embodiments, and details already described will not be repeated. As used above, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0102] Thirdly, embodiments of this application provide an electronic device, Figure 3 This is a block diagram illustrating an electronic device according to an exemplary embodiment. (e.g.) Figure 3 As shown, the electronic device may include a processor 31 and a memory 32 storing computer program instructions.

[0103] Specifically, the processor 31 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0104] The memory 32 may include a mass storage device for data or instructions. For example, and not limitingly, the memory 32 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 32 may include removable or non-removable (or fixed) media. Where appropriate, the memory 32 may be internal or external to a data processing device. In a particular embodiment, the memory 32 is non-volatile memory. In a particular embodiment, the memory 32 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only ROM (PROM), an erasable programmable read-only ROM (EPROM), an electrically erasable programmable read-only ROM (EEPROM), an electrically alterable read-only ROM (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0105] The memory 32 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 31.

[0106] The processor 31 reads and executes computer program instructions stored in the memory 32 to implement any of the multimodal information fusion fault diagnosis methods in the above embodiments.

[0107] In one embodiment, a fault diagnosis device based on multimodal information fusion may further include a communication interface 33 and a bus 30. Wherein, as... Figure 3 As shown, the processor 31, memory 32, and communication interface 33 are connected through bus 30 and complete communication with each other.

[0108] The communication interface 33 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication port 33 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0109] Bus 30 includes hardware, software, or both, that couples together components of a fault diagnosis device that fuses multimodal information. Bus 30 includes, but is not limited to, at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. For example, and not as a limitation, bus 30 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 30 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0110] Fourthly, embodiments of this application provide a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a fault diagnosis method based on multimodal information fusion provided in the first aspect.

[0111] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0112] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to perform steps of implementing a fault diagnosis method for multimodal information fusion provided in the first aspect.

[0113] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A fault diagnosis method based on multimodal information fusion, characterized in that, The method includes: Feature extraction is performed on pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features; Using weighted evidence theory, a basic probability assignment function is calculated for the point cloud features, the image features, and the vibration features to generate the credibility weights of the corresponding features. According to the federated learning framework, the point cloud features and the image features are interactively fused at the feature layer, and a panoramic information architecture model is constructed based on the confidence weights. Based on the panoramic information architecture model, fault prediction and diagnosis are performed, and the fault type and corresponding probability are output.

2. The fault diagnosis method based on multimodal information fusion according to claim 1, characterized in that, The process, based on the federated learning framework, involves interactively fusing the point cloud features and the image features at the feature layer, and constructing a panoramic information architecture model based on the confidence weights, including: Based on the aforementioned confidence weights, interaction weights between point cloud features and image features are assigned through an attention mechanism; Based on the interaction weights, a cross-modal association model is adopted to generate a point cloud and image feature association matrix on the local client of federated learning. By using the parameter aggregation mechanism of federated learning, the association matrices of multiple clients are merged on the server side to construct a panoramic information architecture model.

3. The fault diagnosis method based on multimodal information fusion according to claim 2, characterized in that, The parameter aggregation mechanism of federated learning, which merges the association matrices of multiple clients on the server side to construct a panoramic information architecture model, includes: Based on the data quality scores of each client, aggregation weights are assigned to the correlation matrix. The data quality scores are obtained through the product of the signal-to-noise ratio and the resolution coefficient. The aggregation weights are used to quantify the contribution of different data sources during the multi-source data fusion process. The association matrix after assigning aggregation weights is mapped to a three-dimensional topological structure of a unified semantic space using the OWL semantic model. Based on the credibility weight of vibration characteristics, a spectral feature layer is embedded in the three-dimensional topology to construct a panoramic information framework model.

4. The fault diagnosis method based on multimodal information fusion according to claim 1, characterized in that, The step involves calculating a basic probability assignment function for the point cloud features, image features, and vibration features using weighted evidence theory, and generating corresponding feature confidence weights, including: Based on the historical fault dataset, initial probability values ​​are assigned to the point cloud features, the image features, and the vibration features, respectively. The initial probability values ​​of the point cloud features and vibration features are fused by a preset combination rule. The probability allocation weights of the image features are adjusted based on the conflict factor K value; Based on the initial probability value and the probability allocation weight, the credibility weight of each modality feature is generated.

5. The fault diagnosis method based on multimodal information fusion according to claim 1, characterized in that, The fault prediction and diagnosis based on the panoramic information architecture model, outputting the fault type and its corresponding probability, includes: Based on the multidimensional fusion features of the panoramic information architecture model, temporal fault mode features are extracted through a pre-trained recurrent neural network model. Based on the probability distribution of historical fault datasets, calculate the matching degree between the current fused features and each fault type, and generate initial probabilities; By using a Bayesian inference model, and combining the initial probability with the confidence level of real-time sensor data, the fault probability output is corrected. When the output failure probability exceeds the preset probability threshold, the failure type is determined.

6. The fault diagnosis method based on multimodal information fusion according to claim 1, characterized in that, The multi-source heterogeneous data includes point cloud data, RGB image data, and sensor vibration data. The step of extracting features from the pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features includes: The point cloud data is grouped according to a preset sampling threshold, and point cloud features are extracted using a PointNet network. Image features are extracted from the RGB image data using the YOLOv3 algorithm; Perform a Fourier transform operation on the vibration data from the sensor to extract frequency domain features as vibration features.

7. The fault diagnosis method based on multimodal information fusion according to claim 1, characterized in that, The panoramic information architecture model is used to deeply integrate the three-dimensional spatial structure information of point clouds with the two-dimensional color and texture features of images through multi-source heterogeneous data aggregation technology, thereby constructing a unified multi-dimensional information framework.

8. A fault diagnosis system based on multimodal information fusion, characterized in that, The system includes: an extraction module, a credibility weighting module, a panoramic information architecture model module, and a fault diagnosis module, wherein: The extraction module is used to extract features from pre-collected multi-source heterogeneous data to obtain point cloud features, image features, and vibration features; The credibility weight module is used to calculate the basic probability allocation function for the point cloud features, the image features and the vibration features using weighted evidence theory, and generate the credibility weight of the corresponding features. The panoramic information architecture model module is used to interactively fuse the point cloud features and the image features at the feature layer according to the federated learning framework, and to construct a panoramic information architecture model based on the credibility weights. The fault diagnosis module is used to predict and diagnose faults based on the panoramic information architecture model, and output the fault type and its corresponding probability.

9. An electronic device, characterized in that, The system includes a memory and a processor, a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a fault diagnosis method based on multimodal information fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a fault diagnosis method based on multimodal information fusion as described in any one of claims 1 to 7.

Images