Fault prediction method and device for multi-modal data fusion

Abstract

The invention provides a fault prediction method and device based on multi-modal data fusion. The method comprises the steps of collecting multi-modal data of a monitoring area; processing the multi-modal data to obtain feature information of each modal; based on the feature information of each mode, performing cross-modal reasoning analysis by using a pre-trained large language model to obtain a first hidden danger prediction result; inputting the feature information of each mode into a trained deep learning model to carry out space hidden danger mode analysis to obtain a second hidden danger prediction result; and carrying out bidirectional integration on the first hidden danger prediction result and the second hidden danger prediction result, and generating fault early warning information according to an integration result. According to the method, high-precision and real-time early warning and intelligent traceability of the safety risk of the chemical industrial park are realized through bidirectional verification of multi-modal data dynamic fusion, large-model intelligent reasoning and deep learning prediction.

Description

Technical Field

[0001] This application relates to the fields of industrial internet and safe production technology, and more specifically, to a fault prediction method and device based on multimodal data fusion. Background Technology

[0002] Fault prediction technology, by analyzing equipment operating data, identifies potential fault signs in advance, which is crucial for improving the reliability of industrial systems and reducing operation and maintenance costs. Existing technical solutions mainly include methods based on statistics, machine learning, deep learning, and time series analysis. For example, some solutions collect multi-source sensor data, perform preprocessing and feature extraction, and then use dynamic models for trend prediction and potential hazard analysis; others use feature filtering and lightweight LSTM networks to predict equipment lifespan; still others utilize recurrent neural network models to extract and identify time-series features of cybersecurity events.

[0003] However, existing technologies have significant limitations. First, in terms of multimodal data fusion, current methods fail to fully consider the dynamic weight changes between different modalities (such as physical sensing, vision, audio, and text), leading to unstable data fusion results in different application scenarios and affecting the overall system performance. Second, in terms of anomaly identification and feature selection, existing technologies mostly rely on static threshold judgments or static weight calculation methods, which are difficult to adapt to nonlinear anomaly patterns or dynamic changes in equipment operating status under complex environments, limiting the real-time performance and accuracy of prediction models. Third, in terms of hazard identification and analysis, existing solutions often rely on only a single type of model (such as time series models) for judgment, lacking effective capture of complex semantic relationships in multimodal data, and lacking dynamic modeling of the correlation between different modalities during the multidimensional analysis stage, resulting in insufficient accuracy of early warning and tracing capabilities when facing complex cross-modal faults or attacks. Therefore, existing technologies are insufficient to meet the needs of high-precision, real-time early warning and intelligent tracing of safety risks in high-risk scenarios such as chemical industrial parks. Summary of the Invention

[0004] The purpose of this application is to provide a fault prediction method and apparatus based on multimodal data fusion, which achieves high-precision, real-time early warning and intelligent source tracing of safety risks in chemical industrial parks through dynamic fusion of multimodal data, bidirectional verification of large-scale intelligent reasoning and deep learning prediction.

[0005] Firstly, a fault prediction method based on multimodal data fusion is provided, which may include: Collect multimodal data of the monitoring area; The multimodal data is processed to obtain the feature information of each modality; Based on the feature information of each modality, cross-modal reasoning analysis is performed using a pre-trained large language model to obtain the first hidden danger prediction result; The feature information of each modality is input into the trained deep learning model for spatial hazard pattern analysis to obtain the second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on the historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The first and second hazard prediction results are bidirectionally integrated, and fault warning information is generated based on the integration results.

[0006] In one possible implementation, the multimodal data includes at least structured physical sensing data, as well as unstructured visual data, audio data, and text data; The multimodal data is processed to obtain feature information for each modality, including: A lightweight convolutional neural network model is used to extract key visual feature vectors from the visual data. The audio feature vector in the audio data is extracted using the Mel frequency cepstral coefficient method. A lightweight large language model is used to perform real-time semantic analysis on the text data and extract keyword embedding vectors. After filtering outliers and standardizing units, the physical sensing data is encapsulated into a data frame with a predetermined format.

[0007] In one possible implementation, after processing the multimodal data to obtain the feature information of each modality, the method further includes: A credibility fusion weight is assigned to the feature information of each modality. The credibility fusion weight is calculated based on the acquisition frequency, data accuracy, data stability of the transmitted data within a preset time period, and modality importance attention weight of the feature information of the corresponding modality.

[0008] In one possible implementation, the formula for calculating the credibility fusion weight is:

[0009] Where z represents the total number of modal data types; The credibility fusion weights for the i-th modality data are... The sampling frequency for the i-th modal data; The acquisition accuracy of the i-th modal data; For the stability of the i-th modal data; The attention weights are for the i-th modality data; , , , These are the weighting coefficients. .

[0010] In one possible implementation, before inputting the feature information of each modality into a trained deep learning model for spatial hazard pattern analysis and obtaining the second hazard prediction result, the method further includes: The monitoring area is divided into multiple grid units, and based on the historical database, the historical hidden danger data of each grid unit in a continuous T historical time period are statistically analyzed to obtain the historical hidden danger sequence of length T corresponding to each grid unit. Using historical hazard sequences of each grid cell as training samples, unsupervised learning is performed on the training samples through a clustering algorithm to classify all grid cells into multiple hazard level categories, thereby obtaining a trained deep learning model.

[0011] In one possible implementation, unsupervised learning is performed on the training samples using a clustering algorithm to classify all grid cells into multiple hazard level categories, including: Calculate the center vector of the historical hazard sequence of all grid cells and use it as the first cluster center; The historical hazard sequence of the grid cell with the largest sum of Euclidean distances to all determined cluster centers is selected as the next cluster center; Repeat the above selection steps until the number of cluster centers reaches the preset value q; Based on the principle of minimum Euclidean distance, each grid cell is assigned to the hazard level category represented by the nearest cluster center.

[0012] In one possible implementation, the feature information of each modality is input into a trained deep learning model for spatial hazard pattern analysis to obtain a second hazard prediction result, including: The location identifier of the target mesh cell is extracted from the feature information of each modality; Based on the location identifier, the historical hazard sequence of the target grid cell is retrieved from the pre-generated historical hazard data; The historical hazard sequence of the target grid cell is input into the trained deep learning model to obtain the current hazard level category of the target grid cell output by the deep learning model, which is used as the second hazard prediction result.

[0013] Secondly, a fault prediction device based on multimodal data fusion is provided, the device comprising: The acquisition unit is used to collect multimodal data from the monitoring area; The processing unit is used to process the multimodal data to obtain the feature information of each modality; The analysis unit is used to perform cross-modal reasoning analysis based on the feature information of each modality using a pre-trained large language model to obtain a first hazard prediction result; and to input the feature information of each modality into a trained deep learning model to perform spatial hazard pattern analysis to obtain a second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The integration unit is used to bidirectionally integrate the first hidden danger prediction result and the second hidden danger prediction result, and generate fault warning information based on the integration result.

[0014] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0015] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0016] This application provides a fault prediction method and apparatus based on multimodal data fusion. The method collects multimodal data from a monitoring area; processes the multimodal data to obtain feature information for each modality; based on the feature information of each modality, performs cross-modal reasoning analysis using a pre-trained large language model to obtain a first hazard prediction result; inputs the feature information of each modality into a trained deep learning model for spatial hazard pattern analysis to obtain a second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on historical hazard data from each grid unit within the monitoring area over T consecutive historical time periods; the first hazard prediction result and the second hazard prediction result are bidirectionally integrated, and fault early warning information is generated based on the integration result. This method achieves high-precision, real-time early warning and intelligent source tracing of safety risks in chemical industrial parks through dynamic fusion of multimodal data, bidirectional verification of intelligent reasoning from a large model and deep learning prediction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the structure of a fault prediction system based on multimodal data fusion provided in this application embodiment; Figure 2 A flowchart illustrating a fault prediction method based on multimodal data fusion provided in this application embodiment; Figure 3 A schematic diagram of the structure of a fault prediction device for multimodal data fusion provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The terms "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The terms "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but does not exclude other elements or objects. The terms "connection," "coupled," or "linked," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0020] The core of the multimodal data fusion-based fault prediction method provided in this application lies in achieving accurate early warning of safety risks in monitored areas such as chemical industrial parks through a hierarchical and collaborative computing architecture. The multimodal data fusion-based fault prediction method provided in this application is applied in fault prediction systems, such as... Figure 1 As shown, the multimodal data fusion fault prediction system is divided into a perception layer, an edge computing layer, a cloud processing layer, and an application layer from top to bottom. Each layer realizes data interaction and command transmission through a 5G private network.

[0021] The perception layer is responsible for collecting multimodal data within the monitoring area. Specifically, it collects environmental monitoring data at a preset frequency (e.g., once every 10 seconds) through deployed physical sensing devices such as temperature sensors, pressure sensors, and toxic gas sensors; it collects video stream data (25 frames / second) of key areas through high-definition video surveillance equipment; it collects audio data such as equipment operating noise and leakage abnormalities through audio sensors (e.g., sampling frequency 44.1kHz); and it obtains unstructured operation and maintenance text data such as equipment maintenance records and fault repair orders from the equipment management system.

[0022] The edge computing layer, deployed locally within the monitoring area, is responsible for real-time or near-real-time preprocessing and feature extraction of multimodal data uploaded from the perception layer, as well as connecting with the cloud processing layer (for example, deploying three edge computing nodes around a chemical industrial park, each node connecting to the perception layer devices and the cloud processing layer via a 5G private network). For visual data, a lightweight convolutional neural network model (e.g., MobileNetV3) is used to extract key visual features (such as flame outlines, smoke textures, and people crossing boundaries) from video frames, generating visual feature vectors. For audio data, the Mel-frequency cepstral coefficient method is used to extract audio feature vectors. For text data, a lightweight large language model (e.g., a quantized version of LLaMA-7B) is used for real-time semantic analysis to identify and extract embedding vectors of keywords related to safety hazards. For physical sensor data, data cleaning (e.g., outlier filtering), unit standardization, and encapsulation into structured data frames of a predetermined format are performed. This approach aims to reduce the cloud transmission load and improve system real-time performance.

[0023] The cloud processing layer, serving as the core analysis engine deployed on a cloud-based big data processing platform, receives multimodal feature information processed by the edge computing layer and performs three core tasks: large-scale intelligent hazard analysis, deep learning spatial hazard pattern analysis, and bidirectional integration of the two. First, based on dynamically calculated credibility fusion weights, the feature information from various modalities is fused and input into a domain-fine-tuned large language model for cross-modal inference, yielding a first hazard prediction result representing the probability of the current hazard occurring. In parallel, a deep learning clustering model trained on historical hazard data performs spatial pattern analysis on the current target area, outputting its hazard severity level category as a second hazard prediction result. Finally, the bidirectional integration module performs logical judgment on the two prediction results to generate the final fault warning information.

[0024] The application layer is responsible for the visual display of early warning information and the issuance of dispatch instructions, providing park management personnel with multiple channels for displaying and interacting with early warning information, such as web terminals, mobile apps, and on-site LED displays.

[0025] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0026] Figure 2 This is a flowchart illustrating a fault prediction method based on multimodal data fusion, provided as an embodiment of this application. Figure 2 As shown, the method may include: Step S210: Collect multimodal data of the monitoring area.

[0027] Multiple sensors and data acquisition devices are deployed within the monitoring area (such as a chemical industrial park) to collect multimodal data. Multimodal data can include structured physical sensor data, as well as unstructured visual, audio, and text data. Specifically: The physical sensing data comes from various sensors deployed throughout the park, such as temperature sensors, pressure sensors, and toxic gas sensors (e.g., hydrogen sulfide and chlorine sensors). These sensors collect real-time monitoring data at preset frequencies; for example, temperature, pressure, and gas concentration sensors collect data every 10 seconds or 6 times per minute; the fire protection facility status monitor collects equipment status data once per minute.

[0028] For visual data, high-definition video surveillance equipment collects video streams from key areas of the park (such as tank areas and production workshops) at a rate of 25 frames per second, which are used to identify visual features such as flames, smoke, and personnel crossing boundaries.

[0029] For audio data, audio sensors deployed in key areas collect audio signals such as equipment operating noise and leakage abnormalities at a sampling frequency of 44.1 kHz.

[0030] For text data, including unstructured text data such as equipment operation and maintenance logs, fault repair orders, and maintenance records, these data are updated at a low frequency, such as 0.1 times per minute.

[0031] Data acquisition is achieved through the perception layer, which consists of the aforementioned sensors and logging system, ensuring comprehensive acquisition of multi-source data. The acquired data is transmitted to the processing layer via wired or wireless networks (such as a 5G private network).

[0032] Step S220: Process the multimodal data to obtain the feature information of each mode.

[0033] Lightweight convolutional neural network models (such as MobileNetV3) are used to extract key visual feature vectors from visual data, such as flame outlines and smoke textures, reducing data transmission volume. For example, the feature vector dimension is 256. The Mel frequency cepstral coefficient (MFCC) method is used to extract audio feature vectors from audio data to identify abnormal noise features of devices. The feature vector dimension is, for example, 128 dimensions. Using a lightweight large language model (such as the quantized version of LLaMA-7B), real-time semantic analysis is performed on text data to extract keyword embedding vectors. For example, it can identify keywords that pose security risks such as aging and leakage. The feature vector dimension is, for example, 768 dimensions. The physical sensor data undergoes preprocessing such as outlier filtering and unit standardization, and is then packaged into a data frame with a predetermined format, such as a 64-dimensional feature vector.

[0034] The above processing can be completed at the edge computing layer to reduce the load on the cloud. After processing, feature vectors for each modality are obtained.

[0035] Furthermore, after obtaining the feature information of each modality, a credibility fusion weight is assigned to the feature information of each modality. This credibility fusion weight is calculated based on the acquisition frequency, data accuracy, data stability transmitted within a preset time period, and modality importance attention weight of the corresponding modality's feature information. The formula for calculating this credibility fusion weight can be expressed as:

[0036] Where z represents the total number of modal data categories; j is a summation index variable used to iterate through all z modal categories; The credibility fusion weights for the i-th modality data are... The acquisition frequency of the i-th modal data (unit: times / minute); is the acquisition precision of the i-th type of modal data (dimensionless, value range 0 to 1); The stability of the i-th modal data (e.g., the data transmission success rate in the past 30 days, with a value ranging from 0 to 1); The attention weight for the i-th modality (calculated by the attention mechanism, reflecting the contribution of this modality to security warning, with a value ranging from 0 to 1); , , , These are the weighting coefficients. .For example, , , , Weight calculation ensures the dynamic adaptability of multimodal data fusion.

[0037] The aforementioned feature extraction process can be completed at the edge computing layer. Edge computing nodes are deployed around the park and connected via a 5G private network to achieve data preprocessing and lightweight analysis, reducing the cloud transmission load.

[0038] Step S230: Based on the feature information of each modality, cross-modal reasoning analysis is performed using a pre-trained large language model to obtain the prediction result of the first hidden danger.

[0039] For safety scenarios in chemical industrial parks, supervised fine-tuning is performed on a basic large language model (such as LLaMA-7B). The fine-tuning data includes three categories: equipment fault classification data (e.g., label pairs for chlorine leaks and valve aging), safety regulation masking data (e.g., masking key clauses in the "Chemical Safety Regulations"), and entity recognition data (labeling equipment names, fault locations, etc.). The fine-tuning objective function is a weighted sum of classification loss, masked language model loss, and named entity recognition loss. Specifically, the fine-tuning objective function can be expressed as:

[0040] in, Classification of losses (used for classifying equipment failure type and risk level); For masked language model loss (used for security specification text understanding and operation and maintenance log semantic analysis); Loss in Named Entity Identification (used to extract entities such as equipment name, fault location, and hazard type); , , For loss weights, for example, a value of , Loss weighting coefficient.

[0041] The feature vectors of non-textual modalities (visual, audio, sensor) are aligned with the feature vectors of textual modalities in the feature space. Specifically, alignment and fusion in the feature space are achieved by multiplying the non-textual modal feature vectors by an adaptation matrix, mapping them to the same dimension as the textual modal feature vectors, and then concatenating them. That is, the adaptation matrix is ​​used to map the non-textual features to the textual feature space: visual feature vectors (Dimension 256) via matrix (Size 256×768) Mapping, Audio Feature Vector (Dimension 128) via matrix (Size 128×768) Mapping, Sensing Feature Vector (Dimension 64) via matrix (Size 64×768) Mapping. Then, it is concatenated with the text embedding vector (dimension 768) to form a unified fused feature vector as input to the large language model:

[0042] in, For text modality embedding vectors (operation logs, security specifications, etc.); , , These are feature vectors for visual, audio, and sensory modalities, respectively. , , The weights are multimodal fusion weights, where the weights of the mapping matrix are dynamically adjusted based on the credibility fusion weights from step S220.

[0043] Next, a large-scale model is used to infer the failure probability; specifically, based on the causal reasoning capabilities of LLM, an accident prediction probability model is implemented. In the chlorine leak scenario, E represents the chlorine leak event from the storage tank valve; C represents the set of causes {equipment aging}. Operational violations Environmental impact }. Trained through an industry case library =0.7 (probability of leakage due to aging) =0.15、 =0.1; Prior probability =0.6 (Percentage of aging in historical leaks) =0.2、 =0.2. The formula for tracing the source of hidden dangers is:

[0044] in, These are currently monitored potential hazards (such as gas leaks or abnormal equipment noises). This is a collection of possible causes of potential hazards (equipment aging, operational violations, environmental impacts, etc.). The conditional probability of event E caused by cause C (obtained through training based on an industry case library); Let C be the prior probability of cause C.

[0045] The large language model outputs the probability value of the potential hazard event as the first hazard prediction result. For example, in a chlorine leak scenario, the model can output the leak probability and root cause analysis.

[0046] This step is implemented in the large model prediction module of the cloud processing layer, relying on cloud computing resources to perform complex inference.

[0047] Step S240: Input the feature information of each modality into the trained deep learning model to perform spatial hazard pattern analysis and obtain the second hazard prediction result.

[0048] The trained deep learning model is based on historical hazard data of each grid unit within the monitoring area over T consecutive historical time periods, trained using a clustering algorithm. This includes data preparation, model training, and model prediction. (1) Data Preparation: The monitoring area is divided into multiple grid units. For example, the chemical industrial park is divided into 100 horizontal and 100 vertical divisions, forming 100×100 grid units, totaling 10,000 grid units. Using a preset time period (e.g., 1 hour) as the unit, based on the historical database, the historical hazard data of each grid unit within a continuous T historical time period is statistically analyzed: if a hazard occurs within each time period, it is marked as 1; if no hazard occurs, it is marked as 0, resulting in a historical hazard sequence of length T for each grid unit. For the i-th grid unit, the sequence is denoted as... ,in This represents the hazard marker in the t-th time period.

[0049] (2) Model training: Using the historical hazard sequences of each grid unit as training samples, unsupervised learning is performed on the training samples through clustering algorithms to divide all grid units into multiple (e.g., k) hazard level categories (k takes values ​​of 100,000, ... (where n is the total number of grid cells), to obtain a trained deep learning model; the training process includes: Calculate the center vector of the historical hazard sequence of all grid cells as the first cluster center. ,in, ; The historical hazard sequence of the grid cell with the largest sum of Euclidean distances to all determined cluster centers is selected as the next cluster center; for example, the second cluster center. Distance The furthest sequence, the third cluster center Distance and The furthest sequence is repeated until the number of cluster centers reaches q.

[0050] Based on the principle of minimum Euclidean distance, each grid cell is assigned to the hazard level category represented by the nearest cluster center. The formula for calculating the Euclidean distance from each grid cell to each cluster center is as follows: .

[0051] Next, the average number of hazards in each hazard category is calculated, and the hazard categories with the highest average number of hazards are defined as high-risk hazard categories.

[0052] (3) Model prediction: During prediction, the location identifier of the target grid cell is extracted from the feature information. Based on the identifier, the historical hazard sequence of the grid cell is searched from the historical database and input into the trained deep learning model. The model outputs the hazard level category (such as high risk, medium risk, low risk) to which the cell belongs as the second hazard prediction result.

[0053] This step is implemented in the deep learning fault prediction module of the cloud processing layer, focusing on spatial pattern analysis.

[0054] Step S250: Integrate the prediction results of the first hidden danger and the prediction results of the second hidden danger in both directions, and generate fault warning information based on the integration results.

[0055] This step integrates the first hazard prediction result (hazard probability) output by the large language model and the second hazard prediction result (hazard severity level) output by the deep learning model. The specific integration rule is as follows: when the first hazard prediction result is greater than a preset threshold (e.g., 0.9), and the second hazard prediction result indicates that the target grid cell belongs to the high-risk hazard category, a fault warning is generated. For example, in a chlorine leak scenario, if the large language model calculates a leak probability of 0.95, and the deep learning model identifies the area as a high-risk category, the system triggers a warning.

[0056] Early warning information is displayed through the application layer (such as a web management terminal, mobile app, or LED display), including the location, type, probability, and handling suggestions of potential hazards, supporting resource scheduling and progress tracking. The integration process is completed in the bidirectional integrated fault module of the cloud processing layer, ensuring the accuracy and reliability of the early warnings.

[0057] This application effectively solves the problems of low efficiency in processing multi-source heterogeneous data, insufficient accuracy in identifying hidden dangers, and high false alarm rate in existing technologies by combining multimodal data fusion, large model inference, and deep learning prediction, and significantly improves the accuracy and real-time performance of safety risk early warning in chemical industrial parks.

[0058] Corresponding to the above method, embodiments of this application also provide a fault prediction device based on multimodal data fusion, such as... Figure 3 As shown, the device includes: Acquisition unit 310 is used to acquire multimodal data of the monitoring area; Processing unit 320 is used to process the multimodal data to obtain feature information of each modality; Analysis unit 330 is used to perform cross-modal reasoning analysis based on the feature information of each modality using a pre-trained large language model to obtain a first hazard prediction result; and to input the feature information of each modality into a trained deep learning model to perform spatial hazard pattern analysis to obtain a second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The integration unit 340 is used to bidirectionally integrate the first hidden danger prediction result and the second hidden danger prediction result, and generate fault warning information based on the integration result.

[0059] The functions of each functional unit of the multimodal data fusion fault prediction device provided in the above embodiments of this application can be implemented through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the multimodal data fusion fault prediction device provided in the embodiments of this application will not be repeated here.

[0060] This application also provides an electronic device, such as... Figure 4 As shown, it includes a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440.

[0061] Memory 430 is used to store computer programs; When the processor 410 executes the program stored in the memory 430, it performs the following steps: Collect multimodal data of the monitoring area; The multimodal data is processed to obtain the feature information of each modality; Based on the feature information of each modality, cross-modal reasoning analysis is performed using a pre-trained large language model to obtain the first hidden danger prediction result; The feature information of each modality is input into the trained deep learning model for spatial hazard pattern analysis to obtain the second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on the historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The first and second hazard prediction results are bidirectionally integrated, and fault warning information is generated based on the integration results.

[0062] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0063] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0064] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0065] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 2 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0067] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the fault prediction method of multimodal data fusion as described in any of the above embodiments.

[0068] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the fault prediction method of multimodal data fusion as described in any of the above embodiments.

[0069] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0074] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A fault prediction method based on multimodal data fusion, characterized in that, The method includes: Collect multimodal data of the monitoring area; The multimodal data is processed to obtain the feature information of each modality; Based on the feature information of each modality, cross-modal reasoning analysis is performed using a pre-trained large language model to obtain the first hidden danger prediction result; The feature information of each modality is input into the trained deep learning model for spatial hazard pattern analysis to obtain the second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on the historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The first and second hazard prediction results are bidirectionally integrated, and fault warning information is generated based on the integration results.

2. The method as described in claim 1, characterized in that, The multimodal data includes at least structured physical sensing data, as well as unstructured visual data, audio data, and text data; The multimodal data is processed to obtain feature information for each modality, including: A lightweight convolutional neural network model is used to extract key visual feature vectors from the visual data. The audio feature vector in the audio data is extracted using the Mel frequency cepstral coefficient method. A lightweight large language model is used to perform real-time semantic analysis on the text data and extract keyword embedding vectors. After filtering outliers and standardizing units, the physical sensing data is encapsulated into a data frame with a predetermined format.

3. The method as described in claim 1 or 2, characterized in that, After processing the multimodal data to obtain the feature information of each modality, the method further includes: A credibility fusion weight is assigned to the feature information of each modality. The credibility fusion weight is calculated based on the acquisition frequency, data accuracy, data stability of the transmitted data within a preset time period, and modality importance attention weight of the feature information of the corresponding modality.

4. The method as described in claim 3, characterized in that, The formula for calculating the credibility fusion weight is as follows: Where z represents the total number of modal data types; The credibility fusion weights for the i-th modality data are... The sampling frequency for the i-th modal data; The acquisition accuracy of the i-th modal data; For the stability of the i-th modal data; The attention weights are for the i-th modality data; , , , These are the weighting coefficients. .

5. The method as described in claim 1, characterized in that, Before inputting the feature information of each modality into the trained deep learning model for spatial hazard pattern analysis and obtaining the second hazard prediction result, the method further includes: The monitoring area is divided into multiple grid units, and based on the historical database, the historical hidden danger data of each grid unit in a continuous T historical time period are statistically analyzed to obtain the historical hidden danger sequence of length T corresponding to each grid unit. Using historical hazard sequences of each grid cell as training samples, unsupervised learning is performed on the training samples through a clustering algorithm to classify all grid cells into multiple hazard level categories, thereby obtaining a trained deep learning model.

6. The method as described in claim 5, characterized in that, Unsupervised learning is performed on the training samples using a clustering algorithm to classify all grid cells into multiple hazard level categories, including: Calculate the center vector of the historical hazard sequence of all grid cells and use it as the first cluster center; The historical hazard sequence of the grid cell with the largest sum of Euclidean distances to all determined cluster centers is selected as the next cluster center; Repeat the above selection steps until the number of cluster centers reaches the preset value q; Based on the principle of minimum Euclidean distance, each grid cell is assigned to the hazard level category represented by the nearest cluster center.

7. The method as described in claim 5, characterized in that, The feature information of each modality is input into a trained deep learning model for spatial hazard pattern analysis to obtain the second hazard prediction result, including: The location identifier of the target mesh cell is extracted from the feature information of each modality; Based on the location identifier, the historical hazard sequence of the target grid cell is retrieved from the pre-generated historical hazard data; The historical hazard sequence of the target grid cell is input into the trained deep learning model to obtain the current hazard level category of the target grid cell output by the deep learning model, which is used as the second hazard prediction result.

8. A fault prediction device based on multimodal data fusion, characterized in that, The device includes: The acquisition unit is used to collect multimodal data from the monitoring area; The processing unit is used to process the multimodal data to obtain the feature information of each modality; The analysis unit is used to perform cross-modal reasoning analysis based on the feature information of each modality using a pre-trained large language model to obtain a first hazard prediction result; and to input the feature information of each modality into a trained deep learning model to perform spatial hazard pattern analysis to obtain a second hazard prediction result; the trained deep learning model is trained using a clustering algorithm based on historical hazard data of each grid unit in the monitoring area over a continuous T historical time period; The integration unit is used to bidirectionally integrate the first hidden danger prediction result and the second hidden danger prediction result, and generate fault warning information based on the integration result.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images