A deep learning-based multi-modal olfactory map fault prediction system

Abstract

This invention discloses a multimodal olfactory atlas-based fault prediction system based on deep learning, comprising: a data acquisition module for acquiring and synchronously processing data to generate a multimodal raw olfactory data sequence; a preprocessing module for generating a preprocessed multimodal olfactory data sequence; an atlas construction module for constructing a multimodal olfactory atlas frame sequence; a feature extraction module for generating a multimodal olfactory high-dimensional feature vector sequence based on a deep learning feature extraction network; an event generation module for generating a multimodal olfactory event sequence based on the multimodal olfactory high-dimensional feature vector; an event prediction module for outputting an event trigger probability sequence through an improved neural Hawkes process prediction network; a risk assessment module for generating equipment fault prediction results; and an early warning output module for outputting fault warning information and providing equipment fault prediction results. This invention achieves equipment fault prediction based on multimodal olfactory atlases, possessing advantages such as accurate identification, strong anti-interference capabilities, and reliable early warning.

Description

Technical Field

[0001] This invention relates to the field of fault prediction technology, and in particular to a multimodal olfactory mapping fault prediction system based on deep learning. Background Technology

[0002] Existing equipment fault prediction technologies largely rely on traditional sensing information such as vibration, acoustics, current, and voltage. They fail to adequately utilize odor information related to equipment aging, leakage, corrosion, or abnormal chemical reactions, such as changes in gas composition, release of volatile substances, the coupling effects of temperature and humidity, and differences in spectral characteristics in the operating environment. While electronic olfactory systems can collect gaseous signals, they typically process only single-modal gas response data and lack the ability to fuse and express temperature, humidity, volatile organic compound, and spectral data. Consequently, they struggle to construct a multimodal olfactory map that comprehensively reflects the changing patterns of odors around the equipment.

[0003] On the other hand, existing deep learning methods are mostly limited to feature extraction from single structured data, failing to simultaneously capture local spatial patterns, cross-modal correlations, and temporal variation features. Existing time series prediction methods also struggle to model discrete events caused by multimodal odor feature changes over continuous time, failing to effectively utilize the correlation between event occurrence time and event intensity. This results in insensitivity to fault precursor detection, insufficient fault prediction lead time, and low early warning reliability. Therefore, a fault prediction system is needed that can integrate multimodal olfactory information, possess high-dimensional feature representation capabilities, and predict event trigger probabilities in the continuous time domain to address these issues.

[0004] Therefore, how to provide a multimodal olfactory mapping fault prediction system based on deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a multimodal olfactory map fault prediction system based on deep learning. This invention achieves equipment fault prediction based on multimodal olfactory maps and has the advantages of accurate identification, strong anti-interference and reliable early warning.

[0006] A deep learning-based multimodal olfactory mapping fault prediction system according to an embodiment of the present invention includes: The data acquisition module is used to collect and synchronously process data to generate multimodal raw olfactory data sequences; The preprocessing module is used to preprocess the raw multimodal olfactory data sequence to generate a preprocessed multimodal olfactory data sequence. The map construction module constructs a multimodal olfactory map frame sequence according to a preset time window from the preprocessed multimodal olfactory data sequence. The feature extraction module inputs the multimodal olfactory map frame sequence into the deep learning feature extraction network to generate a multimodal olfactory high-dimensional feature vector sequence; The event generation module generates multimodal olfactory event sequences based on multimodal olfactory high-dimensional feature vectors; The event prediction module predicts the event trigger probability sequence output by the network through an improved neural Hawkes process. The risk assessment module generates equipment failure prediction results based on the event trigger probability sequence; The early warning output module is used to output fault early warning information and provide equipment fault prediction results when preset trigger conditions are met.

[0007] Optionally, modules can be integrated using the following methods: Gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are collected and processed synchronously according to a unified time reference to form a multimodal raw olfactory data sequence. The original multimodal olfactory data sequence is preprocessed to obtain the preprocessed multimodal olfactory data sequence; The preprocessed multimodal olfactory data sequence is divided into preset time windows, and the features within each time window are jointly mapped to construct a multimodal olfactory atlas frame sequence. The multimodal olfactory map frame sequence is input into a deep learning feature extraction network to extract local spatial patterns, cross-modal association patterns, and temporal variation patterns, resulting in a multimodal olfactory high-dimensional feature vector sequence. Based on the change amplitude, gradient change, or preset event rules of the multimodal olfactory high-dimensional feature vector sequence, event types are generated, and multimodal olfactory event sequences are generated. By inputting multimodal olfactory event sequences into an improved neural Hawkes process prediction network, the event trigger intensity is modeled continuously over time and modulated to obtain an event trigger probability sequence within a future time interval. A fault risk scoring sequence is constructed based on the event trigger probability sequence. The presence of fault precursors is determined based on the fault risk scoring sequence to form a equipment fault prediction result. When the scores in the fault risk scoring sequence meet the preset trigger conditions, a fault warning message is output.

[0008] Optionally, the generation of the preprocessed multimodal olfactory data sequence includes: Baseline correction is performed on the gas response data in the multimodal raw olfactory data sequence to generate baseline-corrected gas response data; Drift compensation is performed on the temperature and humidity data in the original multimodal olfactory data sequence to generate drift-compensated temperature and humidity data; Noise suppression is performed on the spectral data in the original multimodal olfactory data sequence to generate noise-suppressed spectral data; The volatile organic compound (VOC) data in the multimodal raw olfactory data sequence is scaled uniformly to generate VOC data after scale uniformity. The baseline-corrected gas response data, drift-compensated temperature and humidity data, noise-suppressed spectral data, and scale-unified volatile organic compound data are recombined in their original chronological order to form a preprocessed multimodal olfactory data sequence.

[0009] Optionally, the generation of the multimodal olfactory map frame sequence includes: The preprocessed multimodal olfactory data sequence is divided into preset time windows. Within each preset time window, the corresponding gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are extracted to form data segments corresponding to the preset time windows. Feature extraction is performed on the gas response data, temperature data, humidity data, volatile organic compound data, and spectral data in the data segment. Within the preset time window, the gas response features, temperature features, humidity features, volatile organic compound features, and spectral features are obtained. The gas response characteristics, temperature characteristics, humidity characteristics, volatile organic compound characteristics and spectral characteristics corresponding to each preset time window are subjected to joint mapping processing to generate the corresponding multimodal olfactory spectrum frame within each preset time window; The multimodal olfactory atlas frames are arranged sequentially according to the time order of the preset time window to form a multimodal olfactory atlas frame sequence.

[0010] Optionally, the generation of the multimodal olfactory high-dimensional feature vector sequence includes: The multimodal olfactory map frame sequence is input frame by frame into a deep learning feature extraction network; In the convolutional structure, convolution operations are performed on the input multimodal olfactory map frames, and numerical processing is performed on local regions of the map frames within a fixed convolutional region to generate convolutional structure processed data. In the graph structure, numerical propagation processing based on connection relationships is performed on the modal features of each modality in the input multimodal olfactory map frame to generate graph structure processed data; In the attention structure, attention weights are calculated for each feature position in the input multimodal olfactory map frame to generate attention structure processing data; The data processed by the convolutional structure, graph structure, and attention structure are numerically merged in a fixed order. The merged result is then subjected to feature compression to form a single-frame high-dimensional feature vector corresponding to each multimodal olfactory map frame. All single-frame high-dimensional feature vectors are arranged in chronological order according to the multimodal olfactory map frame sequence to form a multimodal olfactory high-dimensional feature vector sequence.

[0011] Optionally, the formation of the multimodal olfactory event sequence includes: The multimodal olfactory high-dimensional feature vector sequence is read one by one in chronological order. At each time position, the corresponding multimodal olfactory high-dimensional feature vector is obtained, and the change amplitude is calculated. The gradient of change is further calculated for the multimodal olfactory high-dimensional feature vector at each time location to obtain the gradient change; Based on the magnitude of change, the gradient of change, or the preset event rules, the event judgment is performed on the multimodal olfactory high-dimensional feature vector corresponding to each time position. When the magnitude of change reaches the preset magnitude of change condition, or the gradient of change reaches the preset gradient change condition, or the triggering condition in the preset event rules is met, the event type corresponding to that time position is determined, and that time position is determined as the event occurrence time. Event records are formed by combining the event type, the time of the event, and the multimodal olfactory high-dimensional feature vector corresponding to that time location. All event records are then arranged in chronological order of the event occurrence time to form a multimodal olfactory event sequence.

[0012] Optionally, the generation of the event triggering probability sequence within the future time interval includes: The multimodal olfactory event sequence is input into an improved neural Hawkes process prediction network. At the time of input, the event type, the time of occurrence of each event and the corresponding multimodal olfactory high-dimensional feature vector are read in the order of the events. In the improved neural Hawkes process prediction network, the triggering relationship between events is calculated based on the difference between the occurrence times of adjacent events, forming a continuous-time modeling result of the event triggering intensity. Based on continuous-time modeling of event trigger intensity, the multimodal olfactory high-dimensional feature vector corresponding to each event is input into an improved neural Hawkes process prediction network to modulate the event trigger intensity, so that the event trigger intensity changes with the numerical value of the multimodal olfactory high-dimensional feature vector to obtain the modulated event trigger intensity. Based on the change of the modulated event trigger intensity over time, the event trigger probability is calculated in the future time interval to form an event trigger probability sequence in the future time interval.

[0013] Optionally, the generation of the equipment fault prediction result includes: The event trigger probability sequence is read sequentially in chronological order, and the corresponding event trigger probability value is obtained at each time position. Based on the preset rules for risk calculation, the event trigger probability value at each time point is used as the basis for calculation. The corresponding risk score value is calculated at each time point, and all risk score values ​​are arranged in the time order of the event trigger probability sequence to form a fault risk score sequence. Based on the fault risk scoring sequence, the risk score value at each time point is used to determine the fault precursor. When the risk score value meets the preset fault precursor conditions, it is determined that there is a fault precursor at that time point. The time location identified as the location of a potential fault precursor is taken as the fault precursor occurrence time, and this fault precursor occurrence time is used as the equipment fault prediction result.

[0014] Optionally, the generation of the fault warning information includes: The fault risk score sequence is read one by one in chronological order, the corresponding risk score value is obtained at each time position, and the risk score value is judged according to the preset triggering conditions used for early warning triggering. When the risk score value meets the preset triggering conditions, determine the risk score value corresponding to that time position to trigger the early warning condition, and output the fault early warning information at that time position; The time and location of the output fault warning information are associated with the corresponding equipment fault prediction results, and the equipment fault prediction results are provided to the operation and maintenance process.

[0015] The beneficial effects of this invention are: This invention constructs a multimodal olfactory map frame sequence that reflects the spatiotemporal variation characteristics of odors by uniformly collecting, preprocessing, and fusing multimodal olfactory information such as gas response data, temperature data, humidity data, volatile organic compound data, and spectral data. This achieves a comprehensive characterization of the odor dimension of the equipment operating environment. Compared with traditional electronic olfaction technology that relies solely on a single gas signal, the multimodal olfactory map construction method of this invention can structurally fuse change information from different physical and chemical modes according to a unified time benchmark, enabling a more complete and finer-grained description of the complex odor change patterns around the equipment. Furthermore, this invention overcomes the shortcomings of existing deep learning models in simultaneously identifying local changes, modal couplings, and long-term trends by extracting local spatial patterns, cross-modal correlation patterns, and global temporal correlations through convolutional structures, graph structures, and attention structures. This generates a more powerful multimodal olfactory high-dimensional feature vector sequence, significantly improving the ability to identify weak change features, latent change patterns, and nonlinear coupling relationships.

[0016] In event-level prediction, this invention maps high-dimensional odor features into multimodal olfactory event sequences by varying the magnitude of change, gradient changes, and pre-defined event rules. It then utilizes an improved neural Hawkes process to model the triggering relationships between events over continuous time, and modulates the event triggering intensity using multimodal olfactory high-dimensional feature vectors. This enables the model to capture the deep relationship between event occurrence time, event correlation strength, and dynamic changes in odor features, overcoming the technical bottleneck of traditional prediction methods in handling the multimodal olfactory event-driven characteristics. By constructing a fault risk scoring sequence from the event trigger probability sequence and performing fault precursor judgment, this invention achieves early prediction and intelligent warning of potential equipment faults. Fault prediction no longer relies on a single threshold trigger but is based on continuous-time probability evolution trends, significantly improving the lead time, stability, and accuracy of fault prediction. This invention not only enhances the state recognition capability of industrial equipment in complex environments but also effectively reduces the risk of misjudgment caused by odor interference, environmental fluctuations, and sensor drift, providing more reliable early warning basis and strategy formulation support for industrial equipment operation and maintenance, demonstrating significant application value and engineering significance. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a deep learning-based multimodal olfactory mapping fault prediction system proposed in this invention. Figure 2 This is a schematic diagram illustrating the process of constructing multimodal olfactory map frames in a deep learning-based multimodal olfactory map fault prediction system proposed in this invention. Figure 3 This is a schematic diagram of continuous-time modeling of an improved neural Hawkes process prediction network in a deep learning-based multimodal olfactory mapping fault prediction system proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-3 A deep learning-based multimodal olfactory mapping fault prediction system includes: The data acquisition module is used to collect and synchronously process data to generate multimodal raw olfactory data sequences; The preprocessing module is used to preprocess the raw multimodal olfactory data sequence to generate a preprocessed multimodal olfactory data sequence. The map construction module constructs a multimodal olfactory map frame sequence according to a preset time window from the preprocessed multimodal olfactory data sequence. The feature extraction module inputs the multimodal olfactory map frame sequence into the deep learning feature extraction network to generate a multimodal olfactory high-dimensional feature vector sequence; The event generation module generates multimodal olfactory event sequences based on multimodal olfactory high-dimensional feature vectors; The event prediction module predicts the event trigger probability sequence output by the network through an improved neural Hawkes process. The risk assessment module generates equipment failure prediction results based on the event trigger probability sequence; The early warning output module is used to output fault early warning information and provide equipment fault prediction results when preset trigger conditions are met.

[0020] In this embodiment, the modules are connected through the following method: Gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are collected and processed synchronously according to a unified time reference to form a multimodal raw olfactory data sequence. The original multimodal olfactory data sequence is preprocessed to obtain the preprocessed multimodal olfactory data sequence; The preprocessed multimodal olfactory data sequence is divided into preset time windows, and the features within each time window are jointly mapped to construct a multimodal olfactory atlas frame sequence. The multimodal olfactory map frame sequence is input into a deep learning feature extraction network to extract local spatial patterns, cross-modal association patterns, and temporal variation patterns, resulting in a multimodal olfactory high-dimensional feature vector sequence. Based on the change amplitude, gradient change, or preset event rules of the multimodal olfactory high-dimensional feature vector sequence, event types are generated, and multimodal olfactory event sequences are generated. By inputting multimodal olfactory event sequences into an improved neural Hawkes process prediction network, the event trigger intensity is modeled continuously over time and modulated to obtain an event trigger probability sequence within a future time interval. A fault risk scoring sequence is constructed based on the event trigger probability sequence. The presence of fault precursors is determined based on the fault risk scoring sequence to form a equipment fault prediction result. When the scores in the fault risk scoring sequence meet the preset trigger conditions, a fault warning message is output.

[0021] In this embodiment, the generation of the preprocessed multimodal olfactory data sequence includes: Baseline correction is performed on the gas response data in the multimodal raw olfactory data sequence. The minimum, maximum and response trends over time are calculated for all sampling points of the gas response data in each time window. The minimum and maximum values ​​are used to form a baseline reference value for correction. This baseline reference value is subtracted from all gas response data sampling points in the corresponding time window to generate baseline-corrected gas response data. Drift compensation is performed on the temperature and humidity data in the multimodal raw olfactory data sequence. Smoothing is performed on the temperature data over time sequence and the humidity data over time sequence to obtain drift reference values. The drift reference values ​​are subtracted from the temperature and humidity data at the corresponding time positions to generate drift-compensated temperature and humidity data. Noise suppression is performed on the spectral data in the multimodal raw olfactory data sequence. The variation amplitude of adjacent sampling points in the spectral data is calculated. The variation amplitude is smoothed by a sliding window. Sampling points with abnormal changes are removed according to the variation threshold used for spectral noise identification. The smoothed variation result is mapped back to the corresponding spectral sampling point to generate noise-suppressed spectral data. Scale-unified processing is performed on the volatile organic compound (VOC) data in the multimodal raw olfactory data sequence. Within each time window, the concentration mean, concentration standard deviation, and concentration variation range of the VOC data are calculated. The concentration mean is used as the central reference value, and the concentration standard deviation is used as the scale reference value. The VOC concentration sampling points are converted into dimensionless concentration ratios based on the central reference value and the scale reference value to generate VOC data after scale unification. The baseline-corrected gas response data, drift-compensated temperature and humidity data, noise-suppressed spectral data, and scale-unified volatile organic compound data are recombined in their original chronological order to form a preprocessed multimodal olfactory data sequence.

[0022] In this embodiment, the generation of the multimodal olfactory map frame sequence includes: The preprocessed multimodal olfactory data sequence is divided into preset time windows. Within each preset time window, the corresponding gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are extracted to form data segments corresponding to the preset time windows. Feature extraction is performed on the gas response data, temperature data, humidity data, volatile organic compound data, and spectral data in the data segment. Within the preset time window, the gas response features, temperature features, humidity features, volatile organic compound features, and spectral features are obtained. The gas response characteristics include the gas response change amplitude and the gas response change rate; the temperature characteristics include the temperature change amplitude and the temperature change rate; the humidity characteristics include the humidity change amplitude and the humidity change rate; the volatile organic compound characteristics include the volatile organic compound concentration change amplitude and the volatile organic compound concentration change rate; and the spectral characteristics include the spectral change amplitude and the spectral change rate. The gas response characteristics, temperature characteristics, humidity characteristics, volatile organic compound characteristics and spectral characteristics corresponding to each preset time window are subjected to joint mapping processing to generate the corresponding multimodal olfactory spectrum frame within each preset time window; The generation of the multimodal olfactory atlas frame specifically includes: within each preset time window, the gas response features, temperature features, humidity features, volatile organic compound features and spectral features obtained through feature extraction are structurally fused according to a unified mapping rule; after aligning each modal feature within the time window, they are organized into a two-dimensional feature matrix, a feature multi-channel structure or a modal association graph structure according to a fixed modal order; and the five types of features are combined into a multimodal olfactory atlas frame that can reflect the spatial change features of odor and the multimodal coupling relationship within the time window by feature splicing, channel stacking or modeling the relationship between modalities. The multimodal olfactory atlas frames are arranged sequentially according to the time order of the preset time window to form a multimodal olfactory atlas frame sequence.

[0023] In this embodiment, the generation of the multimodal olfactory high-dimensional feature vector sequence includes: The multimodal olfactory map frame sequence is input into the deep learning feature extraction network frame by frame, so that each multimodal olfactory map frame is simultaneously fed into the convolutional structure, graph structure and attention structure when entering the deep learning feature extraction network; In the convolutional structure, convolution operations are performed on the input multimodal olfactory map frames, and numerical processing is performed on local regions of the map frames within a fixed convolutional region to generate convolutional structure processed data. The generation of the convolutional structure processing data specifically includes: performing convolution operations on the input multimodal olfactory map frames in the convolutional structure according to the preset convolutional kernel size, stride and convolutional step order; performing numerical operations such as weighted summation of adjacent positions, sliding window aggregation, local pattern enhancement or local difference extraction on the values ​​of the map frames in the local area covered by the convolutional kernel, thereby extracting local response patterns in the spatial neighborhood and generating convolutional structure processing data that can be used for feature merging; In the graph structure, numerical propagation processing based on connection relationships is performed on the modal features of each modal feature in the input multimodal olfactory atlas frame. The numerical propagation processing takes each modal feature of the multimodal olfactory atlas frame as a node in the graph structure, and performs weighted aggregation and propagation on the feature values ​​of adjacent nodes according to the connection relationship between nodes. This allows each node to combine relevant information from other modalities during the propagation process to generate graph structure processing data for feature merging. In the attention structure, attention weights are calculated for each feature position in the input multimodal olfactory atlas frame. The attention weight calculation is performed by calculating the numerical correlation between each feature position in the multimodal olfactory atlas frame in the attention structure, assigning attention weights according to the magnitude of the correlation, and then weighting the values ​​of each feature position with the attention weights to generate attention structure processing data for feature merging. The data processed by the convolutional structure, graph structure, and attention structure are numerically merged in a fixed order. The merged result is then subjected to feature compression to form a single-frame high-dimensional feature vector corresponding to each multimodal olfactory map frame. All single-frame high-dimensional feature vectors are arranged in chronological order according to the multimodal olfactory map frame sequence to form a multimodal olfactory high-dimensional feature vector sequence.

[0024] In this embodiment, the formation of the multimodal olfactory event sequence includes: The multimodal olfactory high-dimensional feature vector sequence is read one by one in chronological order. At each time position, the corresponding multimodal olfactory high-dimensional feature vector is obtained, and the change amplitude is calculated. The calculation of the change amplitude specifically includes: during the process of reading the multimodal olfactory high-dimensional feature vector sequence one by one in chronological order, comparing the multimodal olfactory high-dimensional feature vector corresponding to the current time position with the multimodal olfactory high-dimensional feature vector corresponding to the previous time position, performing component-by-component difference operation on each feature component of the two, calculating the numerical difference of each component, and using the absolute value of all component differences or their combination as the change amplitude at that time position, thereby obtaining the change amplitude used for event judgment; For each time position, the multimodal olfactory high-dimensional feature vector is further calculated by dividing the numerical difference between the multimodal olfactory high-dimensional feature vector at that time position and the multimodal olfactory high-dimensional feature vector at the adjacent time position by the time difference between the two time positions, so as to obtain the gradient change used for event judgment. The calculation of the gradient change specifically includes: at each time position, calculating the component-wise numerical difference between the multimodal olfactory high-dimensional feature vector corresponding to the time position and the multimodal olfactory high-dimensional feature vector corresponding to the adjacent time position, and dividing the numerical difference of each component by the time difference between the two time positions to obtain the gradient of each component, and using the result of these gradient changes as the gradient change for event judgment. Based on the magnitude of change, the gradient of change, or the preset event rules, the event judgment is performed on the multimodal olfactory high-dimensional feature vector corresponding to each time position. When the magnitude of change reaches the preset magnitude of change condition, or the gradient of change reaches the preset gradient change condition, or the triggering condition in the preset event rules is met, the event type corresponding to that time position is determined, and that time position is determined as the event occurrence time. Event records are formed by combining the event type, the time of the event, and the multimodal olfactory high-dimensional feature vector corresponding to that time location. All event records are then arranged in chronological order of the event occurrence time to form a multimodal olfactory event sequence.

[0025] In this embodiment, the generation of the event triggering probability sequence within the future time interval includes: The multimodal olfactory event sequence is input into an improved neural Hawkes process prediction network. At the time of input, the event type, the time of occurrence of each event and the corresponding multimodal olfactory high-dimensional feature vector are read in the order of the events. In the improved neural Hawkes process prediction network, the triggering relationship between events is calculated based on the difference between the occurrence times of adjacent events. In the improved neural Hawkes process prediction network, the difference between the occurrence times of adjacent events in the multimodal olfactory event sequence is used as the time input. This time difference is used to calculate the triggering relationship between events, and the event triggering intensity is made to change with the time difference on a continuous time axis according to the event triggering relationship, thereby forming a continuous time modeling result of event triggering intensity. Based on continuous-time modeling of event trigger intensity, the multimodal olfactory high-dimensional feature vector corresponding to each event is input into an improved neural Hawkes process prediction network to modulate the event trigger intensity, so that the event trigger intensity changes with the numerical value of the multimodal olfactory high-dimensional feature vector to obtain the modulated event trigger intensity. Based on the change of the modulated event trigger intensity over time, the event trigger probability is calculated in the future time interval. The event trigger intensity is used as the basis for calculation in the future time interval according to the change of the modulated event trigger intensity over time. Event trigger probability values ​​in the future time interval are generated according to the preset continuous time probability calculation rules, and these event trigger probability values ​​are used to form the event trigger probability sequence in the future time interval.

[0026] In this embodiment, the generation of the equipment fault prediction result includes: The event trigger probability sequence is read sequentially in chronological order, and the corresponding event trigger probability value is obtained at each time position. Based on the preset rules for risk calculation, the event trigger probability value at each time point is used as the basis for calculation. The corresponding risk score value is calculated at each time point, and all risk score values ​​are arranged in the time order of the event trigger probability sequence to form a fault risk score sequence. Based on the fault risk scoring sequence, the risk score value at each time point is used to determine the fault precursor. When the risk score value meets the preset fault precursor conditions, it is determined that there is a fault precursor at that time point. The time location identified as the location of a potential fault precursor is taken as the fault precursor occurrence time, and this fault precursor occurrence time is used as the equipment fault prediction result.

[0027] In this embodiment, the generation of the fault warning information includes: The fault risk score sequence is read one by one in chronological order, the corresponding risk score value is obtained at each time position, and the risk score value is judged according to the preset triggering conditions used for early warning triggering. When the risk score value meets the preset triggering conditions, determine the risk score value corresponding to that time position to trigger the early warning condition, and output the fault early warning information at that time position; The time and location of the output fault warning information are associated with the corresponding equipment fault prediction result, and the equipment fault prediction result is provided to the operation and maintenance process for processing or recording the equipment operating status. Based on the equipment failure prediction results, the equipment failure prediction results are used to formulate maintenance strategies, so as to provide a basis for equipment maintenance strategies when preset trigger conditions are met.

[0028] Example 1: To verify the feasibility of this invention in practice, it was applied to the continuous operation of a large-scale chemical production plant. Such plants release various complex gaseous components during daily operation, and changes in odor often contain information about the operating status of critical equipment. However, due to significant fluctuations in the on-site environment, frequent changes in temperature and humidity, gas sensors are prone to drift, and spectral data exhibits considerable noise, making it difficult for traditional single-mode monitoring methods to accurately identify early equipment anomalies. Especially when equipment failures have not yet developed into significant symptoms, gas changes are often small and exhibit weak characteristics, making it difficult for traditional threshold-based detection methods to reliably capture them. In such cases, equipment often suddenly shuts down or experiences performance degradation without warning, posing a significant risk to production. This invention is precisely proposed to solve this problem of insufficient reliability.

[0029] In the actual operating environment of this chemical plant, odor components in the surrounding air are continuously and in real-time collected by setting up a gas sensor array, temperature sensor, humidity sensor, volatile organic compound sensor, and spectral detection unit. Due to the variable temperature and humidity in the production area, this invention performs targeted preprocessing on data from different modalities after synchronous acquisition. This corrects the baseline of the gas response data, compensates for signal drift caused by temperature and humidity, smooths noise in the spectral data, and standardizes the scale of the volatile organic compound data. The preprocessed multimodal olfactory data sequence has higher readability than the original data, providing a stable input basis for subsequent analysis.

[0030] Under continuous operation, this invention divides the preprocessed data into predetermined time windows, extracts multimodal features within each window, and jointly maps gas response features, temperature features, humidity features, volatile organic compound features, and spectral features to construct a multimodal olfactory map frame sequence that reflects the evolution of odor over time. Through this mapping method, potential coupling relationships between different modalities can be simultaneously represented, and even subtle trends in odor changes can be presented in the map, providing a more intuitive data foundation for subsequent deep learning models to capture key changes.

[0031] After the olfactory map is constructed, a deep learning feature extraction network is used to analyze the olfactory map frame sequence. In practical applications, odor changes during device operation are often complex; for example, changes in certain odor components exhibit nonlinear fluctuations, and different modalities are not synchronized. This invention extracts local spatial patterns from the multimodal map using convolutional structures, captures the correlations between different modalities using graph structures, and strengthens key spatiotemporal regions using attention structures, enabling the system to extract high-dimensional feature vector sequences from a large number of continuous map frames. These high-dimensional feature vectors contain deep information about the device's operating state and can reflect subtle but important odor change trends.

[0032] In practical deployment, by continuously monitoring odor changes within a specific time period, this invention generates event types based on the magnitude and gradient changes of high-dimensional feature vector sequences, thus constructing a multimodal olfactory event sequence. The construction of this event sequence enables the invention to transform continuous olfactory changes into discrete events that can be analyzed over time, providing a clear structured foundation for subsequent continuous-time inference.

[0033] To achieve early prediction of future fault precursors, this invention inputs an improved neural Hawkes process prediction network into the event sequence. Since events in such chemical plants often exhibit time-dependent occurrences during operation, this invention determines the triggering relationship of events by calculating the time difference between adjacent events, allowing the event triggering intensity to change continuously over time. Furthermore, because event intensity is not only time-dependent but also influenced by changes in odor characteristics across multiple modalities, the triggering intensity is modulated by inputting a high-dimensional feature vector of multimodal olfaction, enabling the prediction model to possess sensitivity and adaptability to complex odor data. During continuous operation, the model calculates the probability of event triggering within future time intervals based on the modulated event triggering intensity, generating an event triggering probability sequence.

[0034] After obtaining the event trigger probability sequence, this invention further constructs a fault risk scoring sequence and analyzes the risk scores for different time windows. When the device experiences multiple increases in trigger intensity within a time period, the risk score gradually rises. The presence of potential fault precursors can be identified through the risk scoring sequence. In actual operation, this invention successfully identified an abnormal trend in odor changes some time before a fault occurred, and the system generated a fault precursor judgment result based on the increase in the risk score.

[0035] To ensure the system can assist operators in timely maintenance, this invention automatically outputs fault warning information when the risk score meets preset trigger conditions, and provides the fault prediction results to on-site maintenance personnel for troubleshooting potential equipment hazards. In this practical application scenario, after receiving the warning information, maintenance personnel inspect the relevant parts of the device and confirm that the equipment does indeed show a trend of operational instability. This invention not only provides early fault warnings but also helps on-site personnel avoid potential downtime losses, significantly improving the overall reliability of operation.

[0036] Through long-term application in the operating environment of this chemical production plant, this invention has demonstrated excellent stability and forward-looking capabilities. Multimodal olfactory data maintains high-quality analysis even in complex environments. The combination of deep learning and neural Hawkes processes enables the system to accurately identify weakly changing patterns, achieving early prediction of equipment failure precursors. Data analysis throughout the experimental period shows that this invention exhibits significant advantages in fault monitoring accuracy, early warning capabilities, and environmental adaptability, proving its high application value in industrial scenarios.

[0037] Table 1. Performance Comparison Experimental Results of Multimodal Olfactory Map Fault Prediction System and Traditional Methods

[0038] As can be seen from the table data, the "multimodal olfactory mapping + deep learning + neural Hawkes process" method proposed in this invention is significantly better than traditional methods in all key performance indicators.

[0039] First, regarding prediction accuracy, traditional methods based on empirical thresholds only achieve 68%, while the traditional PCA+SVM method improves to 79%, but this is still significantly lower than the 93% of this invention. The main reason for this difference is that traditional methods rely solely on single-modal or linearly reduced-dimensional features, making it difficult to identify weak pattern changes. In contrast, this invention uses multimodal olfactory mapping to structurally fuse gas response, temperature and humidity, volatile organic compounds, and spectral features, enabling the model to capture complex odor patterns and cross-modal information, thus improving its discriminative ability.

[0040] In terms of recall and F1 score, this invention achieves 89% and 0.91 respectively, which is at least 15 percentage points higher than traditional methods. This indicates that this invention can more effectively identify early abnormal events and reduce missed detections. The core reason is that deep learning feature extraction networks can extract cross-modal correlation patterns when processing multimodal maps, while the improved neural Hawkes process can utilize the temporal dependencies between events to improve prediction accuracy.

[0041] Regarding the false alarm rate, this invention significantly reduces it to 6%, far lower than the 22% of the threshold method and the 14% of the PCA+SVM method. This is due to the stable feature representation after map mapping reducing the influence of environmental noise, while the modulation mechanism in the neural Hawkes process can avoid misjudging random fluctuations as potential anomalies.

[0042] The most practically significant indicator is the advance warning time. This invention can achieve an average advance warning of 5.5 hours, while the traditional threshold method only provides 0.5 hours, and the traditional machine learning method provides about 2 hours. The reason why this invention can significantly improve the advance warning is that the continuous-time modeling of the event triggering intensity can keenly capture the weak changing trend of odor, enabling the system to identify the nascent stage of the fault before it has formed obvious characteristics.

[0043] Overall, this invention achieves predictive performance that is difficult to reach by traditional methods by combining multimodal fusion representation, deep learning high-dimensional feature extraction, and time modeling of neural Hawkes processes. In particular, it significantly improves early warning capabilities and is of great significance to the safe operation of industrial equipment.

[0044] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal olfactory mapping fault prediction system based on deep learning, characterized in that, include: The data acquisition module is used to collect and synchronously process data to generate multimodal raw olfactory data sequences; The preprocessing module is used to preprocess the raw multimodal olfactory data sequence to generate a preprocessed multimodal olfactory data sequence. The map construction module constructs a multimodal olfactory map frame sequence according to a preset time window from the preprocessed multimodal olfactory data sequence. The feature extraction module inputs the multimodal olfactory map frame sequence into the deep learning feature extraction network to generate a multimodal olfactory high-dimensional feature vector sequence; The event generation module generates multimodal olfactory event sequences based on multimodal olfactory high-dimensional feature vectors; The event prediction module predicts the event trigger probability sequence output by the network through an improved neural Hawkes process. The risk assessment module generates equipment failure prediction results based on the event trigger probability sequence; The early warning output module is used to output fault early warning information and provide equipment fault prediction results when preset trigger conditions are met.

2. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 1, characterized in that, The modules are connected in the following way: Gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are collected and processed synchronously according to a unified time reference to form a multimodal raw olfactory data sequence. The original multimodal olfactory data sequence is preprocessed to obtain the preprocessed multimodal olfactory data sequence; The preprocessed multimodal olfactory data sequence is divided into preset time windows, and the features within each time window are jointly mapped to construct a multimodal olfactory atlas frame sequence. The multimodal olfactory map frame sequence is input into a deep learning feature extraction network to extract local spatial patterns, cross-modal association patterns, and temporal variation patterns, resulting in a multimodal olfactory high-dimensional feature vector sequence. Based on the change amplitude, gradient change, or preset event rules of the multimodal olfactory high-dimensional feature vector sequence, event types are generated, and multimodal olfactory event sequences are generated. By inputting multimodal olfactory event sequences into an improved neural Hawkes process prediction network, the event trigger intensity is modeled continuously over time and modulated to obtain an event trigger probability sequence within a future time interval. A fault risk scoring sequence is constructed based on the event trigger probability sequence. The presence of fault precursors is determined based on the fault risk scoring sequence to form a equipment fault prediction result. When the scores in the fault risk scoring sequence meet the preset trigger conditions, a fault warning message is output.

3. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the preprocessed multimodal olfactory data sequence includes: Baseline correction is performed on the gas response data in the multimodal raw olfactory data sequence to generate baseline-corrected gas response data; Drift compensation is performed on the temperature and humidity data in the original multimodal olfactory data sequence to generate drift-compensated temperature and humidity data; Noise suppression is performed on the spectral data in the original multimodal olfactory data sequence to generate noise-suppressed spectral data; The volatile organic compound (VOC) data in the multimodal raw olfactory data sequence is scaled uniformly to generate VOC data after scale uniformity. The baseline-corrected gas response data, drift-compensated temperature and humidity data, noise-suppressed spectral data, and scale-unified volatile organic compound data are recombined in their original chronological order to form a preprocessed multimodal olfactory data sequence.

4. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the multimodal olfactory map frame sequence includes: The preprocessed multimodal olfactory data sequence is divided into preset time windows. Within each preset time window, the corresponding gas response data, temperature data, humidity data, volatile organic compound data, and spectral data are extracted to form data segments corresponding to the preset time windows. Feature extraction is performed on the gas response data, temperature data, humidity data, volatile organic compound data, and spectral data in the data segment. Within the preset time window, the gas response features, temperature features, humidity features, volatile organic compound features, and spectral features are obtained. The gas response characteristics, temperature characteristics, humidity characteristics, volatile organic compound characteristics and spectral characteristics corresponding to each preset time window are subjected to joint mapping processing to generate the corresponding multimodal olfactory spectrum frame within each preset time window; The multimodal olfactory atlas frames are arranged sequentially according to the time order of the preset time window to form a multimodal olfactory atlas frame sequence.

5. A multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the multimodal olfactory high-dimensional feature vector sequence includes: The multimodal olfactory map frame sequence is input frame by frame into a deep learning feature extraction network; In the convolutional structure, convolution operations are performed on the input multimodal olfactory map frames, and numerical processing is performed on local regions of the map frames within a fixed convolutional region to generate convolutional structure processed data. In the graph structure, numerical propagation processing based on connection relationships is performed on the modal features of each modality in the input multimodal olfactory map frame to generate graph structure processed data; In the attention structure, attention weights are calculated for each feature position in the input multimodal olfactory map frame to generate attention structure processing data; The data processed by the convolutional structure, graph structure, and attention structure are numerically merged in a fixed order. The merged result is then subjected to feature compression to form a single-frame high-dimensional feature vector corresponding to each multimodal olfactory map frame. All single-frame high-dimensional feature vectors are arranged in chronological order according to the multimodal olfactory map frame sequence to form a multimodal olfactory high-dimensional feature vector sequence.

6. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The formation of the multimodal olfactory event sequence includes: The multimodal olfactory high-dimensional feature vector sequence is read one by one in chronological order. At each time position, the corresponding multimodal olfactory high-dimensional feature vector is obtained, and the change amplitude is calculated. The gradient of change is further calculated for the multimodal olfactory high-dimensional feature vector at each time location to obtain the gradient change; Based on the magnitude of change, the gradient of change, or the preset event rules, the event judgment is performed on the multimodal olfactory high-dimensional feature vector corresponding to each time position. When the magnitude of change reaches the preset magnitude of change condition, or the gradient of change reaches the preset gradient change condition, or the triggering condition in the preset event rules is met, the event type corresponding to that time position is determined, and that time position is determined as the event occurrence time. The event record is formed by combining the multimodal olfactory high-dimensional feature vectors corresponding to the event type, the time of the event, or the location of the time. All event records are then arranged in chronological order of the event occurrence time to form a multimodal olfactory event sequence.

7. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the event triggering probability sequence within the future time interval includes: The multimodal olfactory event sequence is input into an improved neural Hawkes process prediction network. At the time of input, the event type, the time of occurrence of each event and the corresponding multimodal olfactory high-dimensional feature vector are read in the order of the events. In the improved neural Hawkes process prediction network, the triggering relationship between events is calculated based on the difference between the occurrence times of adjacent events, forming a continuous-time modeling result of the event triggering intensity. Based on continuous-time modeling of event trigger intensity, the multimodal olfactory high-dimensional feature vector corresponding to each event is input into an improved neural Hawkes process prediction network to modulate the event trigger intensity, so that the event trigger intensity changes with the numerical value of the multimodal olfactory high-dimensional feature vector to obtain the modulated event trigger intensity. Based on the change of the modulated event trigger intensity over time, the event trigger probability is calculated in the future time interval to form an event trigger probability sequence in the future time interval.

8. The multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the equipment fault prediction result includes: The event trigger probability sequence is read sequentially in chronological order, and the corresponding event trigger probability value is obtained at each time position. Based on the preset rules for risk calculation, the event trigger probability value at each time point is used as the basis for calculation. The corresponding risk score value is calculated at each time point, and all risk score values ​​are arranged in the time order of the event trigger probability sequence to form a fault risk score sequence. Based on the fault risk scoring sequence, the risk score value at each time point is used to determine the fault precursor. When the risk score value meets the preset fault precursor conditions, it is determined that there is a fault precursor at that time point. The time location identified as the location of a potential fault precursor is taken as the fault precursor occurrence time, and this fault precursor occurrence time is used as the equipment fault prediction result.

9. A multimodal olfactory mapping fault prediction system based on deep learning according to claim 2, characterized in that, The generation of the fault warning information includes: The fault risk score sequence is read one by one in chronological order, the corresponding risk score value is obtained at each time position, and the risk score value is judged according to the preset triggering conditions used for early warning triggering. When the risk score value meets the preset triggering conditions, determine the risk score value corresponding to that time position to trigger the early warning condition, and output the fault early warning information at that time position; The time and location of the output fault warning information are associated with the corresponding equipment fault prediction results, and the equipment fault prediction results are provided to the operation and maintenance process.

Images