A method and system for industrial equipment fault prediction and diagnosis based on a multi-modal large model

Abstract

The application provides an industrial equipment fault prediction and diagnosis method and system based on a multi-modal large model. First, multi-modal time series data of an industrial rotating machine during operation is collected. Second, the data is input into a pre-trained multi-modal large model to extract frequency domain features, hot spot area temperature change trend features, and abnormal sound component features. Then, a cross-modal attention mechanism is used to fuse the three extracted features to generate a unified multi-modal equipment state embedding representation. Then, a comprehensive health index is calculated, and a sequence prediction method is used to deduce the future change trajectory of the comprehensive health index. Finally, when the future change trajectory indicates that the comprehensive health index will cross the preset warning boundary value, a diagnosis conclusion and maintenance decision are output. The technical solution provided by the application not only accurately predicts the early stage of industrial equipment failure, diagnoses the type, and provides forward-looking maintenance decision support, but also improves the intelligent level of equipment health management and operation and maintenance efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence diagnostic technology, and in particular to a method and system for predicting and diagnosing industrial equipment faults based on a multimodal large model. Background Technology

[0002] In the field of intelligent manufacturing, fault prediction and health management of critical equipment (such as large rotating machinery) has become a core requirement in order to minimize the huge economic losses caused by unplanned downtime. In order to comprehensively and accurately assess the health status of equipment, it is necessary to make comprehensive use of multi-dimensional information from different physical sources, such as sensor data of various modes such as vibration, temperature and noise, to capture the complex and subtle coordinated signs of early failures.

[0003] Currently, existing technical solutions are decision fusion methods based on multiple single-modal analysis models. This approach first establishes independent analysis models (such as vibration analysis model, thermal imaging analysis model, and acoustic analysis model) for each single-modal data, such as vibration, infrared thermography, and sound. Each model independently extracts features from its specific data stream and generates a preliminary health status score or failure probability. Subsequently, at the decision level, through a preset rule or weighted fusion mechanism, these independent analysis results from different models are integrated to finally arrive at a comprehensive diagnostic conclusion or predictive judgment.

[0004] However, the existing solutions have obvious drawbacks. Since the analysis models for each mode are used to extract and judge features independently, the solutions are inherently difficult to capture the intrinsic and deep-seated correlations between different physical phenomena. For example, early wear of a bearing may simultaneously lead to the enhancement of specific frequency components in the vibration spectrum, abnormal accumulation of local heat, and an increase in impact events in the noise. However, these cross-modal cooperative change features are treated separately in their respective independent models. This approach leads to insensitivity to complex failure modes and fails to fully explore the complementary and enhanced information value between multimodal data, thereby limiting the accuracy of fault prediction and the reliability of early warning. Summary of the Invention

[0005] This application provides a method and system for predicting and diagnosing industrial equipment faults based on a multimodal large model. This method addresses the problem in existing technologies where multimodal data such as vibration, thermal imaging, and sound are analyzed using independent models and simply fused at the decision-making level. This results in an inability to deeply explore the intrinsic correlations between different physical modes, leading to insensitivity to complex fault modes and insufficient reliability of early warnings.

[0006] Firstly, this application provides a method for industrial equipment fault prediction and diagnosis based on a multimodal large model, including: Simultaneously collect multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data; The vibration waveform data, the temperature distribution image sequence, and the audio waveform data are input into a pre-trained multimodal large model. The multimodal large model is used to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot areas in the temperature distribution image sequence, and the abnormal noise components in the audio waveform data. Within the multimodal large model, the frequency domain features, the temperature change trend features of the hot spot region, and the abnormal noise component features are fused using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation; Based on the multimodal equipment state embedding representation, a comprehensive health index reflecting the health status of the industrial rotating machinery is calculated, and the future change trajectory of the comprehensive health index is deduced using a sequence prediction method. When the future trajectory indicates that the comprehensive health index will exceed the preset warning boundary value, a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery are output.

[0007] Optionally, multimodal time-series data of industrial rotating machinery during operation are collected synchronously, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequences, and audio waveform data, including: Vibration waveform data is acquired at a first sampling rate using a vibration sensor fixed to the bearing housing of the industrial rotating machinery. A temperature distribution image sequence was acquired at a second sampling rate using an optical infrared thermal imager pointed at the key heat-generating component of the industrial rotating machinery. Audio waveform data is acquired at a third sampling rate using an acoustic sensor installed at a designated location near the industrial rotating machinery. Within the same acquisition cycle, each frame of vibration waveform data, each frame of temperature distribution image sequence, and each frame of audio waveform data acquired synchronously are marked with the same time stamp. The vibration waveform data, the temperature distribution image sequence, and the audio waveform data under the same time marker are integrated into multimodal time series data.

[0008] Optionally, the vibration waveform data, the temperature distribution image sequence, and the audio waveform data are input into a pre-trained multimodal large model. The multimodal large model is used to extract the frequency domain features of the vibration waveform data, the temperature change trend features of hotspot areas in the temperature distribution image sequence, and the abnormal noise component features in the audio waveform data, including: The vibration waveform data is converted to a frequency representation, and the harmonic components and amplitude information associated with the rotation frequency of the industrial rotating machinery are identified from the frequency representation as frequency domain features. In each frame of the temperature distribution image sequence, the region whose temperature value exceeds the average temperature range of the surrounding preset area is identified as a hot spot region, and the temperature change pattern of the hot spot region in the image sequence is tracked as the temperature change trend feature of the hot spot region. Identify audio segments in the audio waveform data that differ from the reference sound pattern during stable operation of the industrial rotating machinery, and separate transient pulse signals with specific repetition patterns from the audio segments as abnormal noise components.

[0009] Optionally, within the multimodal large model, a cross-modal attention mechanism is used to fuse the frequency domain features, the temperature change trend features of the hotspot region, and the abnormal noise component features to generate a unified multimodal device state embedding representation, including: The frequency domain features, the temperature change trend features of the hot spot area, and the abnormal noise component features are respectively mapped to the same semantic space to obtain vibration feature vector, temperature feature vector, and acoustic feature vector; The interaction weights of the vibration feature vector with respect to the temperature feature vector and the acoustic feature vector are calculated to obtain the first interaction weight. At the same time, the interaction weights of the temperature feature vector with respect to the vibration feature vector and the acoustic feature vector are calculated to obtain the second interaction weight. Finally, the interaction weights of the acoustic feature vector with respect to the vibration feature vector and the temperature feature vector are calculated to obtain the third interaction weight. The temperature feature vector is weighted using the weights of the vibration feature vector to the temperature feature vector in the first interaction weights, and the acoustic feature vector is weighted using the weights of the vibration feature vector to the acoustic feature vector in the first interaction weights. The weighted temperature feature vector and the weighted acoustic feature vector are then combined with the vibration feature vector to obtain an enhanced vibration context vector. The vibration feature vector is weighted using the weights of the temperature feature vector to the vibration feature vector in the second interaction weights, and the acoustic feature vector is weighted using the weights of the temperature feature vector to the acoustic feature vector in the second interaction weights. The weighted vibration feature vector and the weighted acoustic feature vector are then combined with the temperature feature vector to obtain an enhanced temperature context vector. The vibration feature vector is weighted using the weights of the acoustic feature vector to the vibration feature vector in the third interactive weighting, and the temperature feature vector is weighted using the weights of the acoustic feature vector to the temperature feature vector in the third interactive weighting. The weighted vibration feature vector and the weighted temperature feature vector are then combined with the acoustic feature vector to obtain an enhanced acoustic context vector. The enhanced vibration context vector, the enhanced temperature context vector, and the enhanced acoustic context vector are combined and transformed to generate a unified multimodal device state embedding representation.

[0010] Optionally, based on the multimodal equipment state embedding representation, a comprehensive health index reflecting the health status of the industrial rotating machinery is calculated, and a sequence prediction method is used to extrapolate the future change trajectory of the comprehensive health index, including: The multimodal device state embedding representation is combined and calculated based on preset weight coefficients to obtain a comprehensive health index representing the overall state of the device at the current moment. The comprehensive health index is recorded continuously at multiple points in time in chronological order, forming a historical sequence of the comprehensive health index. Using the historical sequence of the comprehensive health index as input, a pre-trained sequence analysis model is used to capture the change patterns contained in the historical sequence of the comprehensive health index. Based on the captured change patterns, the values ​​of the comprehensive health index at multiple future time points are continuously extrapolated to obtain the future change trajectory of the comprehensive health index.

[0011] Optionally, based on the captured change patterns, the values ​​of the comprehensive health index at multiple future time points are continuously extrapolated to obtain the future change trajectory of the comprehensive health index, including: The captured change patterns are decomposed into trend components that characterize the long-term evolution direction and cyclical components that characterize periodic fluctuations. Based on the trend components, the overall upward and downward trend of the comprehensive health index over a future preset time period can be inferred; Based on the cyclical components, the fluctuation pattern of the comprehensive health index around the overall trend in the future time period can be inferred; The inferred overall trend of rise and fall is superimposed with the fluctuation pattern to calculate the first predicted comprehensive health index at the first future time point; The first predicted comprehensive health index is incorporated into the end of the historical sequence of the comprehensive health index, the change pattern is updated, and the second predicted comprehensive health index at the second future time point is calculated based on the updated change pattern. Repeat the incorporation and calculation process to obtain the predicted comprehensive health index for multiple consecutive future time points. The predicted comprehensive health indexes arranged in chronological order constitute the future change trajectory of the comprehensive health index.

[0012] Optionally, when the future trajectory indicates that the comprehensive health index will exceed a preset warning boundary value, a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery are output, including: Each predicted comprehensive health index in the future change trajectory is compared with a preset warning boundary value, and the first future change trajectory point with a value lower than the preset warning boundary value is identified as a warning point; Calculate the time length from the current moment to the warning point as the warning lead time; Based on the relative significance of the vibration feature vector, temperature feature vector, and acoustic feature vector in the multimodal device state embedding representation, the main combination of influencing factors leading to the decline of the comprehensive health index is determined. The main influencing factors are combined and matched with preset fault knowledge entries to obtain successfully matched fault knowledge entries. Based on the successfully matched fault knowledge entries, the potential fault types of the industrial rotating machinery are determined, and combined with the early warning period, a diagnostic conclusion and maintenance decision containing the potential fault types and recommended maintenance time windows are generated.

[0013] Secondly, this application provides an industrial equipment fault prediction and diagnosis system based on a multimodal large model, including: The acquisition module is used to synchronously acquire multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data; The extraction module is used to input the vibration waveform data, the temperature distribution image sequence, and the audio waveform data into a pre-trained multimodal large model, and use the multimodal large model to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot areas in the temperature distribution image sequence, and the abnormal noise component features in the audio waveform data; The fusion module is used to fuse the frequency domain features, the temperature change trend features of the hot spot region, and the abnormal noise component features within the multimodal large model using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation; The calculation module is used to calculate a comprehensive health index reflecting the health status of the industrial rotating machinery based on the multimodal equipment state embedding representation, and to use a sequence prediction method to deduce the future change trajectory of the comprehensive health index. The output module is used to output diagnostic conclusions and maintenance decisions pointing to the potential fault types of the industrial rotating machinery when the future change trajectory indicates that the comprehensive health index will exceed the preset warning boundary value.

[0014] Thirdly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the industrial equipment fault prediction and diagnosis method based on a multimodal large model as described in the first aspect above.

[0015] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the industrial equipment fault prediction and diagnosis method based on a multimodal large model as described in the first aspect.

[0016] This application achieves comprehensive and in-depth perception of the operating status of industrial equipment by simultaneously acquiring multimodal time-series data such as vibration, thermal imaging, and audio, and using a pre-trained multimodal large model for deep feature extraction and cross-modal fusion. This method can effectively capture subtle cooperative change patterns between different physical signals, thereby improving the sensitivity of early composite fault signs identification and the accuracy of status assessment, and providing a reliable data foundation for early fault prediction.

[0017] Furthermore, by analyzing the future trajectory of the comprehensive health index and correlating the significance of multimodal feature vectors, this invention not only achieves early warning of faults but also accurately locates the dominant factors and types of potential faults. The method ultimately generates decision recommendations that include specific fault types and optimal maintenance windows, upgrading traditional status monitoring to intelligent predictive maintenance guidance, thereby effectively supporting maintenance personnel to take timely and targeted maintenance actions and avoid unplanned downtime.

[0018] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a method for predicting and diagnosing industrial equipment faults based on a multimodal large model, as provided in this application, is shown. Figure 2A schematic diagram of the structure of an industrial equipment fault prediction and diagnosis system based on a multimodal large model provided in this application is shown. Figure 3 A schematic diagram of the structure of a computing device provided in this application is shown. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0022] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0023] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Figure 1 This application provides a flowchart of a method for industrial equipment fault prediction and diagnosis based on a multimodal large model, such as... Figure 1 As shown, the method includes: Step 101: Synchronously collect multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data.

[0025] Optionally, step 101 may specifically include the following steps: Step 1011: Vibration waveform data is acquired at a first sampling rate using a vibration sensor fixed to the bearing housing of the industrial rotating machinery. Step 1012: Acquire a temperature distribution image sequence at a second sampling rate using an optical infrared thermal imager pointed at the key heat-generating component of the industrial rotating machinery; Step 1013: Acquire audio waveform data at a third sampling rate using an acoustic sensor installed at a designated location near the industrial rotating machinery. Step 1014: Within the same acquisition cycle, mark each frame of vibration waveform data, each frame of temperature distribution image sequence, and each frame of audio waveform data acquired synchronously with the same time stamp. Step 1015: Integrate the vibration waveform data, the temperature distribution image sequence, and the audio waveform data under the same time mark into multimodal time series data.

[0026] In the above scheme, industrial rotating machinery refers to equipment that relies on rotor rotation as its core working mode, such as generators, fans, and water pumps; Multimodal time-series data refers to a collection of multiple types of data with a time sequence collected from different physical sensors within the same time period, used to comprehensively describe the status of the equipment; Vibration waveform data is a one-dimensional data sequence that records the change of vibration amplitude on the surface of equipment over time, and is used to analyze mechanical vibration characteristics. A temperature distribution image sequence is a collection of two-dimensional images taken by an infrared thermal imager that display the temperature distribution on the surface of a device in chronological order, used to monitor thermal status. Audio waveform data is a one-dimensional data sequence that records the change in sound intensity of a device over time, and is used to analyze acoustic characteristics; A vibration sensor is a device that converts mechanical vibrations into electrical signals; An optical infrared thermal imager is a device that receives infrared radiation and generates an image of temperature distribution. An acoustic sensor is a device that converts sound signals into electrical signals (i.e., a microphone). A time stamp is a unique timestamp used to identify data collected by different sensors at the same instant.

[0027] In this scheme, firstly, a vibration sensor tightly mounted on the monitored equipment (such as a bearing housing) continuously records vibration waveform data during equipment operation at a preset first sampling rate (e.g., sampling times per second). Secondly, an optical infrared thermal imager aimed at the heat-prone area of ​​the equipment (such as the motor housing) continuously captures a temperature distribution image sequence at a preset second sampling rate (e.g., frames per second). Next, an acoustic sensor installed near the equipment to clearly capture the location of operating noise continuously records audio waveform data at a preset third sampling rate. Then, the data acquisition system ensures that within each identical acquisition cycle, a completely identical time stamp is assigned to the short segment of vibration waveform data, a frame of temperature distribution image sequence, and a frame of audio waveform data just acquired by the three sensors. Finally, based on this same time stamp, the three types of data belonging to the same moment are associated and packaged into a structured multimodal time-series data unit for subsequent analysis.

[0028] For example, in predictive maintenance of a wind turbine generator (equipment A) in a power plant, technicians installed vibration sensor B on the drive-end bearing housing of the generator (equipment A) to collect vibration waveform data at a rate of 51,200 times per second (first sampling rate). Simultaneously, an optical infrared thermal imager C was aimed at the bearing housing of the generator (equipment A) to capture a sequence of temperature distribution images at a rate of 25 frames per second (second sampling rate). An acoustic sensor D was installed inside the nacelle near the generator (equipment A) to collect audio waveform data at a rate of 44,100 times per second (third sampling rate). The data acquisition system uses a cycle of 0.04 seconds to assign the same millisecond-accurate timestamp to the data synchronously collected by sensors B, C, and D at that moment (e.g., "2023-10-27 10:00:00.000 UTC"). These three types of data with the same timestamp are packaged and stored to form a multimodal time-series data record. By continuing this process, a data foundation for fault analysis is accumulated.

[0029] This solution achieves high-quality synchronous acquisition of three key physical phenomena—vibration, temperature, and sound—of industrial rotating machinery by coordinating the deployment of multiple sensors and employing precise time synchronization technology. The obtained multimodal time-series data are strictly aligned in time, providing a reliable and consistent data foundation for subsequent deep fusion and precise analysis, fundamentally ensuring the comprehensiveness and accuracy of equipment status perception.

[0030] Step 102: Input the vibration waveform data, the temperature distribution image sequence, and the audio waveform data into a pre-trained multimodal large model, and use the multimodal large model to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot areas in the temperature distribution image sequence, and the abnormal noise components in the audio waveform data.

[0031] Optionally, step 102 may specifically include the following steps: Step 1021: Convert the vibration waveform data to a frequency representation, and identify the harmonic components and amplitude information associated with the rotation frequency of the industrial rotating machinery from the frequency representation as frequency domain features; Step 1022: Locate the region in each frame of the temperature distribution image sequence whose temperature value exceeds the preset range of the average temperature of the surrounding preset region as a hot spot region, and track the temperature value change pattern of the hot spot region in the image sequence as the temperature change trend feature of the hot spot region. Step 1023: Identify audio segments in the audio waveform data that differ from the reference sound pattern during stable operation of the industrial rotating machinery, and separate transient pulse signals with specific repetition patterns from the audio segments as abnormal noise components.

[0032] In the above scheme, the pre-trained multimodal large model refers to a complex computational model that has been trained with a large amount of data and is capable of simultaneously understanding and processing multiple types of information (such as vibration, image, and sound). Frequency domain features are the periodic fluctuation components (harmonic components) and their intensity (amplitude information) related to the equipment rotation speed that are identified after converting the vibration signal from a time amplitude representation to a frequency component representation. These features are used to reveal the essential frequency source of the vibration. The hotspot area temperature change trend feature is to identify areas in the infrared image sequence whose temperature is significantly higher than the surrounding average level (hotspot area), and track the pattern of temperature change in this area over time (temperature value change pattern) to monitor the evolution of abnormal hot spots. Abnormal noise component characteristics are sound segments (audio fragments with differences) that are different from the standard sound (reference sound mode) during normal and stable operation of the equipment. They are then separated into brief burst sound signals with a fixed repeating rhythm (transient pulse signals with a specific repeating pattern) to capture abnormal impact or friction sounds.

[0033] In this scheme, firstly, a mathematical transformation is performed on the vibration waveform data, converting it from a graph representing how the amplitude changes over time into a graph representing the frequency components of the signal and the magnitude of each component, i.e., a frequency representation. Then, from this frequency representation, periodic components whose frequency values ​​are integer multiples of the equipment's rotation frequency are identified, i.e., harmonic components, and the intensity of each found harmonic component, i.e., amplitude information, are recorded as frequency domain features. Secondly, each image in the temperature distribution image sequence is processed. First, the average temperature of a small area in the image is calculated, i.e., the average temperature of the surrounding preset area. Then, all areas in the image whose temperature values ​​are higher than this average temperature by a specific value, i.e., the preset range area, are identified. These areas are marked as hotspots. Then, in a continuous image sequence, the temperature at the center point of these hotspots is tracked frame by frame to extract the temperature change pattern as a feature of the hotspot temperature change trend. Finally, the real-time acquired audio waveform data is compared with a pre-stored template representing the stable operating sound of the device under healthy conditions, i.e., a reference sound pattern, to identify parts of the sound waveform that are inconsistent, i.e., audio segments with differences. In these audio segments with differences, pulse signals with extremely short duration but high intensity that repeat according to the device rotation cycle or other fixed intervals are further detected and separated, i.e., transient pulse signals with specific repetition patterns, as features of abnormal noise components.

[0034] Following the specific implementation of the previous scheme, the power plant inputs the collected vibration waveform data, temperature distribution image sequence, and audio waveform data of generator A into a pre-trained multimodal large model. The multimodal large model first processes the vibration data, converting it into a spectrum (frequency representation), and identifies vibration components (harmonic components) with frequencies 1, 2, and 3 times the generator speed (e.g., 20 Hz) and their intensities (amplitude information). Simultaneously, the model analyzes the temperature distribution image sequence captured by infrared thermal imager C, locating areas in each frame that are more than 10 degrees Celsius higher than the average temperature of the surrounding area as hotspots. It also observes that the temperature in these hotspots gradually increases from 85 degrees Celsius to 92 degrees Celsius in consecutive images, forming a temperature change pattern. Furthermore, the multimodal large model compares the current audio waveform data collected by acoustic sensor D with the stored normal operating sound (reference sound pattern), identifying an audio segment with a periodic "click" sound, and separating the transient pulse signal that appears once per revolution. Thus, the model completes the process of extracting key features from the multimodal data.

[0035] This solution uses a pre-trained multimodal large model to perform in-depth analysis of vibration, temperature, and sound data, extracting frequency components that directly reflect the essence of the mechanical state, the dynamic evolution trend of abnormal heating areas, and acoustic pulses corresponding to abnormal physical impacts. These features accurately capture potential abnormal signs of equipment operation from different physical dimensions, laying a precise and reliable feature foundation for subsequent multimodal information fusion and intelligent diagnosis.

[0036] Step 103: Within the multimodal large model, the frequency domain features, the temperature change trend features of the hot spot region, and the abnormal noise component features are fused using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation.

[0037] Optionally, step 103 may specifically include the following steps: Step 1031: Map the frequency domain features, the temperature change trend features of the hot spot area, and the abnormal noise component features to the same semantic space to obtain vibration feature vector, temperature feature vector, and acoustic feature vector; Step 1032: Calculate the interaction weight of the vibration feature vector with respect to the temperature feature vector and the acoustic feature vector to obtain the first interaction weight; simultaneously calculate the interaction weight of the temperature feature vector with respect to the vibration feature vector and the acoustic feature vector to obtain the second interaction weight; and calculate the interaction weight of the acoustic feature vector with respect to the vibration feature vector and the temperature feature vector to obtain the third interaction weight. Step 1033: Use the weights of the vibration feature vector to the temperature feature vector in the first interaction weights to weight the temperature feature vector, and use the weights of the vibration feature vector to the acoustic feature vector in the first interaction weights to weight the acoustic feature vector. Combine the weighted temperature feature vector and the weighted acoustic feature vector with the vibration feature vector to obtain an enhanced vibration context vector. Step 1034: Use the weights of the temperature feature vector to the vibration feature vector in the second interaction weights to weight the vibration feature vector, and use the weights of the temperature feature vector to the acoustic feature vector in the second interaction weights to weight the acoustic feature vector. Combine the weighted vibration feature vector and the weighted acoustic feature vector with the temperature feature vector to obtain an enhanced temperature context vector. Step 1035: Use the weights of the acoustic feature vector to the vibration feature vector in the third interactive weights to weight the vibration feature vector, and use the weights of the acoustic feature vector to the temperature feature vector in the third interactive weights to weight the temperature feature vector. Combine the weighted vibration feature vector and the weighted temperature feature vector with the acoustic feature vector to obtain the enhanced acoustic context vector. Step 1036: Combine the enhanced vibration context vector, the enhanced temperature context vector, and the enhanced acoustic context vector and generate a unified multimodal device state embedding representation through a transformation operation.

[0038] In the above scheme, the cross-modal attention mechanism is a computational method that simulates human attention allocation and is used to evaluate and quantify the degree of correlation between different types of features (such as vibration, temperature, and sound features). Multimodal device state embedding representation is a comprehensive digital vector that integrates deep information from data from different sensors to comprehensively and compactly characterize the overall health status of the device. The same semantic space is a virtual public coordinate system that allows features from different sources to be compared and calculated fairly. Vibration feature vector, temperature feature vector, and acoustic feature vector are a set of numbers converted from frequency domain features, hot spot temperature change trend features, and abnormal noise component features, respectively. They represent their original information in the same semantic space. The first interaction weight, the second interaction weight, and the third interaction weight are the importance scores of other feature vectors to themselves, calculated from the perspectives of vibration, temperature, and sound feature vectors, respectively. Enhanced vibration context vector, enhanced temperature context vector, and enhanced acoustic context vector are new feature vectors containing richer correlation information obtained by incorporating weighted information from other modes into their respective feature vectors.

[0039] In this scheme, firstly, frequency domain features, hotspot temperature change trend features, and abnormal sound component features from different sensors, each with varying forms, are projected into a single semantic space through a transformation process, resulting in vibration feature vectors, temperature feature vectors, and acoustic feature vectors that are dimensionally consistent and comparable. Secondly, within this same semantic space, the correlation between the vibration feature vector and the temperature and acoustic feature vectors is calculated to obtain a first interaction weight. The correlation between the temperature feature vector and the vibration and acoustic feature vectors is then calculated to obtain a second interaction weight. Finally, the correlation between the acoustic feature vector and the vibration and temperature feature vectors is calculated to obtain a third interaction weight. Next, the weight value representing the vibration's sensitivity to temperature in the first interaction weight is used to amplify or reduce the temperature feature vector, and the weight value representing the vibration's sensitivity to sound is used to adjust the acoustic feature vector. Then, the two adjusted temperature and acoustic feature vectors are combined with the original vibration feature vector to form an enhanced vibration context vector containing temperature and sound context information. Next, using the attention weight of temperature to vibration and sound in the second interaction weight, the vibration feature vector and acoustic feature vector are similarly adjusted and combined with the original temperature feature vector to obtain an enhanced temperature context vector. Then, using the attention weight of acoustics to vibration and temperature in the third interaction weight, the vibration feature vector and temperature feature vector are adjusted and combined with the original acoustic feature vector to obtain an enhanced acoustic context vector. Finally, the three enhanced vibration context vector, enhanced temperature context vector, and enhanced acoustic context vector containing cross-modal information are concatenated together and compressed and integrated through a linear transformation operation to generate a single, highly integrated, unified multimodal device state embedding representation.

[0040] Following the specific implementation of the previous scheme, the frequency domain characteristics of generator A (such as the increase in the amplitude of the second harmonic), the temperature change trend characteristics of the hot spot area (the temperature of the bearing part continues to rise), and the abnormal noise component characteristics (impact sound pulses per revolution) are input into a multimodal large model. The model first converts these features into numerical vectors (vibration feature vector, temperature feature vector, and acoustic feature vector). Then, a cross-modal attention mechanism calculates highly correlated weights: the vibration feature vector has a high weight on the acoustic feature vector (because both vibration and abnormal noise exhibit periodic impact characteristics), and the temperature feature vector has a relatively high weight on the vibration feature vector (the temperature rise may be related to increased friction). Then, the model uses these weights to generate enhanced context vectors. For example, the enhanced vibration context vector incorporates weighted abnormal sound and temperature rise information. Finally, all these enhanced vectors are merged and compressed into a unified multimodal device state embedding representation that includes the collaborative fault mode of "periodic impact, temperature rise, and abnormal noise".

[0041] This scheme dynamically evaluates and integrates the deep correlation between vibration, temperature, and sound features through a cross-modal attention mechanism. This results in a unified multimodal device state embedding representation that not only includes the information of each mode itself, but also contains collaborative diagnostic information that corroborates and complements each other. This constructs a comprehensive device state representation that can more fully and accurately reflect the essence of complex faults.

[0042] Step 104: Based on the multimodal equipment state embedding representation, calculate the comprehensive health index reflecting the health status of the industrial rotating machinery, and use the sequence prediction method to deduce the future change trajectory of the comprehensive health index.

[0043] Optionally, step 104 may specifically include the following steps: Step 1041: Perform a combination calculation based on preset weight coefficients on each dimension of the multimodal device state embedding representation to obtain a comprehensive health index representing the overall state of the device at the current moment. Step 1042: Record the comprehensive health index at multiple time points in chronological order to form a historical sequence of the comprehensive health index; Step 1043: Using the historical sequence of the comprehensive health index as input, capture the change patterns contained in the historical sequence of the comprehensive health index through a pre-trained sequence analysis model; Step 1044: Based on the captured change patterns, continuously extrapolate the values ​​of the comprehensive health index at multiple future time points to obtain the future change trajectory of the comprehensive health index.

[0044] Step 1044 may specifically include the following steps: The captured change patterns are decomposed into trend components representing long-term evolution and cyclical components representing periodic fluctuations. Based on the trend components, the overall upward and downward trend of the comprehensive health index over a predetermined future time period is inferred. Based on the cyclical components, the fluctuation pattern of the comprehensive health index around the overall upward and downward trend over the future time period is inferred. The inferred overall upward and downward trend is superimposed with the fluctuation pattern to calculate the first predicted comprehensive health index at the first future time point. The first predicted comprehensive health index is merged into the end of the historical sequence of the comprehensive health index to update the change pattern, and a second predicted comprehensive health index at the second future time point is calculated based on the updated change pattern. This merging and calculation process is repeated to obtain the predicted comprehensive health indices at multiple consecutive future time points. The predicted comprehensive health indices arranged in chronological order constitute the future change trajectory of the comprehensive health index.

[0045] In the above scheme, the comprehensive health index is a single value. It is calculated by combining all the dimensional information of the multimodal device state embedding representation according to a set of preset importance ratios (preset weight coefficients). It is used to quantify the overall state of the device at the current moment. The higher the value, the healthier it is usually. Sequence prediction is a technique that analyzes chronologically ordered data (sequences) to predict future values. The historical sequence of the comprehensive health index is a sequence of comprehensive health indices from multiple past moments arranged in chronological order. A pre-trained sequence analysis model is a computer model that has already learned to identify patterns in time series data; The change pattern is the data change pattern identified by the sequence analysis model from the historical sequence of the comprehensive health index; The future trajectory of the comprehensive health index is a curve formed by the predicted values ​​of the comprehensive health index at a series of future time points, derived from the change pattern. Trend components are the directional parts of a change pattern that represent a long-term, slow rise or fall in data. Cyclical components are the parts of the change pattern that represent data that fluctuate periodically around a long-term trend. The overall trend is the general direction of future health index changes (continuous improvement or deterioration) inferred from the trend components. The fluctuation pattern is the rhythm and amplitude of future health index fluctuations around a major trend, inferred from cyclical components. The first predicted comprehensive health index is the predicted health index value at the first point in the future; The updated change pattern is a new pattern obtained by reanalyzing the historical sequence after adding the first predictive comprehensive health index to the end. The second predictive composite health index is a forecast value for the next point in time derived from the updated change pattern.

[0046] In this scheme, firstly, the multimodal device state, representing the overall state of the equipment, is embedded into a vector containing multiple numbers. Each number (dimension) is multiplied by a pre-defined coefficient reflecting the importance of the dimension (pre-defined weighting coefficient). All the multiplication results are summed to obtain a single value, the comprehensive health index, which intuitively reflects the overall state of the equipment at the current moment. Secondly, the system continuously runs, and each time a new comprehensive health index is calculated, it is added to the previously recorded indices in chronological order, forming a continuously extending data sequence arranged by time—the comprehensive health index historical sequence. Next, this comprehensive health index historical sequence, recording the changes in the equipment's health status over a period of time, is input into a pre-trained sequence analysis model to analyze the sequence and identify patterns in the data changes, i.e., capturing the inherent change patterns. Finally,… Based on the captured change patterns, future index values ​​are deduced step by step: First, the change patterns are broken down into trend components representing long-term direction and cyclical components representing periodic fluctuations. The trend component determines whether the index will rise or fall overall in the future (overall upward and downward trend), and the cyclical component determines how the index will fluctuate around the overall trend (fluctuation pattern). These two are superimposed to calculate the first predicted comprehensive health index. Then, this predicted value is treated as actual data that has already occurred and is hypothetically added to the end of the historical sequence of the comprehensive health index. Based on this extended new sequence, the sequence analysis model will fine-tune its understanding of the pattern to obtain an updated change pattern, and calculate the second predicted comprehensive health index accordingly. By repeating the process of "prediction-addition-update-re-prediction", a series of future predicted values ​​can be obtained. Arranged in chronological order, they form the future change trajectory of the comprehensive health index.

[0047] Following the specific implementation of the previous scheme, the power plant system, based on the unified multimodal equipment state embedding representation of generator A that integrates multimodal information, calculates its comprehensive health index as 85 (out of 100) through weighted summation. The system continuously records this index hourly over the past week, forming a historical sequence of the comprehensive health index (e.g., [88, 87, 86, 85, 85, 84, ...]). This sequence is input into a pre-trained sequence analysis model, which analyzes that the sequence exhibits a slow downward trend and slight daily cyclical fluctuations. Based on this, the model predicts the future trajectory of the comprehensive health index for the next 24 hours: it predicts that the index will continue to decline slowly at a rate of approximately 0.5 points per day in the coming days (overall upward and downward trend), with slight diurnal fluctuations (fluctuation pattern), thus obtaining a series of predicted comprehensive health indices such as [83.5, 83.2, 82.8, ...].

[0048] This solution condenses complex multimodal fusion information into a single health indicator that is easy to monitor. By analyzing the historical change patterns of this indicator, it enables the prediction of the future health status trend of the equipment, thereby elevating the system's capabilities from current status assessment to forward-looking prediction. This provides a critical time window and decision-making basis for early detection of potential risks and planning of maintenance actions.

[0049] Step 105: When the future change trajectory indicates that the comprehensive health index will exceed the preset warning boundary value, output a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery.

[0050] Optionally, step 105 may specifically include the following steps: Step 1051: Compare each predicted comprehensive health index in the future change trajectory with a preset warning boundary value, and identify the first future change trajectory point with a value lower than the preset warning boundary value as a warning point; Step 1052: Calculate the time length from the current moment to the warning point as the warning lead time; Step 1053: Based on the relative significance of the vibration feature vector, temperature feature vector, and acoustic feature vector in the multimodal device state embedding representation, determine the combination of main influencing factors that lead to the decline of the comprehensive health index; Step 1054: Match the combination of the main influencing factors with the preset fault knowledge entries to obtain successfully matched fault knowledge entries. Step 1055: Based on the successfully matched fault knowledge entries, determine the potential fault types of the industrial rotating machinery, and in conjunction with the early warning lead time, generate a diagnostic conclusion and maintenance decision that includes the potential fault types and recommended maintenance time windows.

[0051] In the above scheme, the preset warning boundary value is a pre-set health index threshold. When the index is lower than this value, the equipment is considered to be in a dangerous state. Potential fault types are the specific fault categories that the system infers may occur in the equipment (such as bearing wear, imbalance, etc.). The diagnostic conclusion is the system's final assessment of the equipment's health status; Maintenance decisions are specific maintenance action recommendations based on diagnostic findings. The future trajectory points are each specific numerical point on the predicted future health index curve; The warning point is the first point on the future trajectory that falls below the warning value, marking the moment when an anomaly is expected to occur. The early warning lead time is the time from the current moment to the warning point, allowing time for maintenance preparations; Relative significance refers to the relative proportion of the contributions of vibration, temperature, and sound feature vectors to the overall representation in the state embedding representation of a multimodal device; The main influencing factor combination is the main characteristic category that leads to a decline in the health index, identified based on the relative significance (e.g., "mainly vibration and sound abnormalities"). The preset fault knowledge entries are rules pre-stored in the knowledge base, recording the correspondence between different combinations of influencing factors and fault types; Successfully matched fault knowledge entries are those found by querying the knowledge base and matching the current combination of major influencing factors; The recommended maintenance window is a time interval for suggested maintenance determined by combining early warning points and time buffer periods.

[0052] In this scheme, firstly, the system examines each predicted comprehensive health index (i.e., future trajectory point) on the future change trajectory one by one, compares them with preset warning boundary values, and identifies the first point whose value is less than the boundary value, marking this point as a warning point. Secondly, the system calculates the time interval between the "current" time point of prediction and the warning point identified in step 1051; this time interval is the warning lead time. Next, the system retrospectively analyzes the multimodal device state embedding representation on which the health index is based, and by analyzing its internal composition, calculates the weight ratio of the vibration feature vector, temperature feature vector, and acoustic feature vector in the composition of this comprehensive representation, i.e., their relative significance, and determines which feature(s) (e.g., the most prominent vibration anomaly) is the main cause of the index decline. The system identifies a combination of influencing factors. Then, it uses the identified main influencing factor combination (e.g., "highly significant vibration and acoustic characteristics") as a query condition to match entries in a pre-established fault knowledge base (pre-defined fault knowledge entries, such as "if both vibration and sound characteristics are significantly abnormal, then the corresponding bearing is damaged"). This allows the system to find one or more successfully matched fault knowledge entries. Finally, the system extracts the corresponding fault description from the successfully matched fault knowledge entries, identifies it as a potential fault type, and combines this with the early warning lead time calculated in step 1052. An additional buffer time for preparing for maintenance is added to determine a recommended maintenance time window. Ultimately, this information is integrated into a clear diagnostic conclusion (e.g., "suspected early bearing damage") and a specific maintenance decision (e.g., "It is recommended to schedule a shutdown inspection within the next two weeks").

[0053] Following on from the previous specific implementation, the power plant system will predict future change trajectories (such as [83.5, 83.2, 82.8, ... 79.0, 78.5, ... 78.5, ... 79.0, ...9. ...]) is compared with the preset warning boundary value (e.g., 80), and it is found that the 18th predicted point (value 79.0) is the first point below 80, and this point is identified as the warning point; the system calculates the warning lead time from the current point to the 18th point (assuming a time interval of 18 hours) as 18 hours; at the same time, the system analysis finds that in the multimodal equipment state embedding representation that leads to the decline in health index, the relative significance of vibration feature vector and acoustic feature vector is much higher than that of temperature feature vector, thus determining that the main influencing factor combination is "vibration and acoustic anomaly dominance"; this combination is matched with the preset fault knowledge base, and the successfully matched fault knowledge entry indicates that the combination corresponds to "rolling bearing damage"; therefore, the system generates the diagnostic conclusion "early damage risk exists in the drive end bearing of generator A", and combined with the 18-hour warning lead time and the required preparation time, generates the maintenance decision "it is recommended to arrange inspection and replacement within the next 24-36 hours".

[0054] This solution combines predictive information with domain knowledge to transform abstract health index predictions into specific, actionable maintenance guidance. It not only indicates "when problems may occur" (early warning points and lead times), but also diagnoses "what problems may occur" (potential fault types) and "when and how to respond" (maintenance decisions). This completes a closed loop from state perception, predictive warning to decision support, providing a direct basis for implementing precise and efficient predictive maintenance.

[0055] Figure 2 This application provides a schematic diagram of the structure of an industrial equipment fault prediction and diagnosis system based on a multimodal large model, as shown below. Figure 2 As shown, the system includes: The acquisition module 21 is used to synchronously acquire multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data; Extraction module 22 is used to input the vibration waveform data, the temperature distribution image sequence and the audio waveform data into a pre-trained multimodal large model, and use the multimodal large model to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot area in the temperature distribution image sequence and the abnormal noise component features in the audio waveform data; Fusion module 23 is used to fuse the frequency domain features, the temperature change trend features of the hot spot area, and the abnormal noise component features within the multimodal large model using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation; The calculation module 24 is used to calculate a comprehensive health index reflecting the health status of the industrial rotating machinery based on the multimodal equipment state embedding representation, and to use a sequence prediction method to deduce the future change trajectory of the comprehensive health index. The output module 25 is used to output a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery when the future change trajectory indicates that the comprehensive health index will exceed the preset warning boundary value.

[0056] Figure 2 The aforementioned industrial equipment fault prediction and diagnosis system based on a multimodal large model can perform... Figure 1 The implementation principle and technical effects of the industrial equipment fault prediction and diagnosis method based on a multimodal large model described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit performs its operations in the industrial equipment fault prediction and diagnosis system based on a multimodal large model in the above embodiments have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0057] In one possible design, Figure 2 The industrial equipment fault prediction and diagnosis system based on a multimodal large model, as shown in the embodiment, can be implemented as a computing device, such as... Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32; The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are invoked and executed by the processing component 32.

[0058] The processing component 32 is used for the above Figure 1 The embodiment describes a method for predicting and diagnosing industrial equipment faults based on a multimodal large model.

[0059] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above-described method. Alternatively, the processing component may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0060] Storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0061] Of course, computing devices may also include other components, such as input / output interfaces, display components, communication components, etc.

[0062] Input / output interfaces provide interfaces between processing components and peripheral interface modules, which can be output devices, input devices, etc.

[0063] The communication components are configured to facilitate wired or wireless communication between computing devices and other devices.

[0064] The computing device can be a physical device or an elastic computing host provided by a cloud computing platform. In this case, the computing device can refer to a cloud server, and the aforementioned processing components, storage components, etc., can be basic server resources rented or purchased from the cloud computing platform.

[0065] This application also provides a computer storage medium storing a computer program, which, when executed by a computer, can perform the above-described functions. Figure 1 The embodiment shown is a method for predicting and diagnosing industrial equipment faults based on a multimodal large model.

[0066] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for predicting and diagnosing industrial equipment faults based on a multimodal large model, characterized in that, include: Simultaneously collect multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data; The vibration waveform data, the temperature distribution image sequence, and the audio waveform data are input into a pre-trained multimodal large model. The multimodal large model is used to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot areas in the temperature distribution image sequence, and the abnormal noise components in the audio waveform data. Within the multimodal large model, the frequency domain features, the temperature change trend features of the hot spot region, and the abnormal noise component features are fused using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation; Based on the multimodal equipment state embedding representation, a comprehensive health index reflecting the health status of the industrial rotating machinery is calculated, and the future change trajectory of the comprehensive health index is deduced using a sequence prediction method. When the future trajectory indicates that the comprehensive health index will exceed the preset warning boundary value, a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery are output.

2. The method according to claim 1, characterized in that, Synchronously acquire multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequences, and audio waveform data, including: Vibration waveform data is acquired at a first sampling rate using a vibration sensor fixed to the bearing housing of the industrial rotating machinery. A temperature distribution image sequence was acquired at a second sampling rate using an optical infrared thermal imager pointed at the key heat-generating component of the industrial rotating machinery. Audio waveform data is acquired at a third sampling rate using an acoustic sensor installed at a designated location near the industrial rotating machinery. Within the same acquisition cycle, each frame of vibration waveform data, each frame of temperature distribution image sequence, and each frame of audio waveform data acquired synchronously are marked with the same time stamp. The vibration waveform data, the temperature distribution image sequence, and the audio waveform data under the same time marker are integrated into multimodal time series data.

3. The method according to claim 1, characterized in that, The vibration waveform data, the temperature distribution image sequence, and the audio waveform data are input into a pre-trained multimodal large model. The multimodal large model is then used to extract the frequency domain features of the vibration waveform data, the temperature change trend features of hotspot regions in the temperature distribution image sequence, and the abnormal noise components in the audio waveform data, including: The vibration waveform data is converted to a frequency representation, and the harmonic components and amplitude information associated with the rotation frequency of the industrial rotating machinery are identified from the frequency representation as frequency domain features. In each frame of the temperature distribution image sequence, the region whose temperature value exceeds the average temperature range of the surrounding preset area is identified as a hot spot region, and the temperature change pattern of the hot spot region in the image sequence is tracked as the temperature change trend feature of the hot spot region. Identify audio segments in the audio waveform data that differ from the reference sound pattern during stable operation of the industrial rotating machinery, and separate transient pulse signals with specific repetition patterns from the audio segments as abnormal noise components.

4. The method according to claim 1, characterized in that, Within the multimodal large model, a cross-modal attention mechanism is used to fuse the frequency domain features, the temperature change trend features of the hotspot region, and the abnormal noise component features to generate a unified multimodal device state embedding representation, including: The frequency domain features, the temperature change trend features of the hot spot area, and the abnormal noise component features are respectively mapped to the same semantic space to obtain vibration feature vector, temperature feature vector, and acoustic feature vector; The interaction weights of the vibration feature vector with respect to the temperature feature vector and the acoustic feature vector are calculated to obtain the first interaction weight. At the same time, the interaction weights of the temperature feature vector with respect to the vibration feature vector and the acoustic feature vector are calculated to obtain the second interaction weight. Finally, the interaction weights of the acoustic feature vector with respect to the vibration feature vector and the temperature feature vector are calculated to obtain the third interaction weight. The temperature feature vector is weighted using the weights of the vibration feature vector to the temperature feature vector in the first interaction weights, and the acoustic feature vector is weighted using the weights of the vibration feature vector to the acoustic feature vector in the first interaction weights. The weighted temperature feature vector and the weighted acoustic feature vector are then combined with the vibration feature vector to obtain an enhanced vibration context vector. The vibration feature vector is weighted using the weights of the temperature feature vector to the vibration feature vector in the second interaction weights, and the acoustic feature vector is weighted using the weights of the temperature feature vector to the acoustic feature vector in the second interaction weights. The weighted vibration feature vector and the weighted acoustic feature vector are then combined with the temperature feature vector to obtain an enhanced temperature context vector. The vibration feature vector is weighted using the weights of the acoustic feature vector to the vibration feature vector in the third interactive weighting, and the temperature feature vector is weighted using the weights of the acoustic feature vector to the temperature feature vector in the third interactive weighting. The weighted vibration feature vector and the weighted temperature feature vector are then combined with the acoustic feature vector to obtain an enhanced acoustic context vector. The enhanced vibration context vector, the enhanced temperature context vector, and the enhanced acoustic context vector are combined and transformed to generate a unified multimodal device state embedding representation.

5. The method according to claim 1, characterized in that, Based on the multimodal equipment state embedding representation, a comprehensive health index reflecting the health status of the industrial rotating machinery is calculated, and the future change trajectory of the comprehensive health index is extrapolated using a sequence prediction method, including: The multimodal device state embedding representation is combined and calculated based on preset weight coefficients to obtain a comprehensive health index representing the overall state of the device at the current moment. The comprehensive health index is recorded continuously at multiple points in time in chronological order, forming a historical sequence of the comprehensive health index. Using the historical sequence of the comprehensive health index as input, a pre-trained sequence analysis model is used to capture the change patterns contained in the historical sequence of the comprehensive health index. Based on the captured change patterns, the values ​​of the comprehensive health index at multiple future time points are continuously extrapolated to obtain the future change trajectory of the comprehensive health index.

6. The method according to claim 5, characterized in that, Based on the captured change patterns, the values ​​of the comprehensive health index at multiple future time points are continuously extrapolated to obtain the future change trajectory of the comprehensive health index, including: The captured change patterns are decomposed into trend components that characterize the long-term evolution direction and cyclical components that characterize periodic fluctuations. Based on the trend components, the overall upward and downward trend of the comprehensive health index over a future preset time period can be inferred; Based on the cyclical components, the fluctuation pattern of the comprehensive health index around the overall trend in the future time period can be inferred; The inferred overall trend of rise and fall is superimposed with the fluctuation pattern to calculate the first predicted comprehensive health index at the first future time point; The first predicted comprehensive health index is incorporated into the end of the historical sequence of the comprehensive health index, the change pattern is updated, and the second predicted comprehensive health index at the second future time point is calculated based on the updated change pattern. Repeat the incorporation and calculation process to obtain the predicted comprehensive health index for multiple consecutive future time points. The predicted comprehensive health indexes arranged in chronological order constitute the future change trajectory of the comprehensive health index.

7. The method according to claim 1, characterized in that, When the future trajectory indicates that the comprehensive health index will exceed a preset warning boundary value, a diagnostic conclusion and maintenance decision pointing to the potential fault type of the industrial rotating machinery are output, including: Each predicted comprehensive health index in the future change trajectory is compared with a preset warning boundary value, and the first future change trajectory point with a value lower than the preset warning boundary value is identified as a warning point; Calculate the time length from the current moment to the warning point as the warning lead time; Based on the relative significance of the vibration feature vector, temperature feature vector, and acoustic feature vector in the multimodal device state embedding representation, the main combination of influencing factors leading to the decline of the comprehensive health index is determined. The main influencing factors are combined and matched with preset fault knowledge entries to obtain successfully matched fault knowledge entries. Based on the successfully matched fault knowledge entries, the potential fault types of the industrial rotating machinery are determined, and combined with the early warning period, a diagnostic conclusion and maintenance decision containing the potential fault types and recommended maintenance time windows are generated.

8. A fault prediction and diagnosis system for industrial equipment based on a multimodal large model, characterized in that, include: The acquisition module is used to synchronously acquire multimodal time-series data of industrial rotating machinery during operation, wherein the multimodal time-series data includes at least vibration waveform data, temperature distribution image sequence and audio waveform data; The extraction module is used to input the vibration waveform data, the temperature distribution image sequence, and the audio waveform data into a pre-trained multimodal large model, and use the multimodal large model to extract the frequency domain features of the vibration waveform data, the temperature change trend features of the hot spot areas in the temperature distribution image sequence, and the abnormal noise component features in the audio waveform data; The fusion module is used to fuse the frequency domain features, the temperature change trend features of the hot spot region, and the abnormal noise component features within the multimodal large model using a cross-modal attention mechanism to generate a unified multimodal device state embedding representation; The calculation module is used to calculate a comprehensive health index reflecting the health status of the industrial rotating machinery based on the multimodal equipment state embedding representation, and to use a sequence prediction method to deduce the future change trajectory of the comprehensive health index. The output module is used to output diagnostic conclusions and maintenance decisions pointing to the potential fault types of the industrial rotating machinery when the future change trajectory indicates that the comprehensive health index will exceed the preset warning boundary value.

9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the industrial equipment fault prediction and diagnosis method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements a method for predicting and diagnosing industrial equipment faults based on a multimodal large model as described in any one of claims 1 to 7.

Images