Method, device and medium for fusing multi-modal data augmentation of industrial equipment mechanisms

Abstract

The application discloses a multi-modal data augmentation method fusing industrial equipment mechanism, equipment and medium, and relates to the technical field of data processing. The method comprises the following steps: collecting multi-modal data of multiple industrial data sources, and performing timestamp unification alignment processing on the multi-modal data to generate an aligned multi-modal data set; according to a preset multi-dimensional labeling system, performing label processing on the multi-modal data set to generate a labeled multi-modal data set, and constructing a mechanism knowledge base based on the mechanism knowledge of the industrial equipment; an initial condition diffusion model is constructed, and the knowledge in the mechanism knowledge base is taken as condition information and embedded into the training process of the initial condition diffusion model to generate a target loss function; based on the labeled multi-modal data set and the target loss function, the initial condition diffusion model is iteratively trained, and random noise data and target conditions are input into the trained mechanism condition model to output synthetic multi-modal data samples conforming to the industrial equipment mechanism.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, device and medium for multimodal data augmentation that integrates the mechanisms of industrial equipment. Background Technology

[0002] In the process industry, core equipment such as large compressor units, reactors, and heat exchangers generate massive amounts of highly heterogeneous multimodal data during long-term operation. This data provides the foundation for building industrial artificial intelligence models. However, the sampling frequencies of different data sources vary greatly, timestamps are not synchronized, and data formats are incompatible, resulting in severe modal fragmentation among multimodal data, making it difficult to form a unified and correlated representation. At the same time, industrial field data generally suffers from strong noise interference, information redundancy, and a scarcity of key fault samples. Traditional single-modal data augmentation methods are insufficient to support the high-quality training requirements of subsequent large multimodal models. Summary of the Invention

[0003] This application provides a method, device, and medium for multimodal data augmentation that integrates the mechanisms of industrial equipment, in order to solve the above-mentioned technical problems.

[0004] On the one hand, embodiments of this application provide a multimodal data augmentation method that integrates industrial equipment mechanisms, including: Multimodal data from multiple industrial data sources are collected, and the multimodal data is time-stamped and aligned to generate an aligned multimodal dataset. The multiple industrial data sources include time-series sensors, image acquisition devices, and text databases. According to the preset multidimensional annotation system, the multimodal dataset is labeled to generate an annotated multimodal dataset, and a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment. The multidimensional annotation system includes at least modal dimension labels, state dimension labels, and fault type dimension labels. The mechanism knowledge base includes physical equations, equipment design parameters, and fault propagation causal chains. An initial conditional diffusion model is constructed, and the knowledge in the mechanism knowledge base is used as conditional information and embedded into the training process of the initial conditional diffusion model to generate a target loss function; the target loss function includes a data reconstruction loss term and a physical consistency loss term; Based on the labeled multimodal dataset and the target loss function, the initial conditional diffusion model is iteratively trained, and random noise data and target conditions are input into the trained mechanistic conditional model to output synthetic multimodal data samples that conform to the mechanism of industrial equipment.

[0005] In one implementation of this application, the multimodal dataset is labeled according to a preset multidimensional annotation system to generate an labeled multimodal dataset, specifically including: Based on file metadata or data stream features, the modality type of each data sample in the aligned multimodal dataset is identified to generate corresponding modality dimension labels; the modality type includes time-series data, image data, text data, or structured data; Based on a preset anomaly detection rule engine, the device operation stage corresponding to each data sample is determined to generate a corresponding state dimension label; the device operation stage includes normal stage, abnormal stage, transition stage or edge stage. Based on a pre-set fault case library and expert rules, the fault mode corresponding to each data sample is identified to generate a corresponding fault type dimension label; the fault mode includes at least catalyst deactivation, particle adhesion, abnormal hydrogen concentration, excessive mechanical vibration or instrument drift. For data samples in abnormal phases or failure modes, identify and record the temporal causal relationships and propagation paths between multiple events that trigger the abnormality, and generate corresponding fault propagation causal chain annotation information; On a unified timeline, the start time, duration, and corresponding physical location of each data sample are marked to generate spatiotemporal alignment annotation information. Through the multidimensional annotation system, the modality dimension label, state dimension label, fault type dimension label, fault propagation causal chain annotation information, and spatiotemporal alignment annotation information are integrated to form an annotated multimodal dataset.

[0006] In one implementation of this application, an initial conditional diffusion model is constructed, and the knowledge in the mechanism knowledge base is embedded as conditional information into the training process of the initial conditional diffusion model to generate a target loss function, specifically including: An initial conditional diffusion model based on a denoised diffusion probability model is constructed, and the physical equations, equipment design parameters, and fault propagation causal chains in the mechanism knowledge base are vectorized and encoded to form a mechanism conditional embedding vector; the initial conditional diffusion model includes a forward diffusion process and a backward diffusion process; The embedding vector of the mechanistic conditions and the time step information are input into the noise prediction network of the initial condition diffusion model to learn the data distribution under the preset mechanistic conditions. Based on the difference between the generated samples and the real data samples in the initial conditional diffusion model, a data reconstruction loss term is constructed; The residuals between the key parameters in the generated samples and the calculated values ​​of the physical equations are calculated to determine the physical conservation constraints, and the deviation between the evolution path of the fault features in the generated samples and the defined path in the fault propagation causal chain is calculated to determine the mechanistic causal constraints. Based on the physical conservation constraints and the mechanistic causality constraints, a physical consistency loss term is constructed, and the data reconstruction loss term and the physical consistency loss term are weighted and summed to generate the target loss function.

[0007] In one implementation of this application, the initial conditional diffusion model is iteratively trained based on a labeled multimodal dataset and the target loss function, specifically including: Randomly sample real data samples from the labeled multimodal dataset for the current batch and obtain the mechanistic condition embedding vectors corresponding to the data samples in the current batch; Random noise data of the same size as the real data sample is sampled from a preset noise distribution, and noise is added to the real data sample one by one according to a preset diffusion step size to generate noisy data samples. The noise-added data sample, the current diffusion step size, and the mechanism condition embedding vector are input into the noise prediction network of the initial conditional diffusion model to predict the added noise at the current diffusion step size. Based on the predicted added noise and the actual added noise, the data reconstruction loss term is calculated, and based on the intermediate samples generated by the initial conditional diffusion model and the mechanism knowledge base, the physical consistency loss term is calculated to calculate the target loss function. Based on the target loss function, the network parameters of the noise prediction network are updated using the gradient backpropagation algorithm until the target loss function value converges, thereby obtaining the corresponding network parameters and completing the training of the mechanistic condition diffusion model.

[0008] In one implementation of this application, random noise data and target conditions are input into a trained mechanistic condition model, and synthetic multimodal data samples conforming to the mechanism of industrial equipment are output, specifically including: A pure random noise tensor of the same size as the target generated data is sampled from a standard Gaussian distribution, and the pure random noise tensor is used as the initial value for the back diffusion process. Obtain the target operating conditions or target fault modes specified by the user, and encode the target operating conditions or target fault modes into a target mechanism condition embedding vector; Starting from the preset total diffusion step size, the reverse diffusion process is executed step by step in reverse. In each reverse step size, the noise data tensor of the current step size, the current step size index, and the target mechanism condition embedding vector are input into the noise prediction network of the trained mechanism condition diffusion model to predict the noise at the current step size. Based on the predicted noise, the corresponding noise is deleted from the noise data tensor of the current step size to obtain the data tensor of the next step size. The above reverse stepwise denoising operation is repeated until the step size is zero, and the synthesized multimodal data sample is output. The modality type, data distribution and implicit causal logic of the synthesized multimodal data sample are all constrained by the target mechanism condition embedding vector.

[0009] In one implementation of this application, a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment, specifically including: Obtain the physicochemical laws that must be followed at each stage of industrial equipment, and formalize these physicochemical laws into programmable mathematical equations to construct physical equations; Collect the rated power, design speed, pipeline length, medium density, and catalyst activity curve of industrial equipment to construct equipment design parameters; Based on historical failure cases, hazard and operability analysis reports, and expert experience, directed graph relationships are extracted between failure sources, failure characteristics, failure propagation paths, and final failure consequences to construct a causal chain of failure propagation. Establish an index linking the physical equations, equipment design parameters, and fault propagation causal chains to form a mechanistic knowledge base.

[0010] In one implementation of this application, after collecting multimodal data from multiple industrial data sources and performing timestamp-aligned processing on the multimodal data to generate an aligned multimodal dataset, the method further includes: The aligned multimodal dataset is subjected to a quality pre-assessment to identify low-quality data samples below a preset quality threshold; the dimensions of the quality pre-assessment include data integrity, data timeliness, and intermodal consistency. For the low-quality data samples, missing or delayed time-series data are repaired, and noisy image data or vibration data are enhanced to generate repaired data samples. The repaired data samples are merged with the multimodal data samples that have passed the quality pre-assessment to update the aligned multimodal dataset.

[0011] In one implementation of this application, after inputting random noise data and target conditions into a trained mechanistic condition model and outputting synthetic multimodal data samples that conform to the mechanism of industrial equipment, the method further includes: Construct a cross-modal attention fusion network, and through the cross-modal attention fusion network, map synthetic multimodal data samples or labeled multimodal datasets of different modalities to a unified semantic feature space; In a unified semantic feature space, the similarity between feature vectors of data samples from different modalities is calculated, and a cross-modal retrieval task is performed based on the similarity. The synthetic multimodal data samples and the labeled multimodal dataset are mixed to form an enhanced training set for training industrial multimodal large models.

[0012] On the other hand, embodiments of this application also provide a multimodal data augmentation device that integrates industrial equipment mechanisms, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which enables the at least one processor to perform the multimodal data augmentation method for integrating industrial equipment mechanisms as described above.

[0013] On the other hand, embodiments of this application also provide a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement the multimodal data augmentation method for integrating industrial equipment mechanisms as described above.

[0014] This application provides a multimodal data augmentation method, device, and medium that integrates industrial equipment mechanisms, which has at least the following beneficial effects: By performing unified timestamp alignment on time-series sensor data, image data, and text data from multiple industrial data sources, the problem of fragmentation in time scales of multi-source heterogeneous data is solved. By introducing a multi-dimensional annotation system, data is systematically labeled from three dimensions: modality type, equipment operating status, and fault mode. Simultaneously, a mechanistic knowledge base containing physical equations, equipment design parameters, and fault propagation causal chains is constructed, thus organically integrating the deep mechanistic knowledge of industrial equipment with the data-driven annotation system. By constraining the physical consistency loss term in the constructed objective loss function, the diffusion model is forced to satisfy the physical conservation laws and fault causal propagation logic while learning the data distribution. By iteratively training the initial conditional diffusion model with labeled multimodal datasets and the objective loss function, not only can synthetic samples with a high degree of consistency with the real data distribution be generated, but also, due to the constraint of the mechanistic knowledge base in the generation process, the output synthetic samples have cross-modal physical consistency. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating the multimodal data augmentation method for integrating industrial equipment mechanisms provided in this application embodiment; Figure 2 A schematic diagram of the internal structure of a multimodal data augmentation device that integrates the mechanism of industrial equipment, provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0018] Figure 1 A flowchart illustrating the multimodal data augmentation method for integrating industrial equipment mechanisms provided in this application embodiment.

[0019] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0020] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0021] like Figure 1 As shown in the embodiments of this application, the multimodal data augmentation method integrating industrial equipment mechanisms includes: Step 101: Collect multimodal data from multiple industrial data sources and perform unified timestamp alignment on the multimodal data to generate an aligned multimodal dataset.

[0022] It should be noted that the multiple industrial data sources in this application embodiment include time-series sensors, image acquisition devices, and text databases.

[0023] In this embodiment, raw data first needs to be collected from various data sources deployed in the industrial site. For example, the first category is time-series sensors, such as vibration sensors, temperature sensors, pressure sensors, and current transformers. These sensors continuously generate numerical time-series data at fixed or variable sampling frequencies, typically collected through distributed control systems or programmable logic controllers. The second category is image acquisition devices, such as industrial cameras, infrared thermal imagers, and visible light surveillance cameras, used to acquire visual information such as equipment appearance, thermal distribution, and mechanical structure, outputting image or video stream data. The third category is text databases, such as equipment maintenance records, fault repair reports, standard operating procedure documents, hazard and operability analysis reports, and laboratory data from laboratory information management systems. This data is typically stored in the factory's information management system in unstructured or semi-structured text format.

[0024] It is understandable that, since the three types of data sources mentioned above come from different hardware systems and software platforms, their recorded timestamp bases may differ. For example, the timestamps of time-series sensor data come from the DCS system clock, the timestamps of image data come from the network video recorder clock, and the timestamps of text data come from the LIMS recording time. Directly using these raw data will lead to time misalignment during subsequent cross-modal correlation analysis. To solve this problem, this application performs unified timestamp alignment processing on all acquired multimodal data.

[0025] Specifically, a global time base is first established, for example, using a network time protocol to synchronize the local clocks of each data source with a time server, ensuring that the timestamps of all data are traced back to the same time coordinate system. Then, for each modality of data, it is discretized into time-labeled data points according to its original sampling frequency. For time-series data with higher sampling frequencies, a sliding window averaging method is used to match it with data with lower sampling frequencies in terms of time granularity; for data with lower sampling frequencies or discontinuous data, interpolation methods are used to fill in missing time points on the time axis. In this way, data from all modalities are mapped to a unified time grid, with one or more modal data samples at each timestamp, thus generating an aligned multimodal dataset.

[0026] It should be noted that after performing timestamp alignment, this embodiment further includes data quality pre-assessment and repair operations. Specifically, the aligned multimodal dataset is first subjected to quality pre-assessment. For example, the dimensions of quality pre-assessment include data integrity, data timeliness, and intermodal consistency. Data integrity is characterized by calculating the missing rate of each data sample, data timeliness is characterized by calculating the delay between the data recording time and the actual occurrence time, and intermodal consistency is characterized by calculating the statistical correlation between different modal data at the same timestamp. Data samples that fall below a preset quality threshold in any of the above dimensions are identified as low-quality data samples.

[0027] To address missing time-series data segments in low-quality data samples due to packet loss or acquisition delays, this embodiment employs a time-series interpolation-based repair model. The repair model takes the complete data segments before and after the missing segment as input, uses a Transformer-based time-series prediction algorithm, generates imputation data through model inference, and inserts the imputation data into the missing segment to complete the time-series data repair operation.

[0028] For image or vibration data containing environmental or electrical noise in low-quality data samples, this embodiment employs an enhancement model based on cross-modal denoising. The enhancement model uses complete data from other modalities within the same time window as reference conditions, such as temperature and pressure data from the same moment as input. Through model inference, it separates noise components from the image or vibration data and reconstructs clean image or vibration data based on the separated signals, thus completing the enhancement process. Finally, the newly generated data samples after repair or enhancement are merged with the high-quality data samples that passed the initial quality assessment in the original aligned multimodal dataset, forming an updated aligned multimodal dataset.

[0029] Step 102: According to the preset multidimensional annotation system, the multimodal dataset is labeled to generate an annotated multimodal dataset, and a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment.

[0030] It should be noted that the multidimensional annotation system in this application embodiment includes at least modal dimension labels, state dimension labels, and fault type dimension labels, and the mechanism knowledge base includes physical equations, equipment design parameters, and fault propagation causal chains.

[0031] In this embodiment, the aligned multimodal dataset is first systematically labeled. For example, the modality type of each data sample is automatically identified based on file metadata or data stream features. Specifically, data with regular time intervals and stored in numerical matrix form is classified as time-series data; data containing image header information and whose pixel value distribution conforms to image features is classified as image data; and data stored in natural language text form and containing semantic information is classified as text data or structured data. Through modality dimension labels, different feature extraction algorithms can be invoked for different modalities in subsequent processing.

[0032] When generating state dimension labels, a pre-defined anomaly detection rule engine is used to determine the equipment operation stage corresponding to each data sample. For example, the design operating parameter range of the equipment is first obtained, and a dynamic threshold range is determined by combining the statistical distribution of real-time data and historical normal data. When all key parameters of a data sample fall within the normal threshold range, it is marked as a normal stage; when any key parameter exceeds the threshold but does not trigger a safety interlock, it is marked as an abnormal stage; when the equipment is switching from one operating condition to another, it is marked as a transition stage; and when a data sample is near the boundary between normal and abnormal, it is marked as an edge stage.

[0033] When generating fault type dimension labels, a fault mode corresponding to each data sample is identified based on a pre-set fault case library and expert rules. The fault case library contains various typical fault modes and their corresponding feature templates, such as catalyst deactivation corresponding to an abnormal drop in reaction temperature, particle adhesion corresponding to an increase in differential pressure, abnormal hydrogen concentration corresponding to fluctuations in melt flow index, excessive mechanical vibration corresponding to an increase in amplitude at a specific frequency band, instrument drift corresponding to a slow deviation in values, insufficient cooling water flow, feed ratio imbalance, hot spot formation, gas entrainment, control valve jamming, pipeline corrosion, and accidental operation. By matching the features of the data samples with these templates, corresponding fault type dimension labels can be automatically generated.

[0034] Furthermore, this embodiment also performs causal chain annotation for data samples that have been marked as abnormal stages or have a fault type dimension. Specifically, it identifies and records the temporal causal relationships and propagation paths between multiple events that cause anomalies or faults. For example, a causal graph modeling method is used, with events as nodes and causal directions as edges, to construct a complete propagation path from the initial cause to the intermediate state and then to the final consequence. For example, from a stuck feed valve to a decrease in flow rate and an increase in temperature, and then to product quality defects and equipment shutdown. The causal chain annotation information is stored in a directed graph structure, providing an annotation basis for subsequent mechanism consistency constraints.

[0035] Furthermore, this embodiment also marks the start time, duration, and corresponding physical spatial location of each data sample on a unified timeline, generating spatiotemporal aligned annotation information. For example, for a temperature anomaly event at a certain measuring point inside the reactor, not only are the start and end times of the anomaly marked, but also the coordinates of the measuring point in three-dimensional space. Through the above multi-dimensional annotation system, modal dimension labels, state dimension labels, fault type dimension labels, fault propagation causal chain annotation information, and spatiotemporal aligned annotation information are integrated to form an annotated multimodal dataset.

[0036] In this embodiment, the application also constructs a mechanism knowledge base based on the mechanistic knowledge of industrial equipment. Specifically, firstly, the physicochemical laws followed by industrial equipment in each stage of design, operation, and maintenance are obtained, and these laws are formalized into programmable mathematical equations to construct physical equations. For example, for heat exchangers, a heat balance equation is constructed; for reactors, a material conservation equation and a reaction kinetic equation are constructed. Secondly, the inherent design parameters of the industrial equipment are collected. These parameters can be extracted from equipment ledgers, design drawings, or technical specifications provided by manufacturers. Design parameters include rated power, design speed, pipe length, medium density, and catalyst activity curves, etc. Thirdly, based on historical failure cases, hazard and operability analysis reports, and expert experience, directed graph relationships between failure sources, failure characteristics, failure propagation paths, and final failure consequences are extracted to construct a failure propagation causal chain. Finally, an association index is established between physical equations, equipment design parameters, and the failure propagation causal chain to form a mechanism knowledge base that can be quickly queried and vectorized.

[0037] Step 103: Construct an initial conditional diffusion model and embed the knowledge in the mechanism knowledge base as conditional information into the training process of the initial conditional diffusion model to generate the target loss function.

[0038] It should be noted that the target loss function in this application embodiment includes a data reconstruction loss term and a physical consistency loss term.

[0039] In this embodiment, an initial conditional diffusion model is first constructed, and a mechanistic knowledge base is used as the training process for embedding conditional information into the model. Specifically, an initial conditional diffusion model based on a denoising diffusion probability model is constructed. This model includes a forward diffusion process and a backward diffusion process. The forward diffusion process gradually adds Gaussian noise to the original data, and after multiple time steps, the original data is completely destroyed into pure random noise; the backward diffusion process learns to gradually remove noise from the pure noise and recover the original data. For example, the U-Net architecture is used as the main structure of the noise prediction network.

[0040] To incorporate mechanistic knowledge into the model, this embodiment first vectorizes the physical equations, equipment design parameters, and fault propagation causal chains in the mechanistic knowledge base, forming mechanistic condition embedding vectors. Specifically, for the physical equations, they are transformed into a computable function that takes equipment operating parameters as input and outputs theoretical expected values. This function is then parameterized and mapped to the embedding space. For the equipment design parameters, they are organized into key-value pair feature vectors and mapped to the embedding space through a fully connected layer. For the fault propagation causal chain, a graph neural network is used to encode it, obtaining embedding vectors that characterize the topological relationships and propagation weights of each node in the causal chain. Then, these three embedding vectors are fused to obtain a comprehensive mechanistic condition embedding vector.

[0041] During model training, the mechanistic conditional embedding vector and the current diffusion step size information are input into the noise prediction network of the initial conditional diffusion model. Through learning, this network can predict the distribution of noise added at the current step size. Due to the existence of the mechanistic conditional embedding vector, the noise prediction network not only relies on the statistical regularity of the data itself but is also guided by the physical mechanism, thus learning the data distribution that satisfies the mechanistic constraints.

[0042] This step also constructs a target loss function, which includes a data reconstruction loss term and a physical consistency loss term. The data reconstruction loss term is constructed based on the difference between the generated samples from the initial conditional diffusion model and the real data samples, and is calculated using mean squared error to ensure the authenticity of the generated samples at the data distribution level. The physical consistency loss term includes physical conservation constraint terms and mechanistic causality constraint terms.

[0043] Specifically, the physical conservation constraint term is determined by calculating the residual between the key parameters in the generated sample and the calculated values ​​of the physical equations. The larger the residual, the more the generated sample violates the physical conservation laws. The mechanistic causality constraint term is determined by calculating the deviation between the evolution path of the fault features in the generated sample and the defined path in the fault propagation causal chain. The larger the deviation, the less the fault propagation logic of the generated sample conforms to actual experience. Finally, the data reconstruction loss term and the physical consistency loss term are weighted and summed to generate the final target loss function. The weight coefficients of the two loss terms can be adjusted according to the actual application scenario; for example, the weight of the physical consistency loss term can be increased for safety-critical scenarios.

[0044] Step 104: Based on the labeled multimodal dataset and the target loss function, iteratively train the initial conditional diffusion model, and input random noise data and target conditions into the trained mechanism condition model to output synthetic multimodal data samples that conform to the mechanism of industrial equipment.

[0045] In this embodiment, the model training phase is performed first. Specifically, real data samples of the current batch are randomly sampled from the labeled multimodal dataset, and the mechanistic conditional embedding vectors corresponding to these samples are obtained. When obtaining the mechanistic conditional embedding vectors, based on the device identifier and operating condition in the samples, relevant physical equations, device design parameters, and fault propagation causal chain subgraphs are retrieved from the mechanistic knowledge base and vectorized according to the encoding method.

[0046] Then, random noise data of the same size as the real data sample is sampled from a preset noise distribution, and noise is successively added to the real data sample according to a preset diffusion step size schedule to generate noisy data samples. For example, the diffusion step size schedule can be a linear schedule or a cosine schedule, with the amount of noise added increasing step by step at each step.

[0047] Next, the noisy data samples, the current diffusion step size, and the mechanistic condition embedding vector are input into the noise prediction network of the initial conditional diffusion model to predict the noise added at the current diffusion step size. Based on the predicted added noise and the actual added noise, a data reconstruction loss term is calculated; simultaneously, using the intermediate samples generated by the initial conditional diffusion model at the current step size and the mechanistic knowledge base, a physical consistency loss term is calculated. The two losses are then weighted and summed to obtain the target loss function value.

[0048] Finally, based on the target loss function, the network parameters of the noisy prediction network are updated using the gradient backpropagation algorithm. The process of sampling, adding noise, prediction, loss calculation, and backpropagation is repeated until the target loss function converges, obtaining the corresponding network parameters and completing the training of the mechanistic conditional diffusion model.

[0049] In this embodiment, during the model application phase, this step performs the generation of synthetic samples. Specifically, firstly, a random noise tensor of the same size as the target generated data is sampled from a standard Gaussian distribution, and this random noise tensor is used as the initial value for the reverse diffusion process. Then, the target operating conditions or target failure modes specified by the user are obtained. For example, the user can input vibration data under conditions simulating a 5% reduction in reactor inlet temperature or a 20% decrease in catalyst activity through a natural language interface. The system converts these descriptions into structured query requests and retrieves the corresponding physical equation parameter adjustments and fault propagation causal chain subgraphs from the mechanism knowledge base, which are then encoded into a target mechanism condition embedding vector.

[0050] Next, starting from the preset total diffusion step size, the reverse diffusion process is executed stepwise. In each reverse step, the noise data tensor of the current step size, the current step size index, and the target mechanism condition embedding vector are input into the noise prediction network of the trained mechanism condition diffusion model to predict the noise at the current step size. Based on the predicted noise, the corresponding noise is subtracted from the noise data tensor of the current step size to obtain the data tensor of the next step size. This reverse stepwise denoising operation is repeated until the step size is zero. The final output data tensor is the synthesized multimodal data sample. The modality type, data distribution, and implicit causal logic of this synthesized sample are all constrained by the target mechanism condition embedding vector.

[0051] It is understandable that after obtaining synthetic multimodal data samples, this embodiment can further perform cross-modal fusion and downstream task applications. For example, firstly, a cross-modal attention fusion network based on the Transformer architecture is constructed. This network includes multiple parallel modal encoders and a shared semantic mapping layer. The time-series data, image data, and text data from the real-world synthetic multimodal data samples or labeled multimodal datasets are input into their respective modal encoders to extract the original feature vectors of each modality. Then, through cross-modal attention computation, the original feature vectors of different modalities are projected onto the same high-dimensional semantic space to generate a unified semantic embedding vector corresponding to each modality.

[0052] Within this unified semantic space, cross-modal retrieval tasks can be performed. Specifically, the system receives user-input query modality data, generates a query semantic embedding vector, calculates its similarity to the semantic embedding vectors of candidate samples, and outputs candidate samples with similarity exceeding a preset threshold as retrieval results. Furthermore, synthetic multimodal data samples and real data samples are merged according to a preset mixing ratio to form an enhanced multimodal training set, which is used for iterative training of large-scale industrial multimodal models, thereby improving the performance of downstream tasks such as fault diagnosis and risk prediction. Thus, the embodiments of this application fully realize the entire process of multimodal data augmentation integrating industrial equipment mechanisms.

[0053] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a multimodal data augmentation device that integrates industrial equipment mechanisms, the structure of which is as follows: Figure 2 As shown.

[0054] Figure 2 A schematic diagram of the internal structure of a multimodal data augmentation device that integrates the mechanisms of industrial equipment, provided in an embodiment of this application. (See diagram below.) Figure 2 As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: Multimodal data from multiple industrial data sources is collected, and the multimodal data is uniformly aligned with timestamps to generate an aligned multimodal dataset. The multiple industrial data sources include time-series sensors, image acquisition devices, and text databases. According to the pre-defined multi-dimensional annotation system, the multi-modal dataset is labeled to generate an annotated multi-modal dataset, and a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment. The multi-dimensional annotation system includes at least modal dimension labels, state dimension labels, and fault type dimension labels. The mechanism knowledge base includes physical equations, equipment design parameters, and fault propagation causal chains. An initial conditional diffusion model is constructed, and knowledge from the mechanism knowledge base is used as conditional information and embedded into the training process of the initial conditional diffusion model to generate a target loss function; the target loss function includes a data reconstruction loss term and a physical consistency loss term. Based on labeled multimodal datasets and target loss functions, the initial conditional diffusion model is iteratively trained, and random noise data and target conditions are input into the trained mechanistic condition model to output synthetic multimodal data samples that conform to the mechanism of industrial equipment.

[0055] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: Multimodal data from multiple industrial data sources is collected, and the multimodal data is uniformly aligned with timestamps to generate an aligned multimodal dataset. The multiple industrial data sources include time-series sensors, image acquisition devices, and text databases. According to the pre-defined multi-dimensional annotation system, the multi-modal dataset is labeled to generate an annotated multi-modal dataset, and a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment. The multi-dimensional annotation system includes at least modal dimension labels, state dimension labels, and fault type dimension labels. The mechanism knowledge base includes physical equations, equipment design parameters, and fault propagation causal chains. An initial conditional diffusion model is constructed, and knowledge from the mechanism knowledge base is used as conditional information and embedded into the training process of the initial conditional diffusion model to generate a target loss function; the target loss function includes a data reconstruction loss term and a physical consistency loss term. Based on labeled multimodal datasets and target loss functions, the initial conditional diffusion model is iteratively trained, and random noise data and target conditions are input into the trained mechanistic condition model to output synthetic multimodal data samples that conform to the mechanism of industrial equipment.

[0056] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0057] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal data augmentation method integrating industrial equipment mechanisms, characterized in that, The method includes: Multimodal data from multiple industrial data sources are collected, and the multimodal data is time-stamped and aligned to generate an aligned multimodal dataset. The multiple industrial data sources include time-series sensors, image acquisition devices, and text databases. According to the preset multidimensional annotation system, the multimodal dataset is labeled to generate an annotated multimodal dataset, and a mechanism knowledge base is constructed based on the mechanism knowledge of industrial equipment. The multidimensional annotation system includes at least modal dimension labels, state dimension labels, and fault type dimension labels. The mechanism knowledge base includes physical equations, equipment design parameters, and fault propagation causal chains. An initial conditional diffusion model is constructed, and the knowledge in the mechanism knowledge base is used as conditional information and embedded into the training process of the initial conditional diffusion model to generate a target loss function; the target loss function includes a data reconstruction loss term and a physical consistency loss term; Based on the labeled multimodal dataset and the target loss function, the initial conditional diffusion model is iteratively trained, and random noise data and target conditions are input into the trained mechanism condition model to output synthetic multimodal data samples that conform to the mechanism of industrial equipment. According to a pre-defined multidimensional annotation system, the multimodal dataset is labeled to generate an annotated multimodal dataset, specifically including: Based on file metadata or data stream features, the modality type of each data sample in the aligned multimodal dataset is identified to generate corresponding modality dimension labels; the modality type includes time-series data, image data, text data, or structured data; Based on a preset anomaly detection rule engine, the device operation stage corresponding to each data sample is determined to generate a corresponding state dimension label; the device operation stage includes normal stage, abnormal stage, transition stage or edge stage. Based on a pre-set fault case library and expert rules, the fault mode corresponding to each data sample is identified to generate a corresponding fault type dimension label; the fault mode includes at least catalyst deactivation, particle adhesion, abnormal hydrogen concentration, excessive mechanical vibration or instrument drift. For data samples in abnormal phases or failure modes, identify and record the temporal causal relationships and propagation paths between multiple events that trigger the abnormality, and generate corresponding fault propagation causal chain annotation information; On a unified timeline, the start time, duration, and corresponding physical location of each data sample are marked to generate spatiotemporal alignment annotation information. Through the multidimensional annotation system, the modality dimension label, state dimension label, fault type dimension label, fault propagation causal chain annotation information, and spatiotemporal alignment annotation information are integrated to form an annotated multimodal dataset. An initial conditional diffusion model is constructed, and the knowledge in the aforementioned mechanism knowledge base is used as conditional information and embedded into the training process of the initial conditional diffusion model to generate the target loss function, specifically including: An initial conditional diffusion model based on a denoised diffusion probability model is constructed, and the physical equations, equipment design parameters, and fault propagation causal chains in the mechanism knowledge base are vectorized and encoded to form a mechanism conditional embedding vector; the initial conditional diffusion model includes a forward diffusion process and a backward diffusion process; The embedding vector of the mechanistic conditions and the time step information are input into the noise prediction network of the initial condition diffusion model to learn the data distribution under the preset mechanistic conditions. Based on the difference between the generated samples and the real data samples in the initial conditional diffusion model, a data reconstruction loss term is constructed; The residuals between the key parameters in the generated samples and the calculated values ​​of the physical equations are calculated to determine the physical conservation constraints, and the deviation between the evolution path of the fault features in the generated samples and the defined path in the fault propagation causal chain is calculated to determine the mechanistic causal constraints. Based on the physical conservation constraints and the mechanistic causality constraints, a physical consistency loss term is constructed, and the data reconstruction loss term and the physical consistency loss term are weighted and summed to generate the target loss function.

2. The multimodal data augmentation method for integrating industrial equipment mechanisms according to claim 1, characterized in that, Based on the labeled multimodal dataset and the target loss function, the initial conditional diffusion model is iteratively trained, specifically including: Randomly sample real data samples from the labeled multimodal dataset for the current batch and obtain the mechanistic condition embedding vectors corresponding to the data samples in the current batch; Random noise data of the same size as the real data sample is sampled from a preset noise distribution, and noise is added to the real data sample one by one according to a preset diffusion step size to generate noisy data samples. The noise-added data sample, the current diffusion step size, and the mechanism condition embedding vector are input into the noise prediction network of the initial conditional diffusion model to predict the added noise at the current diffusion step size. Based on the predicted added noise and the actual added noise, the data reconstruction loss term is calculated, and based on the intermediate samples generated by the initial conditional diffusion model and the mechanism knowledge base, the physical consistency loss term is calculated to calculate the target loss function. Based on the target loss function, the network parameters of the noise prediction network are updated using the gradient backpropagation algorithm until the target loss function value converges, thereby obtaining the corresponding network parameters and completing the training of the mechanistic condition diffusion model.

3. The multimodal data augmentation method for integrating industrial equipment mechanisms according to claim 1, characterized in that, Random noise data and target conditions are input into a pre-trained mechanistic condition model, which outputs synthetic multimodal data samples that conform to the mechanism of industrial equipment, specifically including: A pure random noise tensor of the same size as the target generated data is sampled from a standard Gaussian distribution, and the pure random noise tensor is used as the initial value for the back diffusion process. Obtain the target operating conditions or target fault modes specified by the user, and encode the target operating conditions or target fault modes into a target mechanism condition embedding vector; Starting from the preset total diffusion step size, the reverse diffusion process is executed step by step in reverse. In each reverse step size, the noise data tensor of the current step size, the current step size index, and the target mechanism condition embedding vector are input into the noise prediction network of the trained mechanism condition diffusion model to predict the noise at the current step size. Based on the predicted noise, the corresponding noise is deleted from the noise data tensor of the current step size to obtain the data tensor of the next step size. The above reverse stepwise denoising operation is repeated until the step size is zero, and the synthesized multimodal data sample is output. The modality type, data distribution and implicit causal logic of the synthesized multimodal data sample are all constrained by the target mechanism condition embedding vector.

4. The multimodal data augmentation method for integrating industrial equipment mechanisms according to claim 1, characterized in that, Based on the mechanistic knowledge of industrial equipment, a mechanistic knowledge base is constructed, specifically including: Obtain the physicochemical laws that must be followed at each stage of industrial equipment, and formalize these physicochemical laws into programmable mathematical equations to construct physical equations; Collect the rated power, design speed, pipeline length, medium density, and catalyst activity curve of industrial equipment to construct equipment design parameters; Based on historical failure cases, hazard and operability analysis reports, and expert experience, directed graph relationships are extracted between failure sources, failure characteristics, failure propagation paths, and final failure consequences to construct a causal chain of failure propagation. Establish an index linking the physical equations, equipment design parameters, and fault propagation causal chains to form a mechanistic knowledge base.

5. The multimodal data augmentation method for integrating industrial equipment mechanisms according to claim 1, characterized in that, After collecting multimodal data from multiple industrial data sources and performing timestamp-aligned processing on the multimodal data to generate an aligned multimodal dataset, the method further includes: The aligned multimodal dataset is subjected to a quality pre-assessment to identify low-quality data samples below a preset quality threshold; the dimensions of the quality pre-assessment include data integrity, data timeliness, and intermodal consistency. For the low-quality data samples, missing or delayed time-series data are repaired, and noisy image data or vibration data are enhanced to generate repaired data samples. The repaired data samples are merged with the multimodal data samples that have passed the quality pre-assessment to update the aligned multimodal dataset.

6. The multimodal data augmentation method for integrating industrial equipment mechanisms according to claim 1, characterized in that, After inputting random noise data and target conditions into a trained mechanistic condition model and outputting synthetic multimodal data samples that conform to the mechanism of industrial equipment, the method further includes: Construct a cross-modal attention fusion network, and through the cross-modal attention fusion network, map synthetic multimodal data samples or labeled multimodal datasets of different modalities to a unified semantic feature space; In a unified semantic feature space, the similarity between feature vectors of data samples from different modalities is calculated, and a cross-modal retrieval task is performed based on the similarity. The synthetic multimodal data samples and the labeled multimodal dataset are mixed to form an enhanced training set for training industrial multimodal large models.

7. A multimodal data augmentation device integrating industrial equipment mechanisms, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the multimodal data augmentation method for integrating industrial equipment mechanisms as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the multimodal data augmentation method for integrating industrial equipment mechanisms as described in any one of claims 1-6 is implemented.

Images