Method and apparatus for multi-modal data alignment and augmentation in industrial large model training

Abstract

This specification provides a method and apparatus for multimodal data alignment and enhancement in industrial large-scale model training. By integrating an industrial knowledge graph with physical law constraints into a cross-modal semantic distillation architecture, heterogeneous features such as vibration, vision, and process parameters are aligned in a semantic space that conforms to engineering laws, effectively solving the problem of fragmented semantic expression and enhancing the depth of understanding of industrial processes. By introducing a dual-discriminator adversarial enhancement strategy that includes a realism discriminator and a physical consistency discriminator, and combining the constraints of an industrial anomaly pattern library, it ensures that the generated data satisfies both statistical distribution characteristics and physical reality, significantly reducing the data distortion rate. This provides high-fidelity training data for industrial large-scale models, ultimately achieving a comprehensive improvement in model convergence speed, anomaly detection accuracy, and decision credibility, meeting the application requirements of high precision and high real-time performance in industrial scenarios.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of data processing technology, and in particular to a method for multimodal data alignment and enhancement in industrial large-scale model training. Background Technology

[0002] In the field of industrial artificial intelligence, the training of large models is highly dependent on the quality and consistency of multimodal data. However, the data acquisition environment in industrial sites is complex. There are common differences in sampling frequency, transmission delay, and clock drift among heterogeneous data sources such as vibration sensors, thermal imagers, and process logs. This makes it difficult to accurately align multimodal data on the time axis. Traditional methods based on hardware synchronization or static timestamp correction show a significant decrease in alignment accuracy under highly dynamic industrial conditions. In addition, the industrial information carried by different modal data varies greatly in the feature space. Vibration signals reflect mechanical state, while thermal imaging reflects temperature distribution. Existing data-driven fusion methods lack the constraints of industrial physical mechanisms, resulting in fragmented cross-modal semantic associations and making it difficult to form a unified understanding of industrial processes. At the same time, the noise sources in industrial environments, such as electromagnetic interference and mechanical vibration, are complex. Traditional data augmentation methods often ignore physical realism when generating new samples, resulting in high distortion rates in the generated data. This fails to meet the stringent requirements for the credibility of model decisions in safety-critical scenarios. The existence of these problems seriously restricts the training efficiency, generalization ability, and actual deployment effect of large industrial models.

[0003] Therefore, a better solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for multimodal data alignment and enhancement in industrial large-scale model training. One or more embodiments of this specification also relate to a multimodal data alignment and enhancement apparatus, a computing device, a computer-readable storage medium, and a computer program for industrial large-scale model training, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for multimodal data alignment and enhancement in industrial large-scale model training is provided, comprising: Acquire raw multimodal data streams in industrial scenarios. The raw multimodal data streams include at least vibration signals, thermal imaging videos, and process parameters. By detecting physical events in the original multimodal data stream to determine dynamic anchor points, constructing a local time scale based on the dynamic anchor points, and performing spatiotemporal alignment processing on the original multimodal data stream according to the local time scale, spatiotemporal aligned multimodal data is obtained. An industrial knowledge graph containing industrial physical laws is constructed. Multimodal data that has been spatiotemporally aligned is then subjected to feature mapping and semantic alignment under the constraints of the industrial knowledge graph to obtain semantically aligned features. An adversarial augmentation generative network is constructed, which includes a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to generate augmented data, and the realism discriminator and the physical consistency discriminator jointly discriminate the augmented data to output high-quality training data that meets statistical characteristics and physical laws.

[0006] In one possible implementation, dynamic anchor points are determined by detecting physical events in the original multimodal data stream. A local time scale is constructed based on the dynamic anchor points, and the original multimodal data stream is spatiotemporally aligned according to the local time scale to obtain spatiotemporally aligned multimodal data, including: Sliding window analysis is performed on vibration signals, thermal imaging videos and process parameters. The start-up, shutdown or process switching times of equipment are identified as physical events by a multi-scale event detection function, and the times corresponding to the physical events are determined as dynamic anchor points. A local linear time scale is constructed using the time interval between two adjacent dynamic anchor points. The local linear time scale is used to map the original timestamps within the interval to the corrected timestamps. A joint loss function containing synchronization loss and physical constraint loss is constructed. The joint loss function is minimized by an optimization algorithm to adjust the parameters of the local linear time scale. The original multimodal data stream is then resampled based on the optimized local linear time scale to obtain spatiotemporally aligned multimodal data aligned on the time axis.

[0007] In one possible implementation, an industrial knowledge graph containing industrial physical laws is constructed. Multimodal data aligned in time and space is then subjected to feature mapping and semantic alignment under the constraints of this industrial knowledge graph to obtain semantically aligned features, including: By integrating expert knowledge and industry standards and specifications, an industrial knowledge graph is constructed with equipment entities, physical quantities, and process parameters as nodes and physical and process relationships as edges. Vibration features, visual features, and process features were extracted from the spatiotemporally aligned multimodal data, respectively. A semantic alignment loss function is constructed, which includes contrast loss and physical constraint loss. Using the industrial knowledge graph as a constraint, vibration features, visual features, and process features are mapped to a unified semantic space and aligned by minimizing the semantic alignment loss function, thus obtaining semantic alignment features.

[0008] In one possible implementation, an adversarial augmentation generative network is constructed, comprising a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to produce augmented data, and the realism discriminator and the physical consistency discriminator jointly discriminate the augmented data, outputting high-quality training data that satisfies statistical characteristics and physical laws, including: A conditional generative adversarial network is constructed, whose generator generates augmented data based on noise vectors and semantic alignment features. Construct a realism discriminator to determine whether the input data is real or generated data, and calculate the realism loss; A pre-trained physical law verification network is constructed as a physical consistency discriminator to determine the physical law conformity of the input data and to calculate the physical consistency loss. An industrial anomaly pattern library is constructed, and the augmented data is matched using an anomaly pattern detector, and the anomaly pattern matching loss is calculated. The generator, realism discriminator, and physical consistency discriminator are jointly trained using realism loss, physical consistency loss, and anomalous pattern matching loss until the augmented data output by the generator is validated by the dual discriminators, thus obtaining high-quality training data.

[0009] In one possible implementation, the generator, the realism discriminator, and the physical consistency discriminator are jointly trained based on authenticity loss, physical consistency loss, and anomalous pattern matching loss, including training using a curriculum learning strategy: In the initial stage, only historical data under normal operating conditions are used to train the generator and the authenticity discriminator; In the intermediate stage, mildly anomalous data and physical consistency loss are introduced to train the generator and physical consistency discriminator. In the advanced stage, heavy anomalous data and anomalous pattern matching loss are introduced to conduct joint adversarial training on the generator, authenticity discriminator and physical consistency discriminator.

[0010] In one possible implementation, after obtaining the semantic alignment features, the following is also included: Construct teacher and student models, process semantic alignment features through the teacher model, and output the first semantic distribution; The semantic alignment features are processed using a student model to output a second semantic distribution; We construct a knowledge distillation loss function that includes task loss and KL divergence loss. By minimizing the knowledge distillation loss function, we transfer the semantic knowledge of the teacher model to the student model and use the student model to output lightweight semantic alignment features.

[0011] In one possible implementation, after obtaining high-quality training data, the following is also included: Construct an end-to-end joint loss function, which includes the joint loss function in the spatiotemporal alignment process, the semantic alignment loss function in the semantic alignment process, and the generator and discriminator loss functions in the adversarial enhancement generation process; The gradient backpropagation algorithm is used to collaboratively optimize the three modules of dynamic anchor point calibration, cross-modal semantic distillation, and adversarial enhancement generation, and update the network parameters of each module.

[0012] According to a second aspect of the embodiments of this specification, a multimodal data alignment and enhancement apparatus for industrial large-scale model training is provided, comprising: The data acquisition module is configured to acquire raw multimodal data streams in industrial scenarios. The raw multimodal data streams include at least vibration signals, thermal imaging videos, and process parameters. The spatiotemporal alignment module is configured to determine dynamic anchor points by detecting physical events in the original multimodal data stream, construct a local time scale based on the dynamic anchor points, and perform spatiotemporal alignment processing on the original multimodal data stream according to the local time scale to obtain spatiotemporally aligned multimodal data. The semantic alignment module is configured to construct an industrial knowledge graph containing industrial physical laws, and to perform feature mapping and semantic alignment on the spatiotemporally aligned multimodal data under the constraints of the industrial knowledge graph to obtain semantic alignment features. The result discrimination module is configured to construct an adversarial enhancement generative network, which includes a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to generate enhanced data, and the enhanced data is jointly discriminated by the realism discriminator and the physical consistency discriminator to output high-quality training data that meets statistical characteristics and physical laws.

[0013] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the multimodal data alignment and enhancement method in the above-mentioned industrial large model training.

[0014] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal data alignment and enhancement method in the above-described industrial large model training.

[0015] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the multimodal data alignment and enhancement method in the above-described industrial large model training.

[0016] This specification provides a method and apparatus for multimodal data alignment and enhancement in industrial large-scale model training. By constructing a dynamic spatiotemporal anchor calibration mechanism driven by physical events, a local time scale is built based on physical events such as equipment start-up / shutdown or process switching in industrial settings. This achieves adaptive correction of sampling delay and temporal drift, significantly improving the spatiotemporal alignment accuracy of multimodal data under highly dynamic conditions. Through a cross-modal semantic distillation architecture that integrates industrial knowledge graphs and physical constraints, heterogeneous features such as vibration, visual characteristics, and process parameters are aligned in a semantic space that conforms to engineering principles, effectively solving the problem of fragmented semantic expression and enhancing the depth of understanding of industrial processes. By introducing a dual-discriminator adversarial enhancement strategy that includes a realism discriminator and a physical consistency discriminator, combined with constraints from an industrial anomaly pattern library, the generated data is ensured to satisfy both statistical distribution characteristics and physical reality, significantly reducing data distortion. This provides high-fidelity training data for industrial large-scale models, ultimately achieving a comprehensive improvement in model convergence speed, anomaly detection accuracy, and decision credibility, meeting the application requirements of high precision and high real-time performance in industrial scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a multimodal data alignment and enhancement method in industrial large-scale model training, provided in one embodiment of this specification. Figure 2 This is a system schematic diagram of a multimodal data alignment and enhancement method in industrial large model training provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of a multimodal data alignment and enhancement device for industrial large-scale model training provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0018] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0019] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0020] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0021] This specification provides a method for multimodal data alignment and enhancement in industrial large-scale model training. This specification also relates to a device for multimodal data alignment and enhancement in industrial large-scale model training, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0022] See Figure 1 , Figure 1 A flowchart is shown of a multimodal data alignment and enhancement method in industrial large model training according to an embodiment of this specification, which specifically includes the following steps.

[0023] Step 201: Obtain the raw multimodal data stream in the industrial scenario. The raw multimodal data stream includes at least vibration signals, thermal imaging videos, and process parameters. Step 202: Determine dynamic anchor points by detecting physical events in the original multimodal data stream, construct a local time scale based on the dynamic anchor points, and perform spatiotemporal alignment processing on the original multimodal data stream according to the local time scale to obtain spatiotemporally aligned multimodal data; Step 203: Construct an industrial knowledge graph containing industrial physical laws, and perform feature mapping and semantic alignment on the spatiotemporally aligned multimodal data under the constraints of the industrial knowledge graph to obtain semantically aligned features; Step 204: Construct an adversarial augmentation generative network. The adversarial augmentation generative network includes a generator, a realism discriminator, and a physical consistency discriminator. The semantic alignment features are input into the generator to generate augmented data. The realism discriminator and the physical consistency discriminator jointly discriminate the augmented data and output high-quality training data that meets statistical characteristics and physical laws.

[0024] The raw multimodal data stream can refer to the data collection synchronously acquired by various types of sensors in an industrial setting, such as high-frequency vibration waveform data collected by vibration sensors installed on rotating machinery, video frame sequences of equipment temperature distribution collected by infrared thermal imagers, and time series of process parameters (such as pressure, flow rate, and rotational speed) recorded by distributed control systems. Physical events can refer to changes in operating conditions with clear physical significance that occur during industrial processes, such as the start-up and shutdown of pumps, process switching moments on production lines, or sudden changes in equipment load. These events exhibit significant numerical abrupt changes in multimodal data. Dynamic anchor points can refer to spatiotemporal reference markers established based on the occurrence times of these physical events, used to correct time offsets between different sensors. Local time scales can refer to linear or nonlinear time correction functions constructed between two adjacent dynamic anchor points, enabling unified time mapping for data with different sampling rates. An industrial knowledge graph can refer to a structured knowledge base where nodes contain equipment entities (such as centrifugal compressors), physical quantities (such as vibration amplitude and temperature gradient), and process parameters (such as inlet pressure), while edges describe physical relationships between nodes (such as vibration frequency being proportional to the square of rotational speed) and process correlations (such as pressure fluctuations caused by temperature anomalies). This graph provides engineering semantic constraints for data fusion. Semantic alignment features can refer to joint representation vectors carrying physical meaning, formed by mapping vibration, visual, and process data to a unified feature space. A generator can refer to a deep learning-based generative network, such as a conditional generative adversarial network with a U-shaped network architecture, capable of synthesizing new multimodal data samples based on input semantic features and random noise. A realism discriminator can refer to a binary classification neural network used to evaluate the similarity in statistical distribution between generated and real data. A physical consistency discriminator can refer to a pre-trained physical law verification network, such as a constraint network trained based on thermodynamic equations, used to determine whether generated data violates basic engineering physics laws. High-quality training data can refer to a dataset that, after undergoing spatiotemporal calibration, semantic alignment, and physical verification, retains real industrial characteristics and conforms to physical laws, making it suitable for training large models.

[0025] The present application will be further described below through a detailed embodiment: In a condition monitoring scenario for a centrifugal compressor unit in a chemical plant, high-quality training data is required for a large-scale industrial model. This method first performs a data acquisition step. Multiple vibration acceleration sensors are deployed on the compressor unit's bearing housings, casing, and pipes, with a sampling frequency of 12,800 points per second to capture high-frequency vibration signals. Infrared thermal imagers are installed at key locations within the unit to acquire thermal imaging video at a rate of 30 frames per second, recording the temperature field distribution. Simultaneously, process parameters, including inlet pressure, outlet pressure, lubricating oil temperature, and flow rate, are read every second through the plant's distributed control system interface.

[0026] Subsequently, a spatiotemporal alignment step is performed. The method monitors the root mean square value of the vibration signal and the average temperature of a specific area in the thermal imaging video in real time. When a vibration amplitude is detected to rise sharply from the normal value to more than double within a short period of time, and the bearing housing temperature in the thermal imaging begins to show an upward trend, it is determined that a physical event of equipment loading or operating condition switching has occurred. Using the precise moment of this event as a dynamic anchor point, a time window extending several seconds forward and backward is constructed within this window. This scale dynamically calculates the relative delay between vibration data, thermal imaging frames, and process parameters by analyzing the fluctuation characteristics of each sensor data stream within this window, and resamples all data to strictly align them on the time axis, resulting in spatiotemporally aligned vibration waveform segments, temperature field image sequences, and process parameter curves.

[0027] Next, a semantic alignment step is performed. An industrial knowledge graph for rotating machinery is constructed, containing equipment entities such as compressors, bearings, and seals; physical quantity entities such as vibration frequency, harmonic amplitude, and temperature gradient; and process parameter entities such as pressure and flow rate. The graph clearly defines physical relationships such as bearing failure causing an increase in vibration sidebands at specific frequencies, and temperature increases accelerating lubricant degradation. The spatiotemporally aligned vibration data is subjected to Fourier transform to obtain spectral features, the thermal image is gridded to obtain temperature distribution features, and the process parameters are normalized. These features are projected into the semantic space defined by the graph through a contrastive learning network. This network ensures that, under the same operating conditions, vibration features, temperature features, and process parameters reflecting the same physical process are sufficiently close in space, ultimately outputting a semantically aligned feature vector that integrates multi-source information and conforms to physical common sense.

[0028] Finally, the adversarial enhancement generation step is executed. A conditional generative adversarial network (GAN) is constructed. The generator of this network receives a random noise vector and the semantically aligned feature vector output from the previous steps as input, attempting to generate a new set of multimodal data (i.e., augmented data) simulating the operating condition. The generated data is simultaneously fed into two discriminators: a realism discriminator distinguishes whether the data comes from real data collection or is fabricated by the generator; a physical consistency discriminator (a network pre-trained with thermodynamic equations and vibration theory) checks whether the generated data satisfies physical laws such as energy conservation and the relationship between vibration frequency and rotational speed. The generator iteratively optimizes until its generated data can fool both discriminators simultaneously, i.e., it looks like real data collection and conforms to physical laws. Finally, the validated high-quality training data is output for subsequent training of large-scale industrial models.

[0029] The beneficial effects of this embodiment include at least the following: by using physical event-driven dynamic anchor point calibration, the problem of unifying the time axis of multi-source sensor data under highly dynamic industrial conditions is solved, providing the model with a high-fidelity input with consistent time sequence; by fusing semantic alignment with industrial knowledge graphs, data from different modalities are associated in a physical sense, avoiding semantic fragmentation caused by purely data-driven methods; and by introducing an adversarial generation mechanism of a physical consistency discriminator, it is ensured that augmented data does not deviate from engineering reality while improving data diversity, thereby significantly improving the training efficiency and anomaly detection accuracy of subsequent large industrial models.

[0030] In one possible implementation, dynamic anchor points are determined by detecting physical events in the original multimodal data stream. A local time scale is constructed based on the dynamic anchor points, and the original multimodal data stream is spatiotemporally aligned according to the local time scale to obtain spatiotemporally aligned multimodal data, including: Sliding window analysis is performed on vibration signals, thermal imaging videos and process parameters. The start-up, shutdown or process switching times of equipment are identified as physical events by a multi-scale event detection function, and the times corresponding to the physical events are determined as dynamic anchor points. A local linear time scale is constructed using the time interval between two adjacent dynamic anchor points. The local linear time scale is used to map the original timestamps within the interval to the corrected timestamps. A joint loss function containing synchronization loss and physical constraint loss is constructed. The joint loss function is minimized by an optimization algorithm to adjust the parameters of the local linear time scale. The original multimodal data stream is then resampled based on the optimized local linear time scale to obtain spatiotemporally aligned multimodal data aligned on the time axis.

[0031] Sliding window analysis can refer to a data processing method where a fixed-length time window (e.g., 5 seconds) is set on a continuous data stream. The window slides over time, analyzing only the data within the window to capture local change characteristics. Multi-scale event detection functions can refer to a comprehensive index calculation formula that integrates the change rates of different data sources. For example, a weighted sum of the peak value of the second derivative of a vibration signal, the average temperature change rate of a thermal image, and the change rate of process parameters can be used to quantify the degree of abrupt changes in the data stream. Equipment start-up / shutdown or process switching moments can refer to critical time points where the operating state of industrial equipment undergoes a fundamental change, such as the instant a compressor starts from rest or the moment a reactor feed valve opens. These moments are often ideal benchmarks for data alignment. A local linear time scale can refer to a time correction model established between two anchor points. It assumes that the time offset changes linearly between the anchor points, interpolating the offset at the anchor points to calculate the corrected time corresponding to any original time point within the interval. Synchronization loss can refer to an error term that measures the similarity between different signals after alignment, such as the mean square error between the aligned vibration signal and the theoretical vibration signal derived from process parameters. Physical constraint loss can refer to a penalty term used to ensure that the alignment result does not violate physical laws; for example, the aligned temperature rise curve must match the theoretical curve calculated based on energy conservation. Optimization algorithms can refer to iterative calculation methods used to find the optimal scale parameters, such as the improved Levenberg-Marquardt algorithm, which minimizes synchronization error while considering physical constraints.

[0032] The present application will be further described below through a detailed embodiment, which is a refinement of the spatiotemporal alignment step in the aforementioned method: On the hot rolling production line of a steel plant, it is necessary to align the vibration, temperature, and rolling force data of the roughing mill to train a roll condition monitoring model. This method first performs sliding window analysis. An analysis window of 5 seconds with a step size of 0.5 seconds slides across synchronously acquired vibration signals, thermal imaging video, and rolling force data. Within each window, the value of a multi-scale event detection function is calculated. This function integrates the maximum value of the second derivative of the vibration signal acceleration (reflecting impact), the rate of increase of the average temperature in the roll area in the thermal imaging, and the rate of change of the rolling force. When this integrated value exceeds a preset high threshold, the system determines the center moment of the window as a physical event occurrence point. For example, at the instant the workpiece bites into the mill, the rolling force increases sharply, and the vibration impact intensifies; the function value will rise instantaneously, and this bite moment is determined as the dynamic anchor point.

[0033] Subsequently, a local linear time scale is constructed using two adjacent steel-biting event anchor points as boundaries, for example, the time interval between the nth steel-biting event and the (n+1)th steel-biting event. Within this interval, the system assumes that the offset of each sensor relative to the ideal time axis changes linearly. By analyzing the difference between the actual arrival time and the theoretical time of each data point at the anchor point, the start and end offsets of the scale can be determined, and a mapping function can be constructed to convert each original sampling time point within the interval into a unified, corrected timestamp.

[0034] Finally, a joint loss function is constructed to optimize this local scale. This function consists of two parts: a synchronization loss, which measures the temporal consistency between the time-corrected vibration signal and the rolling force signal on key features (such as impact peaks); and a physical constraint loss, for example, ensuring that the roll deformation calculated based on the rolling force has a reasonable temporal correspondence with the roll temperature field changes observed by thermal imaging. The linear parameters of the local time scale are continuously fine-tuned using an improved Levenberg-Marquardt algorithm to minimize the value of this joint loss function. After parameter optimization, the original vibration, thermal imaging, and rolling force data are resampled using cubic spline interpolation based on the final determined local time scale, ultimately generating a set of strictly time-aligned multimodal data for subsequent processing.

[0035] The beneficial effects of this embodiment include at least the following: by accurately identifying physically significant points of change in operating conditions as anchor points through a multi-scale event detection function, a reliable benchmark is provided for time correction, avoiding the drift error caused by the reliance on a fixed clock in traditional methods; by constructing a local linear time scale based on anchor points, it can flexibly adapt to the nonlinear and non-stationary time offset characteristics in industrial processes; by introducing a joint optimization function that includes physical constraint losses, it ensures that the alignment results not only meet statistical synchronicity but also fundamentally conform to the physical mechanism of industrial processes, thereby significantly improving the accuracy and robustness of spatiotemporal alignment.

[0036] In one possible implementation, an industrial knowledge graph containing industrial physical laws is constructed. Multimodal data aligned in time and space is then subjected to feature mapping and semantic alignment under the constraints of this industrial knowledge graph to obtain semantically aligned features, including: By integrating expert knowledge and industry standards and specifications, an industrial knowledge graph is constructed with equipment entities, physical quantities, and process parameters as nodes and physical and process relationships as edges. Vibration features, visual features, and process features were extracted from the spatiotemporally aligned multimodal data, respectively. A semantic alignment loss function is constructed, which includes contrast loss and physical constraint loss. Using the industrial knowledge graph as a constraint, vibration features, visual features, and process features are mapped to a unified semantic space and aligned by minimizing the semantic alignment loss function, thus obtaining semantic alignment features.

[0037] Among these, domain expert knowledge and industry standards and specifications can refer to rules of thumb accumulated within the industry and publicly available technical documents, such as the calculation formula for "bearing failure characteristic frequency" in equipment maintenance manuals, or standard documents on "mechanical vibration assessment" published by the International Organization for Standardization. Equipment entities, physical quantities, and process parameters can refer to the three core node types in a knowledge graph; for example, "centrifugal pump" is an equipment entity, "vibration intensity" is a physical quantity, and "outlet flow rate" is a process parameter. Physical relationships and process relationships can refer to the edge types connecting nodes; for example, "vibration intensity exceeding a threshold will lead to a shortened bearing life" is a physical relationship, and "increased inlet pressure will lead to an increased outlet flow rate" is a process relationship. Vibration features, visual features, and process features can refer to high-dimensional information representations extracted from raw data, such as frequency domain energy distribution features extracted from vibration signals, temperature field gradient features extracted from thermal imaging, and time-series variation features extracted from process parameters. Contrastive loss can refer to a loss function used in metric learning, aiming to bring features from different modalities within the same physical state closer together, while simultaneously widening the gap between features from different states. Physical constraint loss can refer to a penalty term used to ensure that the relationships between the mapped feature vectors conform to the physical laws defined in the knowledge graph. A unified semantic space can refer to a high-dimensional vector space in which feature vectors of different modalities will be close to each other if they describe the same physical object or process.

[0038] The present application will be further described below through a detailed embodiment, which is a refinement of the semantic alignment step in the aforementioned method: In the application of gearbox fault diagnosis for wind turbines, it is necessary to fuse vibration, temperature, and power data at the semantic level. This method first constructs an industrial knowledge graph for wind turbine gearboxes. The graph integrates parameter relationships from gearbox design manuals (e.g., the gear ratio determines the meshing frequency) and fault modes from an industry fault case library (e.g., tooth surface wear leads to sideband modulation). Nodes in the graph include equipment entities such as gearboxes, high-speed shaft bearings, and lubricating oil; physical quantities such as vibration acceleration, temperature, and instantaneous power; and process parameters such as speed and load. The edges between nodes describe physical and process relationships such as "the meshing frequency of vibration acceleration and its harmonics are directly related to speed and number of teeth" and "the lubricating oil temperature rise rate is related to load and heat dissipation conditions."

[0039] Next, feature extraction is performed on the spatiotemporally aligned data. Envelope spectrum analysis is conducted on the vibration signal to extract a vibration feature vector with gear meshing frequency and its sideband energy as key components. For thermal imaging data, the mean temperature and gradient of key observation areas of the gearbox are extracted to form a visual (temperature distribution) feature vector. For process parameters such as power and speed, their mean and rate of change within a certain time window are extracted to form a process feature vector.

[0040] Finally, semantic alignment is performed. A semantic alignment loss function is constructed, comprising contrastive loss and physical constraint loss. The contrastive loss aims to shorten the distance between vibration, temperature, and power features extracted at the same time and under the same operating condition in the feature space, as they all physically reflect the current health status of the gearbox. Simultaneously, it pushes features from different operating conditions further apart. The physical constraint loss utilizes the constructed knowledge graph for constraint. For example, if the graph indicates that "vibration energy and temperature should rise synchronously when the load increases," then for a set of vibration and temperature features describing the moment of load increase, if they are far apart in the mapped semantic space, the physical constraint loss will increase, thus penalizing this mapping that does not conform to physical common sense. By minimizing this joint loss function, a feature mapping network is finally obtained, which projects the input vibration, temperature, and power features into a unified, physically meaningful semantic space, outputting a fused semantic alignment feature vector. This vector can be directly used for subsequent fault classification or state prediction.

[0041] The beneficial effects of this embodiment include at least the following: by constructing an industrial knowledge graph containing explicit physical relationships, interpretable engineering wisdom is injected into data fusion, overcoming the limitation of pure data-driven methods that "know what but not why"; through the joint optimization of contrastive loss and physical constraint loss, not only is statistical alignment of multimodal data at the feature level achieved, but also the alignment relationship is ensured to be physically self-consistent and interpretable; the semantic alignment features obtained in the end have stronger robustness and generalization ability, and can better support intelligent decision-making in complex industrial environments.

[0042] In one possible implementation, an adversarial augmentation generative network is constructed, comprising a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to produce augmented data, and the realism discriminator and the physical consistency discriminator jointly discriminate the augmented data, outputting high-quality training data that satisfies statistical characteristics and physical laws, including: A conditional generative adversarial network is constructed, whose generator generates augmented data based on noise vectors and semantic alignment features. Construct a realism discriminator to determine whether the input data is real or generated data, and calculate the realism loss; A pre-trained physical law verification network is constructed as a physical consistency discriminator to determine the physical law conformity of the input data and to calculate the physical consistency loss. An industrial anomaly pattern library is constructed, and the augmented data is matched using an anomaly pattern detector, and the anomaly pattern matching loss is calculated. The generator, realism discriminator, and physical consistency discriminator are jointly trained using realism loss, physical consistency loss, and anomalous pattern matching loss until the augmented data output by the generator is validated by the dual discriminators, thus obtaining high-quality training data.

[0043] Conditional Generative Adversarial Networks (GANs) can refer to a generative model where the generator relies not only on random noise but also on an additional conditional variable, such as semantic features extracted from real data, to control the type or attributes of the generated data. The noise vector can refer to a low-dimensional vector randomly sampled from a standard normal distribution, introducing randomness into the generation process and ensuring the diversity of the generated data.

[0044] A realism discriminator can refer to a deep neural network classifier whose goal is to accurately distinguish between real data samples and samples fabricated by the generator. A pre-trained physics verification network can refer to a network pre-trained on a large amount of simulation data or ideal experimental data that conforms to physical laws; it can evaluate whether the input data conforms to fundamental physical laws such as the conservation of energy and the second law of thermodynamics. An industrial anomaly pattern library can refer to a database storing various typical industrial fault data features, such as typical vibration spectrum templates for faults like bearing pitting, misalignment, and imbalance.

[0045] Anomaly pattern matching loss can refer to a metric that quantifies the similarity between generated data and known fault patterns in a database. It constrains the generated data to ensure its fault characteristics match real physical faults, avoiding the generation of "pseudo-faults" that do not exist in reality. Joint training refers to the process of alternately optimizing the generator and discriminator. The generator attempts to generate more realistic data to deceive the discriminator, while the discriminator strives to improve its discrimination ability, ultimately reaching a Nash equilibrium.

[0046] The present application will be further described below through a detailed embodiment, which is a refinement of the adversarial enhancement generation step in the aforementioned method: In the condition monitoring of bearings in the main pumps of nuclear power plants, bearing fault data is extremely scarce, necessitating the generation of high-quality simulated fault data to train fault diagnosis models. This method first constructs a conditional generative adversarial network (GAN). The generator of this network adopts a U-shaped network structure, and its input consists of two parts: a random noise vector sampled from a standard normal distribution, and a semantically aligned feature vector representing the current health state of the main pump, obtained from the aforementioned semantic alignment step. The generator's goal is to output a set of simulated multimodal data, such as a vibration waveform containing fault characteristics and the corresponding temperature change curve.

[0047] Simultaneously, two discriminators are constructed. One is a authenticity discriminator, which is a convolutional neural network trained to distinguish between input data that is actually collected vibration / temperature data and data fabricated by the generator. The other is a physical consistency discriminator, which is a network pre-trained with a large amount of simulation data that conforms to the principles of heat transfer and rotor dynamics. It is specifically used to check whether the input data violates basic physical laws, such as whether there are impossible frequency components in the generated vibration waveform that are independent of the rotational speed.

[0048] In addition, an industrial anomaly pattern library has been built within the system, which stores vibration spectrum templates and temperature change characteristics of typical faults (such as wear, cracks, and burns) of nuclear main pump bearings. During training, the data generated by the generator is not only fed into two discriminators, but also compared with the templates in the anomaly pattern library to calculate the anomaly pattern matching loss, penalizing those generated data that do not match any known fault template.

[0049] The training of the entire network is a joint adversarial process. The generator continuously attempts to generate data that can simultaneously fool both the realism discriminator (making it believe it is real) and the physical consistency discriminator (making it believe it conforms to physical laws), while minimizing the matching loss with known fault templates. After multiple rounds of iterative training, the generator can consistently output high-quality data: this data is statistically indistinguishable from real-world fault data, passes the physical consistency check, and its features match pre-stored typical fault patterns. Finally, this generated high-quality augmented data is used to expand the training set to train subsequent large-scale fault diagnosis models.

[0050] The beneficial effects of this embodiment include at least the following: by using a dual discriminator architecture, while pursuing statistical authenticity of data, the physical authenticity of the data is strictly guaranteed, solving the pain point of "high distortion rate and reluctance to use" of traditional generative models in critical industrial safety scenarios; by introducing constraints from an industrial anomaly pattern library, the generator is guided to produce fault data with clear physical meaning, avoiding the generation of meaningless noise or "pseudo-faults" that are impossible to occur in reality; and the high-quality augmented data obtained can effectively improve the detection accuracy and generalization ability of downstream models under extremely imbalanced sample conditions.

[0051] In one possible implementation, the generator, the realism discriminator, and the physical consistency discriminator are jointly trained based on authenticity loss, physical consistency loss, and anomalous pattern matching loss, including training using a curriculum learning strategy: In the initial stage, only historical data under normal operating conditions are used to train the generator and the authenticity discriminator; In the intermediate stage, mildly anomalous data and physical consistency loss are introduced to train the generator and physical consistency discriminator. In the advanced stage, heavy anomalous data and anomalous pattern matching loss are introduced to conduct joint adversarial training on the generator, authenticity discriminator and physical consistency discriminator.

[0052] Among these, "course learning strategy" can refer to a machine learning training method that mimics the human learning process, starting with simple tasks and gradually transitioning to complex ones to ultimately achieve better learning outcomes and model stability. "Historical data under normal operating conditions" refers to the large amount of rich, multimodal data collected when equipment is running in a healthy state. This type of data is easy to obtain and its features are relatively stable. "Mildly anomalous data" refers to data samples from the early stages of a fault, where features are not obvious, such as data on bearings showing slight wear but not yet affecting macroscopic vibrations. This type of data falls between normal and faulty conditions, with moderate learning difficulty. "Severely anomalous data" refers to data where the fault has fully developed and its features are very significant, such as strong impact vibration data from severely damaged bearings. This type of data is often sparse but has strong features. "Joint adversarial training" refers to a complex game-like training process within a generative adversarial network framework, where the generator attempts to deceive all discriminators, while all discriminators work together to identify the forged data.

[0053] The present application will be further described below through a detailed embodiment, which is a refinement of the training strategy in the aforementioned method: In training the data augmentation network for aero-engine blade crack detection, a curriculum learning strategy was employed, training the entire model in stages, from easy to difficult. The initial stage aimed to teach the model to generate high-quality "normal" data. At this stage, all training data came from historical datasets of engines operating normally and blades in good condition. Training only activated the generator and the realism discriminator. The generator learned to generate realistic vibration and temperature data from the semantic features of normal operating conditions, while the realism discriminator focused on distinguishing between genuinely collected and generated data. After this stage of training, the generator was able to accurately simulate various data patterns of the engine under normal conditions.

[0054] After completing the initial training, the intermediate stage begins. At this point, the training set begins to include "minor anomalies"—data showing tiny cracks in the blades that haven't yet significantly impacted engine performance. Simultaneously, a physical consistency discriminator and its corresponding physical consistency loss are introduced into the training process. The generator now needs to generate data that can not only fool the realism discriminator but also ensure that the generated data (especially the local temperature and vibration response in the cracked area) conforms to the fundamental physical laws of materials mechanics and heat transfer. The physical consistency discriminator checks the generated data, for example, determining whether the local temperature rise caused by the crack is within physically permissible limits. This stage of training gives the model the initial ability to generate physically grounded early fault data.

[0055] Finally, the advanced stage begins. The training set is further supplemented with extreme case data of "severe anomalies," such as obvious blade breakage. At this point, all three losses (realism loss, physical consistency loss, and anomaly pattern matching loss) are activated, and the generator, realism discriminator, and physical consistency discriminator enter into joint adversarial training. The generator is required to generate fault data with highly significant characteristics. This data must not only appear to be a real severe fault and conform to physical laws, but also match the typical "blade breakage" templates stored in the anomaly pattern library. The realism discriminator and the physical consistency discriminator work together to put pressure on the generator, prompting it to generate the most realistic, reasonable, and typical fault-specific data.

[0056] The beneficial effects of this embodiment include at least the following: by using a course learning strategy, the complex multi-target adversarial training is decomposed into several stages, effectively avoiding the model from falling into an unstable state in the early stages of training due to the excessive complexity of the task; by starting with a large amount of readily available normal data and then gradually introducing abnormal data with increasing difficulty, the generator can gradually master the ability to generate realistic and physically consistent data; and the final trained augmented network can generate various fault samples from minor to severe in a more stable and high-quality manner, providing subsequent models with training data that is rich in information and has clear physical meaning.

[0057] In one possible implementation, after obtaining the semantic alignment features, the following is also included: Construct teacher and student models, process semantic alignment features through the teacher model, and output the first semantic distribution; The semantic alignment features are processed using a student model to output a second semantic distribution; We construct a knowledge distillation loss function that includes task loss and KL divergence loss. By minimizing the knowledge distillation loss function, we transfer the semantic knowledge of the teacher model to the student model and use the student model to output lightweight semantic alignment features.

[0058] In this context, the teacher model can refer to a complex, parameter-rich, and high-performance deep neural network, such as a large Transformer model, which can learn very rich semantic representations from data. The student model can refer to a relatively simple, parameter-free, and fast inference network, such as a multilayer perceptron or a small convolutional network.

[0059] The first semantic distribution can refer to the distribution of the final classification probability output or feature representation of the teacher model on the input data, which contains the "knowledge" learned by the teacher model from massive amounts of data. KL divergence loss can be a metric used to measure the difference between two probability distributions; in knowledge distillation, it is used to minimize the difference between the output distribution of the student model and the output distribution of the teacher model. Task loss can refer to a loss function defined for a specific downstream task (such as fault classification or remaining lifetime prediction), such as cross-entropy loss or mean squared error loss. The knowledge distillation loss function can refer to the total loss function that weights and combines the task loss and KL divergence loss. By optimizing this total loss, the student model can mimic the "thinking style" of the teacher model as much as possible while completing a specific task. Lightweight semantic alignment features can refer to feature vectors generated by the student model that retain the main knowledge of the teacher model but have lower computational cost, making them easier to deploy in real time on resource-constrained edge computing devices.

[0060] The present application will be further described below through a detailed embodiment, which is a refinement of the model lightweighting step in the aforementioned method: When deploying the trained industrial large-scale model to embedded controllers in the field, the model needs to be compressed. This method first constructs a teacher model and a student model. The teacher model is a deep neural network with hundreds of millions of parameters. It can perform complex fault mode recognition by taking the semantic alignment features with rich physical information obtained by the aforementioned method as input, and output a fine-grained fault probability distribution (first semantic distribution).

[0061] To achieve the same functionality on a controller with limited computing power, a simple multilayer perceptron is constructed as the student model. The input to the student model is also semantic alignment features, but it can only output a relatively coarse fault probability distribution (second semantic distribution).

[0062] Next, a knowledge distillation loss function is constructed. This function consists of two parts: one is the task loss, which is the cross-entropy loss between the student model's output and the true fault labels, used to ensure the accuracy of the student model in specific fault diagnosis tasks; the other is the KL divergence loss, used to calculate the difference between the probability distribution of the student model's output and the probability distribution of the teacher model's output. The goal of training the student model is to minimize this joint loss function. This means that the student model must not only learn to determine the fault category itself (minimizing the task loss), but also imitate the "teacher's" thinking as much as possible; that is, for faults that are difficult to distinguish, its judgment of the "fuzzy area" should also be consistent with the teacher model.

[0063] In this way, the "knowledge" of the teacher model is successfully distilled into the student model. Finally, during the inference phase, only this lightweight student model is used to run on the embedded controller. It takes online-acquired semantic alignment features as input and quickly outputs semantic alignment features that are highly similar to the teacher model (or directly outputs diagnostic results), thereby significantly reducing computational latency and resource consumption while ensuring diagnostic accuracy.

[0064] The beneficial effects of this embodiment include at least the following: through knowledge distillation technology, the semantic understanding ability containing physical knowledge mastered by a large, high-performance teacher model is effectively transferred to a small student model; this solves the contradiction between insufficient computing power on the edge side of the industrial site and the excellent performance of large cloud models; the resulting lightweight semantic alignment feature generation model can complete inference in milliseconds, making it possible for high-precision industrial intelligent applications to run in real time on resource-constrained embedded devices.

[0065] In one possible implementation, after obtaining high-quality training data, the following is also included: Construct an end-to-end joint loss function, which includes the joint loss function in the spatiotemporal alignment process, the semantic alignment loss function in the semantic alignment process, and the generator and discriminator loss functions in the adversarial enhancement generation process; The gradient backpropagation algorithm is used to collaboratively optimize the three modules of dynamic anchor point calibration, cross-modal semantic distillation, and adversarial enhancement generation, and update the network parameters of each module.

[0066] In this context, the end-to-end joint loss function can refer to a single objective function obtained by weighted summation of the loss functions of all modules in the entire data processing pipeline. The gradient backpropagation algorithm can refer to a chain rule that calculates the gradients of network parameters layer by layer from the output layer to the input layer based on the value of the loss function, guiding the update of network parameters.

[0067] The dynamic anchor calibration module can refer to the module described in claim 2, which is responsible for detecting physical events and correcting data timing. The cross-modal semantic distillation module can refer to the module described in claims 3 and 6, which is responsible for feature alignment and lightweight processing under physical knowledge constraints. The adversarial enhancement generation module can refer to the module described in claims 4 and 5, which is responsible for generating high-quality augmented data. Collaborative optimization can refer to the process where, instead of training each module in isolation, an end-to-end loss function is used to allow the three modules to adapt to each other and improve together under the guidance of gradients, in order to achieve global optimum.

[0068] The present application will be further described below through a detailed embodiment, which is a refinement of the overall synergistic optimization of the aforementioned method: When building a large model for fault diagnosis in chemical processes, the entire data processing pipeline comprises three core modules: a dynamic anchor calibration module, a cross-modal semantic distillation module, and an adversarial enhancement generation module. Traditionally, this involves independent training in stages, but this can lead to conflicts in optimal objectives between modules. This embodiment employs an end-to-end collaborative optimization strategy.

[0069] First, a global end-to-end joint loss function is constructed. This total loss function is not simply the sum of the losses of each module, but rather carefully designed with weights to reflect the final goal. It includes the joint loss used for time correction within the dynamic anchor calibration module (ensuring time alignment accuracy), the semantic alignment loss function of the cross-modal semantic distillation module (ensuring physical consistency of features), and the adversarial loss of the generator and discriminator in the adversarial enhancement generation module (ensuring the quality of generated data).

[0070] Subsequently, end-to-end joint training is initiated. Each training iteration comprises one complete forward propagation: a batch of raw multimodal data first enters the dynamic anchor calibration module, is corrected, and then outputs to the cross-modal semantic distillation module to extract semantic alignment features. These features are then fed into the adversarial enhancement generation module to generate enhanced data. Finally, based on the enhanced data and the intermediate results generated during the process, the value of the end-to-end joint loss function is calculated.

[0071] Then, gradient backpropagation is initiated. Using automatic differentiation, the gradient of the total loss function is calculated with respect to each network parameter in each module. This gradient information contains the "expectation" of the final objective's adjustment direction for each module's parameters. For example, if the total loss indicates poor physical consistency of the generated data, the gradient will not only update the generator but also backpropagate to the semantic distillation module, guiding it to adjust its feature mapping to make its output features more conducive to generating physically consistent data; it may even propagate to the calibration module, fine-tuning its time alignment parameters to provide more accurate input. The parameters of all modules are updated synchronously based on these gradients.

[0072] Through multiple rounds of such iterations, the entire system is collaboratively optimized. Ultimately, the dynamic anchor calibration module learns to provide the most favorable alignment data for the generation module, the semantic distillation module learns to extract the physical features that best meet the needs of generation and discrimination, and the generation module, with the cooperation of the preceding modules, can more stably generate high-quality data. The entire system reaches a globally optimal harmonious state.

[0073] The beneficial effects of this embodiment include at least the following: by constructing an end-to-end joint loss function and backpropagation algorithm, the three originally isolated links of data preprocessing, feature engineering, and data generation are connected into an organic whole, achieving global optimal system-level optimization; this collaborative optimization mechanism breaks down the barriers between modules, enabling the front-end calibration and distillation processes to adaptively adjust according to the needs of the back-end generation tasks, avoiding the trap of local optima; and the overall efficiency and the quality of the final generated data of the entire data processing system trained are significantly better than the step-by-step independent training scheme.

[0074] In one overall embodiment, see Figure 2 , Figure 2 The system architecture diagram for this application shows a unidirectional pipeline for data flow: raw multimodal data → dynamic spatiotemporal anchor calibration module → cross-modal semantic distillation module → adversarial enhancement generation module → high-quality training dataset. The modules are connected via standardized data interfaces, supporting both real-time and batch processing modes.

[0075] The dynamic spatiotemporal anchor point calibration module comprises three sub-modules: a physical event detection unit, a local time scale construction unit, and a spatiotemporal alignment optimization unit. Its workflow is as follows: Step 1: Physical event detection, as detailed below: Input: Raw multimodal data stream, including vibration signal (sampling frequency 10kHz), thermal imaging video (30fps), and process parameters (1Hz); Technical solution: A multi-scale event detection algorithm is designed to identify physical event anchor points by analyzing data abrupt change characteristics. The specific formula is as follows: Let the vibration signal be The thermal imaging temperature field is The process parameter vector is Define the event detection function:

[0076] in, These are weighting coefficients, adjusted according to the industrial scenario; when At that time, the judgment Physical events occur constantly. This is the event detection threshold.

[0077] Key technology: A sliding window mechanism is adopted (window size 3-10 seconds). The event detection results within the window are determined by majority voting to avoid single misjudgments.

[0078] Step 2: Constructing a local time scale: Technical solution: Construct a local linear time scale using detected physical events as anchor points. Let the first... The anchor point time is Its corresponding time offset is The local time scale function is then:

[0079] This function uses linear interpolation between anchor points to ensure time continuity.

[0080] Physical constraints: Time offset The maximum permissible delay constraint of industrial equipment must be met, i.e. ,in To meet the real-time requirements of industry.

[0081] Step 3: Spatiotemporal alignment optimization: Technical solution: Define a physical constraint loss function and optimize alignment parameters.

[0082] in, The synchronization loss measures the similarity of the aligned signals. For physical constraint loss, These are functions of industrial physical laws (such as thermodynamic equations and mechanical equilibrium equations).

[0083] Optimization algorithm: An improved Levenberg-Marquardt algorithm is adopted to minimize synchronization loss while ensuring physical constraints.

[0084] Technical results: In tests on 10 production lines, the module achieved a spatiotemporal alignment accuracy of 98.5%, which is 30.2% higher than the traditional method, and the alignment error was reduced from 50ms to less than 2ms.

[0085] The cross-modal semantic distillation module includes an industrial knowledge graph construction unit, a feature mapping unit, and a semantic alignment optimization unit. Its workflow is as follows: Step 1: Construction of Industrial Knowledge Graph Technical Solution: Constructing a Knowledge Graph for the Industrial Sector , where the node set Includes physical equipment (such as "centrifugal pump", "heat exchanger"), physical quantities (such as "vibration amplitude", "temperature"), and process parameters (such as "flow rate" and "pressure"); edge sets It indicates the relationship between nodes, including physical relationships (such as "vibration amplitude affects bearing life") and process relationships (such as "temperature rise leads to pressure increase").

[0086] The graph construction method integrates domain expert knowledge and industry standards and specifications (such as ISO 10814 mechanical vibration standard), automatically generates an initial graph through a rule engine, and then optimizes the graph structure through data-driven methods (such as the TransE algorithm).

[0087] Step 2: Feature Mapping and Semantic Alignment: Technical solution: Design a contrastive learning framework with mechanistic constraints to map different modal data to a unified semantic space. Let the vibration characteristics be... Visual features are The process characteristics are Define semantic alignment loss:

[0088] Among them, comparative loss , For cosine similarity, Positive samples (visual features under the same working conditions); constraint loss , For predicting physical laws, It is an actual physical quantity.

[0089] Step 3: Semantic distillation optimization: Technical solution: A knowledge distillation mechanism is employed to transfer the semantic knowledge of the large model (teacher model) to the lightweight model (student model). The distillation loss is:

[0090] in, KL divergence measures the difference in output distribution between the teacher model and the student model.

[0091] Technical results: The semantic consistency score of this module reached 0.93, which is 49.2% higher than that of traditional methods. In 500 samples evaluated by experts, the semantic matching accuracy reached 91%.

[0092] The adversarial enhancement generation module includes a generator network, a realism discriminator, and a physical consistency discriminator, which are trained using a course learning strategy.

[0093] Step 1: Generator Design: Technical solution: Design a Conditional Generative Adversarial Network (CGINet), generator Received noise vector and conditional features (From the semantic distillation module), generating enhanced data. The generator structure adopts the U-Net architecture, which includes an encoder-decoder structure, and skip connections preserve detailed information.

[0094] Step 2: Dual Discriminator Architecture: Authenticity discriminator To determine the statistical difference between generated data and real data, the loss function is:

[0095] Physical consistency discriminator The goal is to determine whether the generated data conforms to physical laws, which is achieved through a pre-trained physical law verification network. Let the physical law verification function be... ,but:

[0096] Industrial Anomaly Pattern Library Constraints: Construct a constraint library containing typical industrial anomaly patterns (such as bearing wear, pipeline leakage, and temperature anomalies). The generated data must be validated through anomaly pattern matching.

[0097] in, For the first Anomaly pattern detector, This is the abnormal intensity coefficient.

[0098] Step 3: Course Learning and Training Strategies Technical solution: Employ a progressive training strategy, with training difficulty increasing with each round. Initial stage (rounds 1-10): Training is conducted using only normal operating condition data, focusing on basic statistical characteristics; Intermediate stage (rounds 11-30): Incorporate mildly anomalous data and introduce physical constraint loss; Advanced stage (31 rounds or more): Incorporate severely anomalous data to enhance adversarial training of the dual discriminator.

[0099] Technical results: The module achieves a noise suppression rate of 85.7%, reduces the data distortion rate to 4.8%, and generates an anomaly detection F1 score of 0.92 on the power equipment fault dataset.

[0100] Furthermore, in another embodiment, it is applied to multimodal data processing of catalytic cracking units in petrochemical enterprises, as detailed below.

[0101] Data sources: 10 vibration sensors (10kHz sampling, 3-axis accelerometer), 2 infrared thermal imagers (30fps, 640×480 resolution), 1 DCS system (process parameters: temperature / pressure / flow rate, 1Hz sampling); Hardware configuration: NVIDIA A100 GPU×4, 128GB memory, 10GbE network; Software configuration: Ubuntu 22.04, PyTorch 2.0, industrial knowledge graph engine (based on Neo4j).

[0102] Implementation steps: Dynamic spatiotemporal anchor point calibration: A start-up and shutdown event of the main blower of the catalytic cracking unit was detected (vibration amplitude suddenly increased from 0.1g to 2.5g); a local time scale was constructed with the start-up and shutdown time as the anchor point, with a window size of 5 seconds; alignment parameters were optimized, and physical constraint loss weights were applied. The alignment error was reduced to 1.8ms.

[0103] Cross-modal semantic distillation: Construct a knowledge graph in the petrochemical field, containing 200+ equipment entities and 500+ physical relationships; Vibration features and thermal imaging features are aligned through comparative learning, with a semantic consistency score of 0.91; After distillation, the number of parameters in the student model is reduced by 75%, and the inference speed is increased by 3 times.

[0104] Adversarial enhancement generation: The generator generates 1000 sets of enhanced data, including normal and abnormal operating conditions; Dual discriminator verification: The accuracy of the authenticity discriminator is 95.2%, and the accuracy of the physical consistency discriminator is 93.8%; The abnormal mode library contains 5 typical failure modes, and all generated data have passed verification.

[0105] Results: Spatiotemporal alignment accuracy: 98.2% (traditional method 67.5%); Semantic consistency score: 0.91 (traditional method 0.60); Noise suppression rate: 84.3% (traditional method 59.8%). Final results: Anomaly detection accuracy increased from 78% to 94%, model training time was reduced from 12 hours to 4.5 hours, and the number of convergence rounds was reduced from 45 to 17.

[0106] Furthermore, in yet another embodiment, it is applied to power transformer fault diagnosis data enhancement, as detailed below.

[0107] Data sources: 8-channel vibration sensor (20kHz sampling), infrared thermal imager (60fps), partial discharge detector, oil chromatograph; Abnormal mode library: Includes 8 typical fault modes such as winding deformation, core loosening, and insulation aging.

[0108] Implementation steps: Dynamic spatiotemporal anchor point calibration: Transformer load switching events are identified as anchor points to construct a dynamic time scale; sliding window optimization is adopted, with a window size of 3 seconds and alignment error controlled within 2ms.

[0109] Cross-modal semantic distillation: Constructing a knowledge graph of power equipment, integrating the IEEE C57.12.00 standard; vibration-thermal imaging-partial discharge data are aligned through a physical constraint loss function.

[0110] Adversarial enhancement generation: The generator generates 500 sets of fault data, and dual discriminators verify physical consistency; Course learning strategy: Starting with slight overheating, serious fault data is gradually generated.

[0111] Effects: The generated data achieved an F1 score of 0.93 on the real test set; the model's sensitivity to early fault detection was improved by 40%; the false alarm rate was reduced from 15% to 3%, significantly improving operational security.

[0112] Corresponding to the above method embodiments, this specification also provides embodiments of a multimodal data alignment and enhancement device for industrial large-scale model training. Figure 3 This specification illustrates a schematic diagram of a multimodal data alignment and enhancement device for industrial large-scale model training, provided in one embodiment. Figure 3 As shown, the device includes: The data acquisition module 301 is configured to acquire raw multimodal data streams in industrial scenarios. The raw multimodal data streams include at least vibration signals, thermal imaging videos, and process parameters. The spatiotemporal alignment module 302 is configured to determine dynamic anchor points by detecting physical events in the original multimodal data stream, construct a local time scale based on the dynamic anchor points, and perform spatiotemporal alignment processing on the original multimodal data stream according to the local time scale to obtain spatiotemporally aligned multimodal data. The semantic alignment module 303 is configured to construct an industrial knowledge graph containing industrial physical laws, and to perform feature mapping and semantic alignment on the spatiotemporally aligned multimodal data under the constraints of the industrial knowledge graph to obtain semantic alignment features. The result discrimination module 304 is configured to construct an adversarial enhancement generation network, which includes a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to generate enhanced data, and the enhanced data is jointly discriminated by the realism discriminator and the physical consistency discriminator to output high-quality training data that meets statistical characteristics and physical laws.

[0113] In one possible implementation, dynamic anchor points are determined by detecting physical events in the original multimodal data stream. A local time scale is constructed based on the dynamic anchor points, and the original multimodal data stream is spatiotemporally aligned according to the local time scale to obtain spatiotemporally aligned multimodal data, including: Sliding window analysis is performed on vibration signals, thermal imaging videos and process parameters. The start-up, shutdown or process switching times of equipment are identified as physical events by a multi-scale event detection function, and the times corresponding to the physical events are determined as dynamic anchor points. A local linear time scale is constructed using the time interval between two adjacent dynamic anchor points. The local linear time scale is used to map the original timestamps within the interval to the corrected timestamps. A joint loss function containing synchronization loss and physical constraint loss is constructed. The joint loss function is minimized by an optimization algorithm to adjust the parameters of the local linear time scale. The original multimodal data stream is then resampled based on the optimized local linear time scale to obtain spatiotemporally aligned multimodal data aligned on the time axis.

[0114] In one possible implementation, an industrial knowledge graph containing industrial physical laws is constructed. Multimodal data aligned in time and space is then subjected to feature mapping and semantic alignment under the constraints of this industrial knowledge graph to obtain semantically aligned features, including: By integrating expert knowledge and industry standards and specifications, an industrial knowledge graph is constructed with equipment entities, physical quantities, and process parameters as nodes and physical and process relationships as edges. Vibration features, visual features, and process features were extracted from the spatiotemporally aligned multimodal data, respectively. A semantic alignment loss function is constructed, which includes contrast loss and physical constraint loss. Using the industrial knowledge graph as a constraint, vibration features, visual features, and process features are mapped to a unified semantic space and aligned by minimizing the semantic alignment loss function, thus obtaining semantic alignment features.

[0115] In one possible implementation, an adversarial augmentation generative network is constructed, comprising a generator, a realism discriminator, and a physical consistency discriminator. Semantic alignment features are input into the generator to produce augmented data, and the realism discriminator and the physical consistency discriminator jointly discriminate the augmented data, outputting high-quality training data that satisfies statistical characteristics and physical laws, including: A conditional generative adversarial network is constructed, whose generator generates augmented data based on noise vectors and semantic alignment features. Construct a realism discriminator to determine whether the input data is real or generated data, and calculate the realism loss; A pre-trained physical law verification network is constructed as a physical consistency discriminator to determine the physical law conformity of the input data and to calculate the physical consistency loss. An industrial anomaly pattern library is constructed, and the augmented data is matched using an anomaly pattern detector, and the anomaly pattern matching loss is calculated. The generator, realism discriminator, and physical consistency discriminator are jointly trained using realism loss, physical consistency loss, and anomalous pattern matching loss until the augmented data output by the generator is validated by the dual discriminators, thus obtaining high-quality training data.

[0116] In one possible implementation, the generator, the realism discriminator, and the physical consistency discriminator are jointly trained based on authenticity loss, physical consistency loss, and anomalous pattern matching loss, including training using a curriculum learning strategy: In the initial stage, only historical data under normal operating conditions are used to train the generator and the authenticity discriminator; In the intermediate stage, mildly anomalous data and physical consistency loss are introduced to train the generator and physical consistency discriminator. In the advanced stage, heavy anomalous data and anomalous pattern matching loss are introduced to conduct joint adversarial training on the generator, authenticity discriminator and physical consistency discriminator.

[0117] In one possible implementation, after obtaining the semantic alignment features, the following is also included: Construct teacher and student models, process semantic alignment features through the teacher model, and output the first semantic distribution; The semantic alignment features are processed using a student model to output a second semantic distribution; We construct a knowledge distillation loss function that includes task loss and KL divergence loss. By minimizing the knowledge distillation loss function, we transfer the semantic knowledge of the teacher model to the student model and use the student model to output lightweight semantic alignment features.

[0118] In one possible implementation, after obtaining high-quality training data, the following is also included: Construct an end-to-end joint loss function, which includes the joint loss function in the spatiotemporal alignment process, the semantic alignment loss function in the semantic alignment process, and the generator and discriminator loss functions in the adversarial enhancement generation process; The gradient backpropagation algorithm is used to collaboratively optimize the three modules of dynamic anchor point calibration, cross-modal semantic distillation, and adversarial enhancement generation, and update the network parameters of each module.

[0119] The above is a schematic scheme of a multimodal data alignment and enhancement device for industrial large-scale model training according to this embodiment. It should be noted that the technical solution of this multimodal data alignment and enhancement device for industrial large-scale model training belongs to the same concept as the technical solution of the aforementioned multimodal data alignment and enhancement method for industrial large-scale model training. Details not described in detail in the technical solution of the multimodal data alignment and enhancement device for industrial large-scale model training can be found in the description of the technical solution of the aforementioned multimodal data alignment and enhancement method for industrial large-scale model training.

[0120] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0121] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0122] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0123] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0124] The processor 420 executes computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal data alignment and enhancement method in the above-described industrial large-scale model training. The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described multimodal data alignment and enhancement method in industrial large-scale model training belong to the same concept. Details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the above-described multimodal data alignment and enhancement method in industrial large-scale model training.

[0125] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multimodal data alignment and enhancement method described above in the training of large industrial models.

[0126] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the multimodal data alignment and enhancement method in the above-mentioned industrial large-scale model training. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the multimodal data alignment and enhancement method in the above-mentioned industrial large-scale model training.

[0127] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the multimodal data alignment and enhancement method in the above-described industrial large model training.

[0128] The above is an illustrative example of a computer program in this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the multimodal data alignment and enhancement method in the above-described industrial large-scale model training. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the multimodal data alignment and enhancement method in the above-described industrial large-scale model training.

[0129] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0130] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0131] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0132] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0133] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for multi-modal data alignment and augmentation in industrial large model training, characterized in that, include: Acquire raw multimodal data streams in industrial scenarios, wherein the raw multimodal data streams include at least vibration signals, thermal imaging videos, and process parameters; By detecting physical events in the original multimodal data stream to determine dynamic anchor points, constructing a local time scale based on the dynamic anchor points, and performing spatiotemporal alignment processing on the original multimodal data stream according to the local time scale, spatiotemporal aligned multimodal data is obtained. An industrial knowledge graph containing industrial physical laws is constructed. The spatiotemporally aligned multimodal data is then subjected to feature mapping and semantic alignment under the constraints of the industrial knowledge graph to obtain semantic alignment features. An adversarial augmentation generative network is constructed, comprising a generator, a realism discriminator, and a physical consistency discriminator. The semantic alignment features are input into the generator to generate augmented data, and the realism discriminator and the physical consistency discriminator jointly discriminate the augmented data to output high-quality training data that satisfies statistical characteristics and physical laws.

2. The method of claim 1, wherein, The process involves determining dynamic anchor points by detecting physical events in the original multimodal data stream, constructing a local time scale based on the dynamic anchor points, and performing spatiotemporal alignment processing on the original multimodal data stream according to the local time scale to obtain spatiotemporally aligned multimodal data, including: A sliding window analysis is performed on the vibration signal, the thermal imaging video, and the process parameters. The equipment start-up or process switching time is identified as the physical event by a multi-scale event detection function, and the time corresponding to the physical event is determined as the dynamic anchor point. A local linear time scale is constructed using the time interval between two adjacent dynamic anchor points. The local linear time scale is used to map the original timestamps within the interval to the corrected timestamps. A joint loss function comprising synchronization loss and physical constraint loss is constructed. The joint loss function is minimized by an optimization algorithm to adjust the parameters of the local linear time scale. The original multimodal data stream is then resampled based on the optimized local linear time scale to obtain the spatiotemporally aligned multimodal data aligned on the time axis.

3. The method of claim 1, wherein, The construction of an industrial knowledge graph containing industrial physical laws involves performing feature mapping and semantic alignment on the spatiotemporally aligned multimodal data under the constraints of the industrial knowledge graph to obtain semantic alignment features, including: By integrating domain expert knowledge and industry standards and specifications, an industrial knowledge graph is constructed, with equipment entities, physical quantities, and process parameters as nodes and physical and process relationships as edges. Vibration features, visual features, and process features are extracted from the spatiotemporally aligned multimodal data, respectively. A semantic alignment loss function is constructed, which includes contrast loss and physical constraint loss. Using the industrial knowledge graph as a constraint, the vibration features, visual features, and process features are mapped to a unified semantic space and aligned by minimizing the semantic alignment loss function, thereby obtaining the semantic alignment features.

4. The method of claim 1, wherein, The adversarial enhancement generative network is constructed, comprising a generator, a realism discriminator, and a physical consistency discriminator. The semantic alignment features are input into the generator to generate enhanced data. The realism discriminator and the physical consistency discriminator jointly discriminate the enhanced data, outputting high-quality training data that satisfies statistical characteristics and physical laws, including: A conditional generative adversarial network is constructed, whose generator generates the augmented data based on the noise vector and the semantic alignment features. Construct the authenticity discriminator to determine whether the input data is real data or generated data, and calculate the authenticity loss; A pre-trained physical law verification network is constructed as the physical consistency discriminator to determine the physical law conformity of the input data and to calculate the physical consistency loss. An industrial anomaly pattern library is constructed, and the enhanced data is matched using an anomaly pattern detector, and the anomaly pattern matching loss is calculated. The generator, the realism discriminator, and the physical consistency discriminator are jointly trained based on the realism loss, the physical consistency loss, and the abnormal pattern matching loss until the augmented data output by the generator passes the verification of the dual discriminators, thus obtaining the high-quality training data.

5. The method of claim 4, wherein, The step of jointly training the generator, the realism discriminator, and the physical consistency discriminator based on the authenticity loss, the physical consistency loss, and the anomalous pattern matching loss includes training using a course learning strategy: In the initial stage, the generator and the authenticity discriminator are trained using only historical data under normal operating conditions; In the intermediate stage, mildly anomalous data and the physical consistency loss are introduced to train the generator and the physical consistency discriminator. In the advanced stage, heavily anomalous data and the anomalous pattern matching loss are introduced to conduct joint adversarial training on the generator, the authenticity discriminator, and the physical consistency discriminator.

6. The method of claim 1, wherein, After obtaining the semantic alignment features, the process also includes: Construct a teacher model and a student model, process the semantic alignment features through the teacher model, and output a first semantic distribution; The semantic alignment features are processed by the student model to output a second semantic distribution; A knowledge distillation loss function is constructed that includes task loss and KL divergence loss. By minimizing the knowledge distillation loss function, the semantic knowledge of the teacher model is transferred to the student model, and the student model is used to output lightweight semantic alignment features.

7. The method of claim 1, wherein, After obtaining high-quality training data, the process also includes: Construct an end-to-end joint loss function, which includes the joint loss function in the spatiotemporal alignment process, the semantic alignment loss function in the semantic alignment process, and the generator and discriminator loss functions in the adversarial enhancement generation process; The gradient backpropagation algorithm is used to collaboratively optimize the three modules of dynamic anchor point calibration, cross-modal semantic distillation, and adversarial enhancement generation, and update the network parameters of each module.

8. A multimodal data alignment and enhancement device for training large industrial models, characterized in that, include: The data acquisition module is configured to acquire raw multimodal data streams in an industrial scenario, wherein the raw multimodal data streams include at least vibration signals, thermal imaging videos, and process parameters; The spatiotemporal alignment module is configured to determine dynamic anchor points by detecting physical events in the original multimodal data stream, construct a local time scale based on the dynamic anchor points, and perform spatiotemporal alignment processing on the original multimodal data stream according to the local time scale to obtain spatiotemporally aligned multimodal data. The semantic alignment module is configured to construct an industrial knowledge graph containing industrial physical laws, and to perform feature mapping and semantic alignment on the spatiotemporally aligned multimodal data under the constraints of the industrial knowledge graph to obtain semantic alignment features. The result discrimination module is configured to construct an adversarial enhancement generation network, which includes a generator, a realism discriminator, and a physical consistency discriminator. The semantic alignment features are input into the generator to generate enhanced data, and the enhanced data is jointly discriminated by the realism discriminator and the physical consistency discriminator to output high-quality training data that meets statistical characteristics and physical laws.

9. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the multimodal data alignment and enhancement method in the industrial large model training according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the multimodal data alignment and augmentation method in industrial large-scale model training as described in any one of claims 1 to 7.

Images