Industrial control multi-mode generation type pre-training model construction method based on data and mechanism fusion

By constructing the timing signal-text data set and using a dual encoder for multimodal representation alignment, the limitations of traditional models in fusion mechanism knowledge are solved, and the robustness of industrial control multimodal generation pre-trained models is improved and Generalization achieves better signal generation capabilities and downstream task performance.

CN118940108AActive Publication Date: 2024-11-12TONGJI UNIV

Patent Information

Application Number
CN202410982683.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The traditional generative pre-trained model has limitations in fusion mechanism knowledge, resulting in poor model robustness and generalization, and it is difficult to effectively coordinate system reliability and high-performance collaborative control in complex industrial processes.

Method used

By collecting industrial timing signals, performing standardized preprocessing, and constructing timing signals-text-to-text data sets based on mechanism knowledge, using dual encoder to align signals with text multimodal representations, and constructing them in 'text+signal->signal' The industrial control multimodal generative pre-training model under the paradigm is pre-trained by cross-modal comparison and mask reconstruction.

Benefits of technology

The model's fusion understanding ability of data and text knowledge is improved, and the signal generation ability is enhanced, so that the pre-trained model shows excellent performance in downstream tasks, improving the robustness and generalization of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940108A_ABST
    Figure CN118940108A_ABST
Patent Text Reader

Abstract

The invention discloses a data and mechanism fused industrial control multi-modal generative pre-training model construction method, which comprises the following steps: collecting a large number of industrial time sequence signals, dividing a certain number of samples in a sliding window cutting manner, and carrying out standardized preprocessing on each sample; performing text description on each time sequence sample on the basis of mechanism knowledge so as to form a large number of time sequence signal-text pairs and construct a data set; constructing a dual encoder capable of aligning the signal and the text multi-mode representation, wherein the dual encoder comprises a signal encoder and a text encoder; and constructing an industrial control multi-modal generation type pre-training model under the normal form of text + signal-gt; and while the accuracy and comprehensiveness of text description are ensured, the text labeling cost is reduced. According to the method, explicit semantics are given to signals through expression alignment, so that multi-modal deep fusion of signal data and text knowledge in a subsequent model is supported. Therefore, the pre-training model has the potential of showing excellent performance on downstream tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of collaborative control of complex dynamic systems and multimodal generative pre-training models, and specifically relates to a method for constructing an industrial control multimodal generative pre-training model that integrates data and mechanism. Background Art

[0002] In recent years, collaborative control of complex dynamic systems has received extensive attention in the field of intelligent manufacturing. Specifically, the complex industrial process technology coupling, variable operating conditions, and time-varying equipment characteristics make it difficult to coordinate system reliability operation and high-performance collaborative control.

[0003] Deep neural networks based on generative pre-training have powerful knowledge encoding and complex task reasoning capabilities, and can adapt to various tasks in the coordinated regulation of complex dynamic systems through parameter fine-tuning. However, due to the diverse modalities and structural differences of large amounts of data and mechanism knowledge, traditional generative pre-training models only consider fitting large amounts of data and have limitations in integrating mechanism knowledge, resulting in poor model robustness and generalization. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for constructing an industrial control multimodal generative pre-training model that integrates data and mechanism, which solves the problem that traditional methods in the prior art have limitations in integrating mechanism knowledge, thereby resulting in poor model robustness and generalization.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A method for constructing a multimodal generative pre-trained model for industrial control that integrates data and mechanism includes the following steps:

[0007] Step 1: Collect a large number of industrial time series signals, divide a certain number of samples by sliding window cutting, and perform standardized preprocessing on each sample;

[0008] Step 2: Based on the mechanism knowledge, a text description is made for each time series sample, thereby forming a large number of time series signal-text pairs and constructing a data set;

[0009] Step 3: construct a dual encoder capable of aligning the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder;

[0010] Step 4: Build a multimodal generative pre-trained model for industrial control under the “text+signal->signal” paradigm.

[0011] In step 1, the sample data is preprocessed by standardization using the following formula:

[0012]

[0013] Among them, s represents a time series signal in the data set. and Var(s) are the mean and variance of the signal, respectively. a is any point of the signal, and a′ represents the standardized value of a.

[0014] Constructing the data set in step 2 includes the following steps:

[0015] Step 2.1, design a description template consisting of a template sentence and two status words, wherein the template sentence remains constant and the two status words describe specific samples;

[0016] Step 2.2, extracting the mathematical attributes of the sample to form the first state word, and extracting the physical attributes of the sample to form the second state word;

[0017] Step 2.3: Complete the text description of each signal sample based on the mechanism knowledge, and pair the sample with its corresponding text description to form a signal-text pair dataset.

[0018] In the new dataset, each signal-text pair is regarded as a sample unit and is expressed as follows:

[0019]

[0020] In the formula, is a signal-text pair dataset, (s i , t i ) is a sample pair, where s i is the signal data, t i is the text data, and N is the total number of sample pairs.

[0021] Building a dual encoder in step 3 includes the following steps:

[0022] Step 3.1: Construct a signal encoder G that can encode the time series signal and text description into a d-dimensional vector representation s With the text encoder G t , expressed by the following formula:

[0023] z i =G s (s i ), w i =G i (t i )

[0024] In the formula, is the sample alignment signal data s i The vector representation of is the text description t in the sample pair iThe vector representation of ;

[0025] Step 3.2: Train the signal encoder G in a cross-modal comparison manner s With the text encoder G t , the training process of the model is supervised by the similarity of the representation vectors of the signal and the text, so that the two encoders can complete the multimodal alignment of the signal and the text.

[0026] The process of expressing vector similarity is expressed by the following formula:

[0027]

[0028] In the formula, z i , w j are the representation vectors of any signal and any text respectively.

[0029] The following contrastive loss function is used to guide the training process of the dual encoder:

[0030]

[0031] In the formula, τ is called the temperature hyperparameter, which is preset by humans and used to adjust Sensitivity to size differences between different similarities.

[0032] The specific process of step 4 is as follows:

[0033] Step 4.1, after connecting the signal encoder and the text encoder in parallel, add a signal decoder module to realize the generation paradigm of "text + signal -> signal";

[0034] Step 4.2: Pre-training in the form of signal mask reconstruction.

[0035] The specific process of step 4.2 is as follows:

[0036] Step 4.2.1, input a sample pair into the model, first masking a part of the signal data;

[0037] Step 4.2.2: The mask signal and the text are encoded by the dual encoder to obtain their representation vectors, and the two representation vectors are concatenated and sent to the signal decoder;

[0038] Step 4.2.3, the signal encoder fuses the features of the mask signal and the semantics of the text description to decode the characteristics of the masked part of the signal, and a mask reconstruction layer completely restores the mask signal;

[0039] Step 4.2.4: Use the masked part of the signal itself as the supervision quantity and apply the following pre-training loss function to guide the model's fusion understanding of multimodal information:

[0040]

[0041] In the formula, is the masked signal, is the model’s prediction of the masked part of the signal, y msk is the true value of the masked part of the signal, and the mean square error between the two is calculated to enable the model to restore the mask signal.

[0042] The industrial control multimodal generative pre-training model includes three main parts: a signal encoder, a text encoder, and a signal decoder. The signal encoder and the text encoder are connected in parallel and then connected to the signal decoder module.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This scheme proposes a method to extract the mechanism knowledge of the signal into a text description. This method uniformly describes a time series signal with a sentence consisting of a template sentence and two types of status words, which reduces the text annotation cost while ensuring the accuracy and comprehensiveness of the text description.

[0045] 2. The similarity comparison method is used to align the signal with the corresponding text description. This method gives explicit semantics to the signal through representation alignment, thereby supporting the multimodal deep fusion of signal data and text knowledge in subsequent models.

[0046] 3. Generative pre-training is performed based on multimodal fusion to complete mask reconstruction, which not only enhances the model's integrated understanding of data and text knowledge, but also improves the model's signal generation ability, making the pre-trained model have the potential to demonstrate excellent performance in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The present invention provides a construction process for the industrial control multimodal generative pre-training model (Signal GPT) with signal data guidance and mechanism knowledge embedding.

[0048] Figure 2 A method for constructing text descriptions corresponding to signals based on mechanism knowledge. DETAILED DESCRIPTION

[0049] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.

[0050] The present invention relates to a signal data-guided and mechanism knowledge-embedded industrial control multimodal generative pre-training model, which is called Signal GPT, and includes: collecting a large number of industrial time series signals; describing each time series sample in text based on mechanism knowledge, and forming a time series signal-text pair data set; constructing a dual encoder for aligning signal and text multimodal representations; and constructing Signal GPT under the "text+signal->signal" paradigm. Compared with the prior art, this method can deeply integrate time series signal data with mechanism knowledge in text form, and support migration to various industrial control downstream tasks involving text mechanism knowledge and signal multimodal data.

[0051] A method for constructing a multimodal generative pre-trained model for industrial control that integrates data and mechanism includes the following steps:

[0052] Step 1: Collect a large number of industrial time series signals, divide a certain number of samples by sliding window cutting, and perform standardized preprocessing on each sample;

[0053] Step 2: Based on the mechanism knowledge, a text description is made for each time series sample, thereby forming a large number of time series signal-text pairs and constructing a data set;

[0054] Step 3: construct a dual encoder capable of aligning the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder;

[0055] Step 4: Build a multimodal generative pre-trained model for industrial control under the “text+signal->signal” paradigm.

[0056] Specific embodiments, such as Figure 1 , Figure 2 As shown,

[0057] A method for constructing an industrial control multimodal generative pre-training model comprises the following steps:

[0058] (1) Data preparation

[0059] The public Paderborn University bearing data set is selected as the original database. The selected data types are motor current and bearing vibration with a sampling frequency of 64kHz, and mechanical parameters (including force, speed and torque) with a sampling frequency of 4kHz. This embodiment selects the monitoring data of 9 different bearings in the original data set under the working conditions of motor speed of 1500rpm, load torque of 0.7Nm, and radial force of 1000N. Every continuous 1024 points in the original signal are taken as a sample. The specific data set details are shown in the following table:

[0060]

[0061] All samples were standardized using the following method:

[0062]

[0063] Where s represents a time series sample, and Var(s) are the mean and variance of the sample respectively, a is any point of the sample, and a′ represents the standardized value of a.

[0064] (2) Constructing temporal signal-text pairs based on mechanism knowledge

[0065] First, the same template statement "a signal of {state word 1} represents {state word 2}" is used for all time series samples. For a specific time series sample, its mathematical properties such as frequency, amplitude, shape characteristics are extracted and filled into {state word 1}, and the physical properties such as the physical quantity represented by the signal itself, the health status of the corresponding bearing, and fault characteristics are filled into {state word 2}. Figure 2 As shown in the figure, a vibration signal from KA30 has the mathematical characteristics of high frequency, high peak value and pulse shape. In the real physical world, it is collected from a bearing with multiple slight indentations on the outer ring raceway. Then this signal is described as "a high-frequency, high-peak, pulse-shaped signal, which represents the vibration of a faulty bearing with multiple slight indentations on the outer ring raceway".

[0066] After completing the text description of all time series samples, a time series-text pair dataset is constructed. This dataset covers the time series signal itself and its corresponding mechanism knowledge. Each signal-text pair is provided as a sample unit to the multimodal model training, namely:

[0067]

[0068] In the formula, For the constructed signal-text pair dataset, (s i , t i ) is a sample pair, where s i is the signal data, t i is the text data, and N is the total number of sample pairs.

[0069] (3) Constructing a dual encoder for signal and text multimodal representation alignment

[0070] Use two Transformer encoders as signal encoder G s With the text encoder G t , respectively encode the time series signal and text description into d-dimensional vector representation, namely:

[0071] z i =G s (s i ), w i=G t (t i )

[0072] In the formula, is the sample alignment signal data s i The vector representation of is the text description t in the sample pair i The vector representation of .

[0073] In training, B sample pairs are used as a training batch, and each batch is simultaneously fed into the signal encoder G s With the text encoder G t Training is performed to obtain representations of B signal data and B text descriptions. In this batch, the signals and texts that make up the sample pairs have similar representations and the signals and texts that do not belong to the same pair have dissimilar representations. The following contrast loss function is used to guide the training process of the dual encoder:

[0074]

[0075] In the formula, It is called the temperature hyperparameter, which is preset by humans to adjust The sensitivity to the size difference between different similarities. And <·, ·> represents the cosine similarity between two vectors, with any two vectors z i , w j For example, the cosine similarity between the two is calculated as:

[0076]

[0077] This multimodal representation alignment process can be seen Figure 1 Upper part. After a certain number of rounds of learning, a signal encoder and a text encoder are obtained that have the ability to map signals and texts to the same representation space.

[0078] (4) Signal GPT, which builds the “text + signal -> signal” paradigm

[0079] like Figure 1 As shown in the lower part, the trained signal encoder with representation alignment capability and the text encoder are connected in parallel, and a signal decoder F based on the Transformer structure is added after the two. d , which constitutes the entire multimodal generative pre-training model. In addition to the three main parts of the signal encoder, text encoder, and signal decoder, this model also includes a mask reconstruction layer F out .

[0080] During training, a sample pair is input to the model, and a part of the signal data is first masked (the masked part, i.e., the signal amplitude is 0). Then, the dual encoder encodes the masked signal and text to obtain its representation vector. The two representation vectors are concatenated and sent to the signal decoder. The signal encoder fuses the features of the masked signal and the semantics of the text description to decode the features of the masked part of the signal. The masked signal is further completely restored by a mask reconstruction layer. This pre-training process uses the masked part of the signal itself as a supervision quantity to guide the model's fusion understanding of multimodal information. The pre-training loss function is as follows:

[0081]

[0082] In the formula, is the masked signal, is the model’s prediction of the masked part of the signal, y msk is the true value of the masked part of the signal. The model aims to accurately restore the mask signal by minimizing the mean square error (MSA) between the two, thereby improving multimodal understanding and signal generation capabilities. It should be noted that the two encoders G are frozen at this time. s and G t Parameters of signal decoder F are updated only d and the mask reconstruction layer F out Parameters θ:

[0083]

[0084] In the formula, α is the learning rate, is the gradient of the parameter θ.

[0085] The industrial control multimodal generative pre-training model includes three main parts: a signal encoder, a text encoder, and a signal decoder. The signal encoder and the text encoder are connected in parallel and then connected to the signal decoder module.

[0086] The trained signal encoder and text encoder with representation alignment capability are connected in parallel, and a signal decoder module based on the Transformer structure is added after the two to receive the signal representation and text representation output by the dual encoders for multimodal deep fusion, thereby completing the signal decoding and generation process and realizing the "text + signal -> signal" generation paradigm.

[0087] Specifically, a sample pair is input into the model, and part of the signal data is first masked (the masked part, that is, the signal amplitude is 0). Then the dual encoder encodes the masked signal and text to obtain its representation vector. The two representation vectors are concatenated and sent to the signal decoder. The signal encoder fuses the features of the masked signal and the semantics of the text description to decode the characteristics of the masked part of the signal. The masked signal is further completely restored by a mask reconstruction layer.

[0088] Those skilled in the art should understand that they can implement variations by combining the prior art and the above embodiments. Such variations do not affect the essence of the present solution and are not described in detail here.

[0089] It should be understood that the present solution is not limited to the above-mentioned specific implementation methods, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present solution without departing from the scope of the technical solution of the present solution, or modify it into an equivalent embodiment with equivalent changes, which does not affect the essential content of the present solution. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present solution without departing from the content of the technical solution of the present solution still falls within the scope of protection of the technical solution of the present solution.

Claims

1. A method for constructing a multi-modal generative pre-training model for industrial control that integrates data and mechanism, characterized by: The steps include: Step 1: Collect a large number of industrial time series signals, divide a certain number of samples by sliding window cutting, and perform standardized preprocessing on each sample; Step 2: Based on the mechanism knowledge, a text description is made for each time series sample, thereby forming a large number of time series signal-text pairs and constructing a data set; Step 3: construct a dual encoder capable of aligning the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder; Step 4: Build an industrial control multimodal generative pre-training model under the "text+signal->signal" paradigm.

2. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 1, characterized in that: In step 1, the sample data is preprocessed by standardization using the following formula: Among them, s represents a time series signal in the data set. and Var(s) are the mean and variance of the signal, respectively. a is any point of the signal, and a′ represents the standardized value of a.

3. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 1, characterized in that: Constructing the data set in step 2 includes the following steps: Step 2.1, design a description template consisting of a template sentence and two status words, wherein the template sentence remains constant and the two status words describe specific samples; Step 2.2, extracting the mathematical attributes of the sample to form the first state word, and extracting the physical attributes of the sample to form the second state word; Step 2.3: Complete the text description of each signal sample based on the mechanism knowledge, and pair the sample with its corresponding text description to form a signal-text pair dataset.

4. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 3, characterized in that: In the new dataset, each signal-text pair is regarded as a sample unit and is expressed as follows: In the formula, is a signal-text pair dataset, (s i , t i ) is a sample pair, where s i is the signal data, t i is the text data, and N is the total number of sample pairs.

5. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 1, characterized in that: Building a dual encoder in step 3 includes the following steps: Step 3.1: Construct a signal encoder G that can encode the time series signal and text description into a d-dimensional vector representation s With the text encoder G t , expressed by the following formula: z i =G s (s i ),w i =G t (t i ) In the formula, is the sample alignment signal data s i The vector representation of is the text description of the sample pair i The vector representation of ; Step 3.2: Train the signal encoder G in a cross-modal comparison manner s With the text encoder G t , the training process of the model is supervised by the similarity of the representation vectors of the signal and the text, so that the two encoders can complete the multimodal alignment of the signal and the text.

6. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 5, characterized in that: The process of expressing vector similarity is expressed by the following formula: In the formula, z i , w j are the representation vectors of any signal and any text respectively.

7. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 6, characterized in that: The following contrastive loss function is used to guide the training process of the dual encoder: In the formula, τ is called the temperature hyperparameter, which is preset by humans and used to adjust Sensitivity to size differences between different similarities.

8. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 1, characterized in that: The specific process of step 4 is as follows: Step 4.1, after connecting the signal encoder and the text encoder in parallel, add a signal decoder module to realize the generation paradigm of "text+signal->signal"; Step 4.2: Pre-training in the form of signal mask reconstruction.

9. The method for constructing a multi-modal generative pre-trained model for industrial control integrating data and mechanism according to claim 8, characterized in that: The specific process of step 4.2 is as follows: Step 4.2.1, input a sample pair into the model, first masking a part of the signal data; Step 4.2.2: The mask signal and the text are encoded by the dual encoder to obtain their representation vectors, and the two representation vectors are concatenated and sent to the signal decoder; Step 4.2.3, the signal encoder fuses the features of the mask signal and the semantics of the text description to decode the characteristics of the masked part of the signal, and a mask reconstruction layer completely restores the mask signal; Step 4.2.4: Use the masked part of the signal itself as the supervision quantity and apply the following pre-training loss function to guide the model's fusion understanding of multimodal information: In the formula, is the masked signal, is the model’s prediction of the masked part of the signal, y msk is the true value of the masked part of the signal, and the mean square error between the two is calculated to enable the model to restore the mask signal.

10. An industrial control multimodal generative pre-training model constructed by the method according to any one of claims 1 to 9, characterized in that: The system comprises three main parts: a signal encoder, a text encoder and a signal decoder. The signal encoder and the text encoder are connected in parallel and then connected to the signal decoder module.

Citation Information

Patent Citations

  • Multi-modal emotion recognition method based on attention enhancing mechanism

    CN112489635A

  • Multi-modal feature alignment method, model training method and related device

    CN117034188A

  • Vehicle-centered multi-modal pre-training system and method based on structural information guidance

    CN117475278A

Cited By

  • Signal-to-text generation type pre-training large model

    CN120068856A

  • Intrusion detection method of food processing remote control system

    CN120342690A