Data and mechanism fusion industrial control multi-modal generative pre-training model construction method

Abstract

The application discloses a kind of data and mechanism fusion's industrial control multi-modal generative pre-training model construction method, a large number of industrial time series are collected, a certain number of samples are divided in the way of sliding window cutting, each sample is standardized and preprocessed;According to the mechanism knowledge, each time series sample is described in text, and then a large number of time series-text pairs are formed and a dataset is constructed;A double encoder that can align signal and text multi-modal representation is constructed, the double encoder includes signal encoder and text encoder;Industrial control multi-modal generative pre-training model under the paradigm of "text+signal->signal" is constructed.In the meantime of ensuring the accuracy and comprehensiveness of text description, reduce the cost of text labeling.The method gives signal explicit semantics through representation alignment, thereby supporting multi-modal deep integration of signal data and text knowledge in subsequent models.The pre-training model has the potential to exhibit excellent performance on downstream tasks.

Description

Technical Field

[0001] This invention belongs to the technical field of collaborative control of complex dynamic systems and multimodal generative pre-trained models, specifically involving a method for constructing an industrial control multimodal generative pre-trained model that integrates data and mechanisms. Background Technology

[0002] In recent years, collaborative control of complex dynamic systems has received widespread attention in the field of intelligent manufacturing. Specifically, the coupled processes, variable operating conditions, and time-varying characteristics of complex industrial processes make it difficult to coordinate reliable system operation with high-performance collaborative control.

[0003] Generative pre-trained deep neural networks possess powerful knowledge encoding and complex task reasoning capabilities, and can adapt to various tasks in the collaborative control of complex dynamic systems through parameter fine-tuning. However, due to the modal diversity and structural differences of large amounts of data and mechanistic knowledge, traditional generative pre-trained models only consider fitting large amounts of data, which limits their ability to integrate mechanistic knowledge, resulting in poor model robustness and generalization. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for constructing a multimodal generative pre-trained model for industrial control that integrates data and mechanisms. This method solves the problem that traditional methods in the prior art have limitations in integrating mechanism knowledge, resulting in poor robustness and generalization of the model.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A method for constructing a multimodal generative pre-trained model for industrial control systems that integrates data and mechanisms includes the following steps:

[0007] Step 1: Collect a large amount of industrial time-series signals, divide them into a certain number of samples by sliding window segmentation, and perform standardization preprocessing on each sample;

[0008] Step 2: Based on mechanistic knowledge, text descriptions are performed on each time series sample to form a large number of time series signal-text pairs and construct a dataset.

[0009] Step 3: Construct a dual encoder that aligns the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder;

[0010] Step 4: Construct an industrial control multimodal generative pre-trained model under the "text + signal -> signal" paradigm.

[0011] In step 1, the sample data is standardized and preprocessed using the following formula:

[0012]

[0013] Where s represents a segment of time-series signal in the dataset, Var(s) and Var(s) are the mean and variance of the signal segment, respectively. a is any point in the signal, and a′ represents the standardized value of a.

[0014] Step 2, which involves building the dataset, includes the following steps:

[0015] Step 2.1: Design a description template consisting of a template statement and two state words. The template statement remains constant, and the two state words describe the specific sample.

[0016] Step 2.2: Extract the mathematical attributes of the sample to form the first state term, and extract the physical attributes of the sample to form the second state term;

[0017] Step 2.3: Complete the text description of each signal sample based on the mechanistic knowledge, and combine the sample with its corresponding text description into pairs to form a signal-text pair dataset.

[0018] In the new dataset, each signal-text pair is treated as a sample unit, represented by the following formula:

[0019]

[0020] In the formula, For signal-text pair datasets, (s i , t i ) represents a sample pair, where s i For signal data, t i The data is text, and N is the total number of sample pairs.

[0021] Step 3, constructing the dual encoder, includes the following steps:

[0022] Step 3.1: Construct a signal encoder G that can encode time-series signals and text descriptions into d-dimensional vector representations, respectively. s With text encoder G t It is expressed by the following formula:

[0023] z i =G s (s i ), w i =G i (t i )

[0024] In the formula, For the sample pair signal data s i The vector representation of , For the sample pair Chinese text description t iVector representation of;

[0025] Step 3.2: Train the signal encoder G using a cross-modal contrastive approach. s With text encoder G t The training process of the model is supervised by the similarity of the representation vectors of the signal and the text, enabling the two encoders to complete the multimodal alignment of the signal and the text.

[0026] The process of representing vector similarity is expressed by the following formula:

[0027]

[0028] In the formula, z i w j Let be the representation vectors for any signal and any text, respectively.

[0029] The following contrastive loss function guides the training process of the dual encoder:

[0030]

[0031] In the formula, τ is called the temperature hyperparameter, which is preset by humans and used for regulation. Sensitivity to size differences between different similarities.

[0032] The specific process of step 4 is as follows:

[0033] Step 4.1: After connecting the signal encoder and text encoder in parallel, add a signal decoder module to realize the generation paradigm of "text + signal -> signal".

[0034] Step 4.2: Perform pre-training in the form of signal mask reconstruction.

[0035] The specific process of step 4.2 is as follows:

[0036] Step 4.2.1: Input a sample pair into the model to initially mask a portion of the signal data;

[0037] Step 4.2.2: The mask signal and text are encoded by the dual encoder to obtain their representation vectors. The two representation vectors are concatenated and then sent to the signal decoder.

[0038] Step 4.2.3: The signal encoder fuses the features of the masked signal and the semantics of the text description to decode the characteristics of the masked part of the signal, and then a mask reconstruction layer completely restores the masked signal.

[0039] Step 4.2.4: Using the masked portion of the signal itself as a supervisory variable, apply the following pre-training loss function to guide the model's understanding of the fusion of multimodal information:

[0040]

[0041] In the formula, This is a masked signal. y represents the model's prediction of the masked portion of the signal. msk The true value of the masked part of the signal is given, and the mean square error between the two is calculated to restore the masked signal in the model.

[0042] The industrial control multimodal generative pre-trained model includes three main parts: a signal encoder, a text encoder, and a signal decoder. The signal encoder and the text encoder are connected in parallel and then connected to the signal decoder module.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. This scheme proposes a method to extract the mechanistic knowledge of signals into text descriptions. This method uniformly describes a time-series signal using a template statement and two types of state words, which reduces the cost of text annotation while ensuring the accuracy and comprehensiveness of the text description.

[0045] 2. A similarity comparison method is used to align the signal with the corresponding text description. This method endows the signal with explicit semantics through representation alignment, thereby supporting the multimodal deep fusion of signal data and text knowledge in subsequent models.

[0046] 3. Generative pre-training is performed by using a multimodal fusion-based mask reconstruction method, which enhances the model's understanding of data and textual knowledge fusion and improves the model's signal generation ability, giving the pre-trained model the potential to demonstrate excellent performance in downstream tasks. Attached Figure Description

[0047] Figure 1 This describes the construction process of the industrial control multimodal generative pre-trained model (Signal GPT) that incorporates signal data guidance and mechanistic knowledge embedding, as described in this invention.

[0048] Figure 2 This is a method for constructing textual descriptions of signals based on mechanistic knowledge. Detailed Implementation

[0049] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0050] This invention relates to a signal data-guided, mechanism-knowledge-embedded multimodal generative pre-trained model for industrial control, called Signal GPT. The model includes: collecting a large number of industrial time-series signals; describing each time-series sample in text form based on mechanism knowledge, and constructing a time-series signal-text pair dataset; building a dual encoder aligned with the signal and text multimodal representations; and constructing Signal GPT under the "text + signal -> signal" paradigm. Compared with existing technologies, this method can deeply integrate time-series signal data with text-based mechanism knowledge, supporting transfer to various downstream industrial control tasks involving text-based mechanism knowledge and signal multimodal data.

[0051] A method for constructing a multimodal generative pre-trained model for industrial control systems that integrates data and mechanisms includes the following steps:

[0052] Step 1: Collect a large amount of industrial time-series signals, divide them into a certain number of samples by sliding window segmentation, and perform standardization preprocessing on each sample;

[0053] Step 2: Based on mechanistic knowledge, text descriptions are performed on each time series sample to form a large number of time series signal-text pairs and construct a dataset.

[0054] Step 3: Construct a dual encoder that aligns the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder;

[0055] Step 4: Construct an industrial control multimodal generative pre-trained model under the "text + signal -> signal" paradigm.

[0056] Specific embodiments, such as Figure 1 , Figure 2 As shown,

[0057] A method for constructing a multimodal generative pre-trained model for industrial control systems includes the following steps:

[0058] (1) Data preparation

[0059] The publicly available Paderborn University bearing dataset was selected as the original database. The selected data types were motor current and bearing vibration sampled at 64 kHz, and mechanical parameters (including force, speed, and torque) sampled at 4 kHz. This embodiment selected monitoring data from the original dataset for nine different bearings operating under conditions of a motor speed of 1500 rpm, a load torque of 0.7 Nm, and a radial force of 1000 N. Each consecutive 1024 points in the original signal was considered a sample. Details of the dataset are shown in the table below.

[0060]

[0061] All samples were standardized using the following method:

[0062]

[0063] Where s represents a certain time series sample. Var(s) and Var(s) are the mean and variance of the sample, respectively. a is any point in the sample, and a′ represents the standardized value of a.

[0064] (2) Constructing time-series signal-text pairs based on mechanistic knowledge

[0065] First, the same template statement is used for all time series samples: "A signal of {state word 1} represents {state word 2}". For a specific time series sample, its mathematical attributes such as frequency, amplitude, and shape characteristics are extracted and filled into {state word 1}, while its physical attributes, such as the physical quantity represented by the signal, the health status of the corresponding bearing, and fault characteristics, are filled into {state word 2}. For example... Figure 2 As shown, a vibration signal from KA30 has the mathematical characteristics of high frequency, high peak value, and pulse shape. In the real physical world, it is collected from a bearing with multiple slight indentations on the outer raceway. This signal is described as "a high-frequency signal with high peak value and pulse shape, which indicates the vibration of a faulty bearing with multiple slight indentations on the outer raceway".

[0066] After completing the textual descriptions of all time-series samples, a time-series-text pair dataset is constructed. This dataset covers the time-series signals themselves and their corresponding mechanistic knowledge. Each signal-text pair is provided as a sample unit for training the multimodal model, i.e.:

[0067]

[0068] In the formula, For the constructed signal-text pair dataset, (s i , t i ) represents a sample pair, where s i For signal data, t i The data is text, and N is the total number of sample pairs.

[0069] (3) Constructing a dual encoder based on the alignment of signal and text multimodal representations.

[0070] Using two Transformer encoders as signal encoders G s With text encoder G t The time-series signal and text description are encoded into d-dimensional vector representations, respectively:

[0071] z i =G s (s i ), w i=G t (t i )

[0072] In the formula, For the sample pair signal data s i The vector representation of , For the sample pair Chinese text description t i The vector representation of .

[0073] During training, B sample pairs are used as a training batch, and each batch is simultaneously fed into the signal encoder G. s With text encoder G t Training is performed to obtain representations of B signal data and B text descriptions. In this batch, the signals that make up the sample pairs have similar representations to the text, while signals that do not belong to the same pair have dissimilar representations to the text. The following contrastive loss function guides the training process of the dual encoder:

[0074]

[0075] In the formula, This is called the temperature hyperparameter, which is preset by humans to regulate... Sensitivity to differences in magnitude between different similarities. <·, ·> represent the cosine similarity between two vectors, where <·, ·> represents the similarity between any two vectors z. i w j For example, the cosine similarity between the two is calculated as follows:

[0076]

[0077] This multimodal representation alignment process is visible. Figure 1 The first half. After a certain number of learning rounds, a signal encoder and a text encoder are obtained, which have the ability to map signals and text to the same representation space.

[0078] (4) Constructing the “text + signal -> signal” paradigm in Signal GPT

[0079] like Figure 1 The lower part shows that the trained signal encoder and text encoder with alignment representation capabilities are connected in parallel, and a signal decoder F based on the Transformer structure is added after them. d This constitutes the entire multimodal generative pre-trained model. Besides the three main components—signal encoder, text encoder, and signal decoder—this model also includes a mask reconstruction layer F. out .

[0080] During training, a sample pair is input into the model, initially masking a portion of the signal data (the masked portion, where the signal amplitude is set to 0). A dual encoder then encodes the masked signal and the text to obtain their representation vectors. These two representation vectors are concatenated and fed into the signal decoder. The signal encoder fuses the features of the masked signal and the semantic description of the text to decode the characteristics of the masked portion of the signal. A further mask reconstruction layer completely restores the masked signal. This pre-training process uses the masked portion of the signal itself as a supervisory variable to guide the model's understanding of multimodal information fusion. The pre-training loss function is as follows:

[0081]

[0082] In the formula, This is a masked signal. y represents the model's prediction of the masked portion of the signal. msk The model aims to accurately reconstruct the masked signal by minimizing the mean squared error (MSA) of the two encoders, thereby improving multimodal understanding and signal generation capabilities. It's important to note that both encoders G are frozen at this point. s and G t The parameters are updated only for the signal decoder F. d and mask reconstruction layer F out The parameter θ:

[0083]

[0084] In the formula, α is the learning rate. Let θ be the gradient of the parameter θ.

[0085] The industrial control multimodal generative pre-trained model includes three main parts: a signal encoder, a text encoder, and a signal decoder. The signal encoder and the text encoder are connected in parallel and then connected to the signal decoder module.

[0086] The trained signal encoder and text encoder with representation alignment capability are connected in parallel. A signal decoder module based on the Transformer structure is added after them to receive the signal representation and text representation output by the dual encoders and perform multimodal deep fusion to complete the signal decoding and generation process, realizing the generation paradigm of "text + signal -> signal".

[0087] Specifically, when a sample pair is input into the model, a portion of the signal data is first masked (the masked portion is where the signal amplitude is 0). Then, the masked signal and the text are encoded by dual encoders to obtain their representation vectors. The two representation vectors are concatenated and sent to the signal decoder. The signal encoder fuses the features of the masked signal and the semantic description of the text to decode the characteristics of the masked part of the signal. Finally, a mask reconstruction layer completely restores the masked signal.

[0088] Those skilled in the art should understand that variations can be implemented by combining existing technology and the above embodiments. Such variations do not affect the substantive content of this solution and will not be elaborated here.

[0089] It should be understood that this solution is not limited to the specific embodiments described above. Devices and structures not described in detail herein should be understood as being implemented in a manner common to the art. Any person skilled in the art can make many possible variations and modifications to this solution, or modify it into equivalent embodiments, without departing from the scope of this solution, using the methods and techniques disclosed above. This does not affect the substantive content of this solution. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this solution, without departing from its scope, still fall within the protection scope of this solution.

Claims

1. A method for constructing a multimodal generative pre-trained model for industrial control systems that integrates data and mechanisms, characterized by: Includes the following steps: Step 1: Collect a large amount of industrial time-series signals, divide them into a certain number of samples by sliding window segmentation, and perform standardization preprocessing on each sample; Step 2: Based on mechanistic knowledge, provide textual descriptions of each time series sample, thereby generating a large number of time series signal-text pairs and constructing a dataset; specifically including the following steps: Step 2.1: Design a description template consisting of a template statement and two state words. The template statement remains constant, and the two state words describe the specific sample. Step 2.2: Extract the mathematical attributes of the sample to form the first state term, and extract the physical attributes of the sample to form the second state term; Step 2.3: Complete the text description of each signal sample based on the mechanistic knowledge, and combine the sample with its corresponding text description into pairs to form a signal-text pair dataset; Step 3: Construct a dual encoder that aligns the signal with the multimodal representation of the text, wherein the dual encoder includes a signal encoder and a text encoder; Step 4: Construct a pre-trained multimodal generative model for industrial control based on the "text + signal -> signal" paradigm; the specific process is as follows: Step 4.1: After connecting the signal encoder and text encoder in parallel, add a signal decoder module to realize the generation paradigm of "text + signal -> signal"; Step 4.2: Perform pre-training in the form of signal mask reconstruction; The specific process of step 4.2 is as follows: Step 4.2.1: Input a sample pair into the model to initially mask a portion of the signal data; Step 4.2.2: The mask signal and text are encoded by the dual encoder to obtain their representation vectors. The two representation vectors are concatenated and then sent to the signal decoder. Step 4.2.3: The signal encoder fuses the features of the masked signal and the semantics of the text description to decode the characteristics of the masked part of the signal, and then a mask reconstruction layer completely restores the masked signal. Step 4.2.4: Using the masked portion of the signal itself as a supervisory variable, apply the following pre-training loss function to guide the model's understanding of the fusion of multimodal information: In the formula, The signal is already masked. This represents the model's prediction of the masked portion of the signal. The true value of the masked part of the signal is given, and the mean square error between the two is calculated to restore the masked signal in the model.

2. The method for constructing a multimodal generative pre-trained model for industrial control based on data and mechanism fusion as described in claim 1, characterized in that: In step 1, the sample data is standardized and preprocessed using the following formula: in, This represents a segment of time-series signal in the dataset. and These are the mean and variance of the signal segment, respectively. It is any point of the signal. express The standardized value.

3. The method for constructing a multimodal generative pre-trained model for industrial control based on data and mechanism fusion as described in claim 1, characterized in that: In the new dataset, each signal-text pair is treated as a sample unit, represented by the following formula: In the formula, For signal-text pair datasets, For a sample pair, where For signal data, For text data, This represents the total number of sample pairs.

4. The method for constructing a multimodal generative pre-trained model for industrial control based on data and mechanism fusion as described in claim 1, characterized in that: Step 3, constructing the dual encoder, includes the following steps: Step 3.1: Construct a system that can encode time-series signals and text descriptions respectively. Signal encoder with 3D vector representation With text encoder It is expressed by the following formula: In the formula, For sample pair signal data The vector representation of , Chinese text description for sample pairs Vector representation of; Step 3.2: Train the signal encoder using a cross-modal contrastive approach. With text encoder The training process of the model is supervised by the similarity of the representation vectors of the signal and the text, enabling the two encoders to complete the multimodal alignment of the signal and the text.

5. The method for constructing a multimodal generative pre-trained model for industrial control based on data and mechanism fusion as described in claim 4, characterized in that: The process of representing vector similarity is expressed by the following formula: In the formula, , Let be the representation vectors for any signal and any text, respectively.

6. The method for constructing a multimodal generative pre-trained model for industrial control based on data and mechanism fusion as described in claim 5, characterized in that: The following contrastive loss function guides the training process of the dual encoder: In the formula, This is called the temperature hyperparameter, which is preset by humans and used for regulation. Sensitivity to size differences between different similarities.

7. An industrial control multimodal generative pre-trained model constructed using the method described in any one of claims 1 to 6, characterized in that: It consists of three main parts: a signal encoder, a text encoder, and a signal decoder. The signal encoder and the text encoder are connected in parallel, and then connected to the signal decoder module.

Images