Dynamic system sequence representation learning method and device based on pre-training network

Abstract

The invention provides a dynamic system sequence representation learning method and device based on a pre-training network, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring observation data of a dynamic system; the observation data comprises an observation value of an object of the dynamic system; dividing the observed value into a first sequence and a second sequence; gaussian noise is added to the first sequence, and the first sequence after normalization processing is obtained through normalization processing; constructing a pre-training model for dynamic system modeling; performing data projection on the normalized first sequence by adopting a linear data projection module to obtain a linearly projected sequence; a pre-trained language model is adopted to reconstruct the sequence after linear projection, and a reconstructed sequence and a prediction sequence are obtained; constructing a loss function of the model by adopting the minimum reconstruction loss; and training according to the loss function to obtain a pre-training model for dynamic system modeling. By adopting the method, the modeling accuracy of the dynamic system can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dynamic system sequence representation learning method and device based on a pre-trained network. Background Art

[0002] In many applications of complex systems in the real world, learning system dynamics from observations is a key issue. In recent years, dynamic modeling learning methods based on neural networks have become the mainstream solution. The basic idea is to embed the original state of the object into the latent space and learn the dynamics using methods based on neural network ordinary differential equations. The current mainstream dynamic modeling method is data-driven, using neural ordinary difference equations to learn the evolution law of the dynamic system from data, and using neural networks to represent the ordinary differential equations that control the evolution of the system; for observations from different systems, the existing method is to train a specific model for each system's observations, resulting in poor generalization of the model. Pre-trained language networks are a model architecture based on pre-training and fine-tuning. In recent years, pre-trained language networks have been applied to sequence processing tasks, encoding time series into a string of numbers, and achieving sequence prediction results by predicting the next token in the text.

[0003] Existing technologies in dynamic system modeling mainly focus on learning specific systems. The main idea is to embed the original state of the object into the latent space, then use a neural network-based method to learn the dynamic system, and finally use a decoder to get the representation of the state in the original space. Although existing methods have been successfully applied to various systems, most methods must be given observations of different systems to train the corresponding specific models, resulting in limited generalization capabilities. Summary of the invention

[0004] In order to solve the technical problem that the existing dynamic system modeling method in the prior art has poor generalization ability and needs to train different models for different systems, the embodiment of the present invention provides a dynamic system sequence representation learning method and device based on a pre-trained network. The technical solution is as follows:

[0005] In one aspect, a method for learning a dynamic system sequence representation based on a pre-trained network is provided. The method is implemented by a dynamic system sequence representation learning device based on a pre-trained network. The method comprises:

[0006] S1. Obtain observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system;

[0007] S2, dividing the observed values ​​into a first sequence and a second sequence;

[0008] S3, constructing an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0009] S4, adding Gaussian noise to the first sequence, performing normalization processing, and obtaining a normalized first sequence; inputting the normalized first sequence into the linear data projection module for data projection, and obtaining a linearly projected sequence;

[0010] S5, inputting the sequence after linear projection into the pre-trained language model for reconstruction, and obtaining a reconstructed sequence and a predicted sequence;

[0011] S6. According to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, a loss function of a pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss;

[0012] S7. Train the initial pre-trained model for dynamic system modeling according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0013] Optionally, the observation data also includes: interactions between objects in the dynamic system.

[0014] Optionally, the pre-trained model for initial dynamic system modeling in S3 includes: a linear data projection module and a pre-trained language model, and also includes a lemma module.

[0015] Optionally, before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the sequence after linear projection in S4, the method further includes:

[0016] The normalized first sequence is input into the lemma module for segmentation to obtain multiple subsequences.

[0017] Optionally, the pre-trained language model includes: a convolutional layer, an encoder and a decoder;

[0018] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input;

[0019] Wherein, the encoder is used to encode the input sequence;

[0020] The decoder includes two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain a reconstructed sequence.

[0021] Optionally, the step of inputting the linearly projected sequence into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence in S5 includes:

[0022] S51, inputting the linearly projected sequence into the pre-trained language model, processing it through a convolutional layer and an encoder, and obtaining a reconstructed sequence;

[0023] S52, inputting the reconstructed sequence into the first linear layer and the second linear layer for calculation at the same time, obtaining the reconstructed sequence through the processing of the first linear layer, and obtaining the predicted sequence through the processing of the second linear layer.

[0024] On the other hand, a dynamic system sequence representation learning device based on a pre-trained network is provided, and the device is applied to a dynamic system sequence representation learning method based on a pre-trained network, and the device comprises:

[0025] A first acquisition unit is used to acquire observation data of the dynamic system; the observation data includes: observation values ​​of objects of the dynamic system;

[0026] A division unit, used for dividing the observation values ​​into a first sequence and a second sequence;

[0027] A first construction unit is used to construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0028] A processing unit, configured to add Gaussian noise to the first sequence, perform normalization processing, and obtain a normalized first sequence; input the normalized first sequence into the linear data projection module for data projection, and obtain a linearly projected sequence;

[0029] A second acquisition unit is used to input the linearly projected sequence into the pre-trained language model for reconstruction, so as to obtain a reconstructed sequence and a predicted sequence;

[0030] A second construction unit is used to construct a loss function of a pre-trained model for dynamic system modeling by minimizing reconstruction loss according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence;

[0031] The training unit is used to train the initial pre-trained model of dynamic system modeling according to the loss function to obtain a trained pre-trained model of dynamic system modeling.

[0032] Optionally, the observation data also includes: interactions between objects in the dynamic system.

[0033] Optionally, the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model, and also includes a lemma module.

[0034] Optionally, before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the sequence after linear projection, the step further includes:

[0035] The normalized first sequence is input into the lemma module for segmentation to obtain multiple subsequences.

[0036] Optionally, the pre-trained language model includes: a convolutional layer, an encoder and a decoder;

[0037] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input;

[0038] Wherein, the encoder is used to encode the input sequence;

[0039] The decoder includes two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain a reconstructed sequence.

[0040] Optionally, the second acquiring unit includes:

[0041] Inputting the linearly projected sequence into the pre-trained language model, processing it through a convolutional layer and an encoder to obtain a reconstructed sequence;

[0042] The reconstructed sequence is input into the first linear layer and the second linear layer for calculation at the same time. The reconstructed sequence is obtained by processing in the first linear layer, and the predicted sequence is obtained by processing in the second linear layer.

[0043] On the other hand, a dynamic system sequence representation learning device based on a pre-trained network is provided, and the dynamic system sequence representation learning device based on a pre-trained network comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned dynamic system sequence representation learning methods based on a pre-trained network is implemented.

[0044] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dynamic system sequence representation learning methods based on a pre-trained network.

[0045] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0046] The embodiment of the present invention first obtains observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system; the observation values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and a normalized first sequence is obtained through normalization processing; the normalized first sequence is input into the linear data projection module for data projection to obtain a sequence after linear projection; the sequence after linear projection is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, the reconstruction loss is minimized to construct a loss function of the pre-trained model for dynamic system modeling; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0047] The embodiment of the present invention proposes a general encoder concept combining a pre-trained language model with a dynamic system for data encoding of dynamic modeling; for the different dimensions of training data, the present invention proposes a linear data projection module for dimensional alignment, so that data of different dimensions can be used as input of the training model at the same time. For complex systems, the present invention can embed the original state into the latent space, so that dynamics can be easily captured. The use of the present invention can improve the accuracy of dynamic system modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a structural schematic diagram of a dynamic system sequence representation learning based on a pre-trained network provided by an embodiment of the present invention;

[0050] Figure 2 It is a flow chart of a dynamic system sequence representation learning method based on a pre-trained network provided by an embodiment of the present invention;

[0051] Figure 3 It is a block diagram of a dynamic system sequence representation learning device based on a pre-trained network provided by an embodiment of the present invention;

[0052] Figure 4 It is a structural schematic diagram of a dynamic system sequence representation learning device based on a pre-trained network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0054] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0055] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0056] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0057] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0058] The embodiment of the present invention provides a method for learning a dynamic system sequence representation based on a pre-trained network. The method can be implemented by a dynamic system sequence representation learning device based on a pre-trained network. The dynamic system sequence representation learning device based on a pre-trained network can be a terminal or a server. Figure 1The figure shows a structural diagram of a dynamic system sequence representation learning based on a pre-trained network provided by an embodiment of the present invention; in a feasible implementation method, the entire model is divided into two parts, namely a pre-training part and a fine-tuning part. In the pre-training part, 154 groups of observation data of 24 dynamic systems are obtained as inputs for model training, and are divided into training data and prediction data in a certain proportion; the training sequence is divided into token form through the lemma module, and the data is projected into one dimension to adapt to the input form of the pre-trained language model, and the sequence is mapped to the latent space through the encoder of the pre-trained language model, and the data is mapped back to the original space and the prediction data is predicted through the decoder of the pre-trained language model plus the linear projection layer. Finally, the pre-trained model for dynamic system modeling is obtained by minimizing the loss of sequence reconstruction and prediction in an end-to-end manner. The fine-tuning part uses the parameters of the pre-trained model for dynamic system modeling as the initial parameters for fine-tuning, models a specific dynamic system, and obtains the ODE network expression of the specific system; wherein the dynamic system may include a meteorological system and a traffic system.

[0059] like Figure 2 The flowchart of the dynamic system sequence representation learning method based on the pre-trained network is shown. The processing flow of the method may include the following steps:

[0060] S1. Obtain observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system.

[0061] In a feasible implementation, the present application obtains generated data of 14 synthetic dynamic systems and real data of 10 real dynamic systems; the dynamic systems include: physical systems, liquid systems, climate systems and traffic systems.

[0062] In one feasible implementation, generating multiple sets of observation data for a traffic system can be expressed as , where each corresponding to different parameters, including the number of objects in the system and the length of the sequence; represents the number of groups; express The observed value of each object; represents the interaction between objects; where the observed values ​​of the objects are represented as ,in, represents the length of the time series, A dimension that represents the state of a dynamic system.

[0063] Optionally, the observation data also includes: interactions between objects of the dynamic system.

[0064] S2. Divide the observations into the first sequence and the second sequence.

[0065] Among them, the observations are manually divided into two parts according to the set ratio, including the first series and the second series.

[0066] S3. Construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model.

[0067] Optionally, the pre-trained model for the initial dynamic system modeling of S3 includes: a linear data projection module and a pre-trained language model, and also includes a lemma module.

[0068] S4, adding Gaussian noise to the first sequence, performing normalization processing to obtain a normalized first sequence; inputting the normalized first sequence into a linear data projection module to perform data projection to obtain a linearly projected sequence.

[0069] Optionally, before the step of inputting the normalized first sequence into a linear data projection module for data projection and obtaining the sequence after linear projection in S4, the method further includes:

[0070] The normalized first sequence is input into the lemma module for segmentation to obtain multiple subsequences.

[0071] In a feasible implementation, in order to adapt to observations of different lengths and to adapt to the input form based on the transformer architecture, the first sequence is divided into multiple subsequences with a certain step size; wherein each subsequence can be regarded as a token in the transformer training data. By dividing the subsequences, local semantics can be maintained in long-term dynamic modeling, thereby reducing the space and time costs of model learning.

[0072] In a feasible implementation, in order to handle the dimensional diversity of states on different systems, a linear data projection module is used to map the sequence dimension to 1 dimension to align the observations and obtain the sequence after linear projection.

[0073] In a feasible implementation, in order to facilitate inter-domain adaptation and generalization, Gaussian noise is added before the first sequence is divided into subsequences and normalization is performed.

[0074] S5. Input the linearly projected sequence into the pre-trained language model for reconstruction to obtain the reconstructed sequence and the predicted sequence.

[0075] Among them, the pre-trained language model (PLM) is a language model that is pre-trained in a self-supervised manner. It optimizes its parameters by learning a large amount of text data and improving its modeling ability for text data.

[0076] Optionally, a pre-trained language model, comprising: a convolutional layer, an encoder, and a decoder;

[0077] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input;

[0078] Among them, the encoder is used to encode the input sequence;

[0079] The decoder includes two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain a reconstructed sequence.

[0080] Optionally, the specific implementation process of S5 may include S51-S52:

[0081] S51, inputting the linearly projected sequence into the pre-trained language model, processing it through the convolution layer and the encoder, and obtaining a reconstructed sequence;

[0082] In a feasible implementation, the process of obtaining the reconstructed sequence can be expressed by the following formula (1):

[0083] (1)

[0084] in, Represents a reconstruction sequence; represents the encoder; represents the convolutional layer; Represents the sequence after linear projection.

[0085] S52, inputting the reconstructed sequence into the first linear layer and the second linear layer for calculation at the same time, obtaining the reconstructed sequence through the processing of the first linear layer, and obtaining the predicted sequence through the processing of the second linear layer.

[0086] In a feasible implementation, the reconstructed sequence can be expressed by the following formula (2):

[0087] (2)

[0088] in, represents the reconstructed sequence; represents the first linear layer.

[0089] In a feasible implementation, the prediction sequence can be expressed by the following formula (3):

[0090] (3)

[0091] in, represents the prediction sequence; represents the second linear layer.

[0092] S6. According to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, the loss function of the pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss.

[0093] In a feasible implementation, the process of constructing the loss function of the pre-trained model for dynamic system modeling by minimizing the reconstruction loss can be expressed by the following formula (4):

[0094] (4)

[0095] in, represents the first sequence; represents the second sequence; Represents the parameters of the encoder and decoder of the pre-trained language model; Represents the parameters of the encoder of the pre-trained language model; Describes the parameters of the decoder of the pre-trained language model; Represents the design of the remaining training parameters; represents the minimization of reconstruction loss; represents the loss function; Indicates the length of the sequence; represents different objects of observation; S represents different systems.

[0096] S7. Train the initial pre-trained model for dynamic system modeling according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0097] Among them, the pre-trained model for dynamic system modeling can be applied to aspects such as transportation and meteorology.

[0098] In a feasible implementation, the parameters of a pre-trained model for modeling a trained dynamic system are used as initial parameters for fine-tuning, and a specific dynamic system is modeled to obtain an ODE network expression of the specific dynamic system; wherein the specific dynamic system may be a traffic system or a meteorological system.

[0099] In a feasible implementation, the present invention uses a large amount of data to train a universal encoder, which can achieve good learning results given any specific dynamic system state sequence fine-tuning. The present invention performs long-term and short-term predictions on 18 dynamic systems of meteorology and traffic in both intra-domain and cross-domain situations, as shown in Table 1. The comparative experimental results show the effectiveness of the algorithm proposed by the present invention.

[0100] Table 1

[0101]

[0102] The embodiment of the present invention first obtains observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system; the observation values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and a normalized first sequence is obtained through normalization processing; the normalized first sequence is input into the linear data projection module for data projection to obtain a sequence after linear projection; the sequence after linear projection is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, the reconstruction loss is minimized to construct a loss function of the pre-trained model for dynamic system modeling; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0103] The embodiment of the present invention proposes a general encoder concept combining a pre-trained language model with a dynamic system for data encoding of dynamic modeling; for the different dimensions of training data, the present invention proposes a linear data projection module for dimensional alignment, so that data of different dimensions can be used as input of the training model at the same time. For complex systems, the present invention can embed the original state into the latent space, so that dynamics can be easily captured. The use of the present invention can improve the accuracy of dynamic system modeling.

[0104] Figure 3 The present invention is a block diagram of a dynamic system sequence representation learning device based on a pre-trained network according to an exemplary embodiment. The device is used in a dynamic system sequence representation learning method based on a pre-trained network. Figure 3 The device includes a first acquisition unit 310, a division unit 320, a first construction unit 330, a processing unit 340, a second acquisition unit 350, a second construction unit 360 and a training unit 370. Among them:

[0105] The first acquisition unit 310 is used to acquire observation data of the dynamic system; the observation data includes: observation values ​​of objects of the dynamic system;

[0106] A division unit 320, configured to divide the observation values ​​into a first sequence and a second sequence;

[0107] The first construction unit 330 is used to construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0108] The processing unit 340 is configured to add Gaussian noise to the first sequence, perform normalization processing, and obtain a normalized first sequence; input the normalized first sequence into the linear data projection module to perform data projection, and obtain a linearly projected sequence;

[0109] A second acquisition unit 350 is used to input the linearly projected sequence into the pre-trained language model for reconstruction, and obtain a reconstructed sequence and a predicted sequence;

[0110] A second construction unit 360 is used to construct a loss function of a pre-trained model for dynamic system modeling by minimizing reconstruction loss according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence;

[0111] The training unit 370 is used to train the initial pre-trained model for dynamic system modeling according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0112] Optionally, the observation data also includes: interactions between objects in the dynamic system.

[0113] Optionally, the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model, and also includes a lemma module.

[0114] Optionally, before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the sequence after linear projection, the step further includes:

[0115] The normalized first sequence is input into the lemma module for segmentation to obtain multiple subsequences.

[0116] Optionally, the pre-trained language model includes: a convolutional layer, an encoder and a decoder;

[0117] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input;

[0118] Wherein, the encoder is used to encode the input sequence;

[0119] The decoder includes two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain a reconstructed sequence.

[0120] Optionally, the second acquiring unit 360 includes:

[0121] Inputting the linearly projected sequence into the pre-trained language model, processing it through a convolutional layer and an encoder to obtain a reconstructed sequence;

[0122] The reconstructed sequence is input into the first linear layer and the second linear layer for calculation at the same time. The reconstructed sequence is obtained by processing in the first linear layer, and the predicted sequence is obtained by processing in the second linear layer.

[0123] The embodiment of the present invention first obtains observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system; the observation values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and a normalized first sequence is obtained through normalization processing; the normalized first sequence is input into the linear data projection module for data projection to obtain a sequence after linear projection; the sequence after linear projection is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, the reconstruction loss is minimized to construct a loss function of the pre-trained model for dynamic system modeling; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0124] The embodiment of the present invention proposes a general encoder concept combining a pre-trained language model with a dynamic system for data encoding of dynamic modeling; for the different dimensions of training data, the present invention proposes a linear data projection module for dimensional alignment, so that data of different dimensions can be used as input of the training model at the same time. For complex systems, the present invention can embed the original state into the latent space, so that dynamics can be easily captured. The use of the present invention can improve the accuracy of dynamic system modeling.

[0125] Figure 4 is a structural diagram of a dynamic system sequence representation learning device based on a pre-trained network provided by an embodiment of the present invention, such as Figure 4 As shown, the dynamic system sequence representation learning device based on the pre-trained network may include the above Figure 3 The dynamic system sequence representation learning device based on the pre-trained network is shown. Optionally, the dynamic system sequence representation learning device based on the pre-trained network 410 may include a first processor 2001.

[0126] Optionally, the dynamic system sequence representation learning device 410 based on a pre-trained network may further include a memory 2002 and a transceiver 2003 .

[0127] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0128] Combine the following Figure 4 The components of the dynamic system sequence representation learning device 410 based on the pre-trained network are specifically introduced:

[0129] The first processor 2001 is the control center of the dynamic system sequence representation learning device 410 based on the pre-trained network, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0130] Optionally, the first processor 2001 can perform various functions of the dynamic system sequence representation learning device 410 based on the pre-trained network by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0131] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0132] In a specific implementation, as an embodiment, the dynamic system sequence representation learning device 410 based on the pre-trained network may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0133] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0134] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0135] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0136] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0137] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be implemented through an interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0138] It should be noted that Figure 4 The structure of the dynamic system sequence representation learning device 410 based on the pre-trained network shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0139] In addition, the technical effects of the dynamic system sequence representation learning device 410 based on a pre-trained network can refer to the technical effects of the dynamic system sequence representation learning method based on a pre-trained network described in the above method embodiment, and will not be repeated here.

[0140] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0141] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0142] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0143] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0144] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0145] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0152] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for learning sequence representation of dynamic systems based on a pre-trained network, characterized in that: The method comprises: S1. Obtain observation data of a dynamic system; the observation data includes: observation values ​​of objects of the dynamic system; S2, dividing the observed values ​​into a first sequence and a second sequence; S3, constructing an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; S4, adding Gaussian noise to the first sequence, performing normalization processing, and obtaining a normalized first sequence; inputting the normalized first sequence into the linear data projection module for data projection, and obtaining a linearly projected sequence; S5, inputting the sequence after linear projection into the pre-trained language model for reconstruction, and obtaining a reconstructed sequence and a predicted sequence; S6. According to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence, a loss function of a pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss; S7. Train the initial pre-trained model for dynamic system modeling according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

2. The method for learning dynamic system sequence representation based on a pre-trained network according to claim 1, characterized in that: The observation data also includes: interactions between objects of the dynamic system.

3. The method for learning dynamic system sequence representation based on a pre-trained network according to claim 1, characterized in that: The pre-trained model for the initial dynamic system modeling described in S3 includes: a linear data projection module and a pre-trained language model, and also includes a lemma module.

4. The method for learning dynamic system sequence representation based on a pre-trained network according to claim 3, characterized in that: Before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the sequence after linear projection in S4, the method further includes: The normalized first sequence is input into the lemma module for segmentation to obtain multiple subsequences.

5. The method for learning dynamic system sequence representation based on a pre-trained network according to claim 1, characterized in that: The pre-trained language model includes: a convolutional layer, an encoder and a decoder; The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input; Wherein, the encoder is used to encode the input sequence; The decoder includes two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain a reconstructed sequence.

6. The method for learning dynamic system sequence representation based on a pre-trained network according to claim 5, characterized in that: The step S5 of inputting the linearly projected sequence into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence includes: S51, inputting the linearly projected sequence into the pre-trained language model, processing it through a convolutional layer and an encoder, and obtaining a reconstructed sequence; S52, inputting the reconstructed sequence into the first linear layer and the second linear layer for calculation at the same time, obtaining the reconstructed sequence through the processing of the first linear layer, and obtaining the predicted sequence through the processing of the second linear layer.

7. A dynamic system sequence representation learning device based on a pre-trained network, the dynamic system sequence representation learning device based on a pre-trained network is used to implement the dynamic system sequence representation learning method based on a pre-trained network as claimed in any one of claims 1 to 6, characterized in that: The device comprises: A first acquisition unit is used to acquire observation data of the dynamic system; the observation data includes: observation values ​​of objects of the dynamic system; A division unit, used for dividing the observation values ​​into a first sequence and a second sequence; A first construction unit is used to construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; A processing unit, configured to add Gaussian noise to the first sequence, perform normalization processing, and obtain a normalized first sequence; input the normalized first sequence into the linear data projection module for data projection, and obtain a linearly projected sequence; A second acquisition unit is used to input the linearly projected sequence into the pre-trained language model for reconstruction, so as to obtain a reconstructed sequence and a predicted sequence; A second construction unit is used to construct a loss function of a pre-trained model for dynamic system modeling by minimizing reconstruction loss according to the reconstructed sequence, the predicted sequence, the first sequence and the second sequence; The training unit is used to train the initial pre-trained model of dynamic system modeling according to the loss function to obtain a trained pre-trained model of dynamic system modeling.

8. The dynamic system sequence representation learning device based on pre-trained network according to claim 7, characterized in that: The observation data also includes: interactions between objects of the dynamic system.

9. A dynamic system sequence representation learning device based on a pre-trained network, characterized in that: The dynamic system sequence representation learning device based on the pre-trained network includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images