A dynamic system sequence representation learning method and device based on a pre-trained network

Abstract

The application provides a kind of dynamic system sequence representation learning method and device based on pre-training network, it is related to artificial intelligence technical field.The method comprises: obtaining the observation data of dynamic system;Observation data includes: the observation value of the object of dynamic system;The observation value is divided into first sequence and second sequence;Add Gaussian noise to first sequence, obtain normalized first sequence by normalization processing;Construct the pre-training model of dynamic system modeling;Linear data projection module is used to project the normalized first sequence, and obtain the sequence after linear projection;The sequence after linear projection is reconstructed using the pre-trained language model, and the reconstructed sequence and the prediction sequence are obtained;The loss function of model is constructed using the minimization reconstruction loss;According to the loss function, the pre-training model of dynamic system modeling is obtained by training.The precision of dynamic system modeling can be improved by using the application.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for learning sequence representations of dynamic systems based on pre-trained networks. Background Technology

[0002] In many applications of complex real-world systems, learning system dynamics from observations is a key problem. In recent years, neural network-based dynamics modeling and learning methods have become the mainstream solution. The basic idea is to embed the original state of an object into a latent space and learn the dynamics using methods based on neural network ordinary differential equations (ODEs). Currently, mainstream dynamics modeling methods are data-driven, using neural ODEs to learn the evolutionary laws of dynamic systems from data and using neural networks to represent the ODEs of the control system's evolution. However, existing methods train a specific model for each system's observations, resulting in poor generalization ability. Pre-trained language networks are a model architecture based on pre-training and fine-tuning. In recent years, pre-trained language networks have been applied to sequence processing tasks, encoding time series as a string of numbers and predicting the next token in the text to achieve sequence prediction results.

[0003] Existing technologies for dynamic system modeling primarily focus on learning specific systems. The main idea is to embed the object's original state into a latent space, then use neural network-based methods to learn the dynamic system, and finally use a decoder to obtain the state representation in the original space. While existing methods have been successfully applied to various systems, most require observations from different systems to train the corresponding specific model, resulting in limited generalization capabilities. Summary of the Invention

[0004] To address the problem that existing dynamic system modeling methods have poor generalization ability and require training different models for different systems, this invention provides a dynamic system sequence representation learning method and apparatus based on pre-trained networks. The technical solution is as follows:

[0005] On the one hand, a dynamic system sequence representation learning method based on a pre-trained network is provided. This method is implemented by a dynamic system sequence representation learning device based on a pre-trained network, and includes:

[0006] S1. Obtain observation data of the dynamic system; the observation data includes: observed values ​​of objects in the dynamic system;

[0007] S2. Divide the observed values ​​into a first sequence and a second sequence;

[0008] S3. Construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0009] S4. Add Gaussian noise to the first sequence and normalize it to obtain the normalized first sequence; input the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence.

[0010] S5. Input the linearly projected sequence into the pre-trained language model for reconstruction to obtain the reconstructed sequence and the predicted sequence;

[0011] S6. Based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, construct a loss function for the pre-trained model of dynamic system modeling by minimizing the reconstruction loss;

[0012] S7. Train the initial pre-trained model for dynamic system modeling according to the loss function to obtain the trained pre-trained model for dynamic system modeling.

[0013] Optionally, the observation data may also include the interactions between objects in the dynamic system.

[0014] Optionally, the pre-trained model for the initial dynamic system modeling described in S3 includes: a linear data projection module and a pre-trained language model, and also includes a lexicalization module.

[0015] Optionally, before the step in S4 of inputting the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence, the method further includes:

[0016] The first sequence after normalization is input into the lexicalization module and divided to obtain multiple subsequences.

[0017] Optionally, the pre-trained language model includes: a convolutional layer, an encoder, and a decoder;

[0018] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input.

[0019] The encoder is used to encode the input sequence;

[0020] The decoder includes two linear layers: a first linear layer and a second linear layer, used to decode the hidden features to obtain the reconstructed sequence.

[0021] Optionally, step S5 involves inputting the linearly projected sequence into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence, including:

[0022] S51. Input the linearly projected sequence into the pre-trained language model, and process it through convolutional layers and encoders to obtain the reconstructed sequence.

[0023] S52. Input the reconstructed sequence into the first linear layer and the second linear layer for calculation. The reconstructed sequence is obtained through the processing of the first linear layer, and the predicted sequence is obtained through the processing of the second linear layer.

[0024] On the other hand, a dynamic system sequence representation learning device based on a pre-trained network is provided. This device is applied to a dynamic system sequence representation learning method based on a pre-trained network. The device includes:

[0025] The first acquisition unit is used to acquire observation data of the dynamic system; the observation data includes: the observation values ​​of objects in the dynamic system;

[0026] A partitioning unit is used to divide the observed values ​​into a first sequence and a second sequence;

[0027] The first building unit is used to build an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0028] The processing unit is used to add Gaussian noise to the first sequence, and obtain the normalized first sequence through normalization processing; the normalized first sequence is input into the linear data projection module for data projection to obtain the linearly projected sequence.

[0029] The second acquisition unit is used to input the linearly projected sequence into the pre-trained language model for reconstruction, and obtain the reconstructed sequence and the predicted sequence.

[0030] The second construction unit is used to construct a loss function for a pre-trained model of dynamic system modeling by minimizing the reconstruction loss based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence.

[0031] The training unit is used to train the initial pre-trained model of the dynamic system modeling according to the loss function, so as to obtain the trained pre-trained model of the dynamic system modeling.

[0032] Optionally, the observation data may also include the interactions between objects in the dynamic system.

[0033] Optionally, the pre-trained model for the initial dynamic system modeling includes: a linear data projection module and a pre-trained language model, and also includes a lexicalization module.

[0034] Optionally, before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence, the method further includes:

[0035] The first sequence after normalization is input into the lexicalization module and divided to obtain multiple subsequences.

[0036] Optionally, the pre-trained language model includes: a convolutional layer, an encoder, and a decoder;

[0037] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input.

[0038] The encoder is used to encode the input sequence;

[0039] The decoder includes two linear layers: a first linear layer and a second linear layer, used to decode the hidden features to obtain the reconstructed sequence.

[0040] Optionally, the second acquisition unit includes:

[0041] The linearly projected sequence is input into the pre-trained language model and processed through convolutional layers and an encoder to obtain the reconstructed sequence.

[0042] The reconstructed sequence is simultaneously input into the first linear layer and the second linear layer for computation. The reconstructed sequence is obtained through the processing of the first linear layer, and the predicted sequence is obtained through the processing of the second linear layer.

[0043] On the other hand, a dynamic system sequence representation learning device based on a pre-trained network is provided. The dynamic system sequence representation learning device based on a pre-trained network includes: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, any one of the methods described above for dynamic system sequence representation learning based on a pre-trained network is implemented.

[0044] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods of dynamic system sequence representation learning methods based on pre-trained networks.

[0045] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0046] This invention first acquires observation data of a dynamic system; the observation data includes: observed values ​​of objects in the dynamic system; the observed values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and normalization is performed to obtain a normalized first sequence; the normalized first sequence is input into the linear data projection module for data projection to obtain a linearly projected sequence; the linearly projected sequence is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, a loss function for the pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0047] This invention proposes a general encoder concept that combines a pre-trained language model with a dynamic system for data encoding in dynamic modeling. For training data with different dimensions, this invention proposes a linear data projection module for dimension alignment, allowing data of different dimensions to be used as input to the training model simultaneously. For complex systems, this invention can embed the original state into the latent space, thus easily capturing dynamics and improving the accuracy of dynamic system modeling. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of a dynamic system sequence representation learning based on a pre-trained network provided by an embodiment of the present invention;

[0050] Figure 2 This is a flowchart of a dynamic system sequence representation learning method based on a pre-trained network provided by an embodiment of the present invention;

[0051] Figure 3 This is a block diagram of a dynamic system sequence representation learning device based on a pre-trained network provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of a dynamic system sequence representation learning device based on a pre-trained network provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0054] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0055] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0056] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0057] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0058] This invention provides a dynamic system sequence representation learning method based on a pre-trained network. This method can be implemented by a dynamic system sequence representation learning device based on a pre-trained network, which can be a terminal or a server. Figure 1The diagram illustrates a structure for dynamic system sequence representation learning based on a pre-trained network, as provided in an embodiment of the present invention. In one feasible implementation, the entire model is divided into two parts: a pre-training part and a fine-tuning part. In the pre-training part, 154 sets of observation data from 24 dynamic systems are acquired as input for model training and divided into training data and prediction data according to a certain ratio. The training sequences are divided into token forms using a lexicalization module, and the data is projected onto a one-dimensional plane to adapt to the input form of the pre-trained language model. The encoder of the pre-trained language model maps the sequences into the latent space, and the decoder of the pre-trained language model, along with a linear projection layer, maps the data back to the original space and makes predictions for the prediction data. Finally, the pre-trained model for dynamic system modeling is obtained by minimizing the sequence reconstruction and prediction losses in an end-to-end manner. The fine-tuning part uses the parameters of the pre-trained model for dynamic system modeling as the initial parameters for fine-tuning. Modeling is performed on a specific dynamic system to obtain the ODE network representation of that specific system; the dynamic system may include meteorological systems and transportation systems.

[0059] like Figure 2 The flowchart shown is for a dynamic system sequence representation learning method based on a pre-trained network. The processing flow of this method may include the following steps:

[0060] S1. Obtain observation data of the dynamic system; the observation data includes: the observed values ​​of objects in the dynamic system.

[0061] In one feasible implementation, this application obtained generated data of 14 synthetic dynamic systems and real data of 10 real dynamic systems; the dynamic systems include: physical systems, liquid systems, climate systems and transportation systems.

[0062] In one feasible implementation, generating multiple sets of observation data for a traffic system can be represented as follows: , of which each Different parameters apply, including the number of objects in the system and the sequence length; Indicates the number of groups; where, express Observations of an object; Represents the interaction between objects; where the observed values ​​of the objects are represented as ,in, Indicates the length of the time series. The dimension representing the state of a dynamic system.

[0063] Optionally, the observation data may also include the interactions between objects in the dynamic system.

[0064] S2. Divide the observed values ​​into a first sequence and a second sequence.

[0065] The observations are manually divided into two parts according to a set ratio, including a first sequence and a second sequence.

[0066] S3. Construct the initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model.

[0067] Optionally, the pre-trained model for the initial dynamic system modeling of S3 includes: a linear data projection module and a pre-trained language model, as well as a lexicalization module.

[0068] S4. Add Gaussian noise to the first sequence and normalize it to obtain the normalized first sequence; input the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence.

[0069] Optionally, before the step in S4 of inputting the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence, the method further includes:

[0070] The first sequence after normalization is input into the lexicalization module and divided to obtain multiple subsequences.

[0071] In one feasible implementation, in order to adapt to observations of different lengths and to adapt to input formats based on the transformer architecture, the first sequence is divided into multiple subsequences with a certain step size; wherein, each subsequence can be regarded as a token in the transformer training data. By dividing the subsequences, local semantics can be maintained in long-term dynamic modeling, reducing the space and time costs of model learning.

[0072] In one feasible implementation, in order to handle the dimensionality diversity of states on different systems, a linear data projection module is used to map the sequence dimension to 1 dimension to align the observations and obtain the linearly projected sequence.

[0073] In one feasible implementation, in order to facilitate adaptation and generalization across domains, Gaussian noise is added and normalization is performed before the first sequence is divided into subsequences.

[0074] S5. Input the linearly projected sequence into the pre-trained language model for reconstruction to obtain the reconstructed sequence and the predicted sequence.

[0075] Among them, pre-trained language models (PLMs) are language models that are pre-trained in a self-supervised manner. They learn from a large amount of text data, optimize their parameters, and improve their ability to model text data.

[0076] Optionally, the pre-trained language model includes: convolutional layers, encoder, and decoder;

[0077] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input.

[0078] The encoder is used to encode the input sequence;

[0079] The decoder consists of two linear layers: a first linear layer and a second linear layer, which are used to decode the hidden features to obtain the reconstructed sequence.

[0080] Optionally, the specific implementation process of S5 may include S51-S52:

[0081] S51. Input the linearly projected sequence into the pre-trained language model, and process it through convolutional layers and encoders to obtain the reconstructed sequence;

[0082] In one feasible implementation, the process of obtaining the reconstructed sequence can be represented by the following formula (1):

[0083] (1)

[0084] in, Indicates the reconstructed sequence; Indicates encoder; Indicates a convolutional layer; This represents the sequence after linear projection.

[0085] S52. Input the reconstructed sequence into the first linear layer and the second linear layer for calculation. The reconstructed sequence is obtained through the processing of the first linear layer, and the predicted sequence is obtained through the processing of the second linear layer.

[0086] In one feasible implementation, the reconstructed sequence can be represented by the following formula (2):

[0087] (2)

[0088] in, This represents the reconstructed sequence; This indicates the first linear layer.

[0089] In one feasible implementation, the predicted sequence can be represented by the following formula (3):

[0090] (3)

[0091] in, Indicates the predicted sequence; This indicates the second linear layer.

[0092] S6. Based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, construct the loss function of the pre-trained model for dynamic system modeling by minimizing the reconstruction loss.

[0093] In one feasible implementation, the process of constructing the loss function of the pre-trained model for dynamic system modeling by minimizing the reconstruction loss can be represented by the following formula (4):

[0094] (4)

[0095] in, Indicates the first sequence; Indicates the second sequence; These represent the parameters of the encoder and decoder of the pre-trained language model; These represent the parameters of the encoder in the pre-trained language model; These represent the parameters of the decoder in the pre-trained language model; This indicates the design of the remaining training parameters; This represents minimizing the reconstruction loss; Represents the loss function; Indicates the sequence length; S represents different objects being observed; S represents different systems.

[0096] S7. Train the pre-trained model of the initial dynamic system modeling according to the loss function to obtain the trained pre-trained model of the dynamic system modeling.

[0097] Among them, the pre-trained models for dynamic system modeling can be applied to transportation, meteorology, and other fields.

[0098] In one feasible implementation, the parameters of a pre-trained model for modeling a dynamic system are used as the initial parameters for fine-tuning. Modeling is performed on a specific dynamic system to obtain the ODE network representation of the specific dynamic system; wherein, the specific dynamic system can be a transportation system or a meteorological system.

[0099] In one feasible implementation, this invention trains a general encoder using a large amount of data, achieving good learning performance under fine-tuning given any specific dynamic system state sequence. This invention performs long-term and short-term predictions on 18 dynamic systems of meteorology and traffic in both intra-domain and cross-domain scenarios. The comparative experimental results shown in Table 1 demonstrate the effectiveness of the proposed algorithm.

[0100] Table 1

[0101]

[0102] This invention first acquires observation data of a dynamic system; the observation data includes: observed values ​​of objects in the dynamic system; the observed values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and normalization is performed to obtain a normalized first sequence; the normalized first sequence is input into the linear data projection module for data projection to obtain a linearly projected sequence; the linearly projected sequence is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, a loss function for the pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0103] This invention proposes a general encoder concept that combines a pre-trained language model with a dynamic system for data encoding in dynamic modeling. For training data with different dimensions, this invention proposes a linear data projection module for dimension alignment, allowing data of different dimensions to be used as input to the training model simultaneously. For complex systems, this invention can embed the original state into the latent space, thus easily capturing dynamics and improving the accuracy of dynamic system modeling.

[0104] Figure 3 This is a block diagram illustrating a dynamic system sequence representation learning apparatus based on a pre-trained network, according to an exemplary embodiment. The apparatus is used in a dynamic system sequence representation learning method based on a pre-trained network. (Refer to...) Figure 3 The device includes a first acquisition unit 310, a partitioning unit 320, a first construction unit 330, a processing unit 340, a second acquisition unit 350, a second construction unit 360, and a training unit 370. Wherein:

[0105] The first acquisition unit 310 is used to acquire observation data of a dynamic system; the observation data includes: the observation values ​​of objects in the dynamic system.

[0106] The partitioning unit 320 is used to divide the observed values ​​into a first sequence and a second sequence;

[0107] The first building unit 330 is used to build an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model;

[0108] Processing unit 340 is used to add Gaussian noise to the first sequence, and obtain the normalized first sequence through normalization processing; input the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence;

[0109] The second acquisition unit 350 is used to input the linearly projected sequence into the pre-trained language model for reconstruction, and obtain the reconstructed sequence and the predicted sequence.

[0110] The second construction unit 360 is used to construct a loss function for a pre-trained model of dynamic system modeling by minimizing the reconstruction loss based on the reconstructed sequence, the predicted sequence, the first sequence and the second sequence.

[0111] Training unit 370 is used to train the initial pre-trained model of dynamic system modeling according to the loss function to obtain the trained pre-trained model of dynamic system modeling.

[0112] Optionally, the observation data may also include the interactions between objects in the dynamic system.

[0113] Optionally, the pre-trained model for the initial dynamic system modeling includes: a linear data projection module and a pre-trained language model, and also includes a lexicalization module.

[0114] Optionally, before the step of inputting the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence, the method further includes:

[0115] The first sequence after normalization is input into the lexicalization module and divided to obtain multiple subsequences.

[0116] Optionally, the pre-trained language model includes: a convolutional layer, an encoder, and a decoder;

[0117] The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input.

[0118] The encoder is used to encode the input sequence;

[0119] The decoder includes two linear layers: a first linear layer and a second linear layer, used to decode the hidden features to obtain the reconstructed sequence.

[0120] Optionally, the second acquisition unit 360 includes:

[0121] The linearly projected sequence is input into the pre-trained language model and processed through convolutional layers and an encoder to obtain the reconstructed sequence.

[0122] The reconstructed sequence is simultaneously input into the first linear layer and the second linear layer for computation. The reconstructed sequence is obtained through the processing of the first linear layer, and the predicted sequence is obtained through the processing of the second linear layer.

[0123] This invention first acquires observation data of a dynamic system; the observation data includes: observed values ​​of objects in the dynamic system; the observed values ​​are divided into a first sequence and a second sequence; secondly, an initial pre-trained model for dynamic system modeling is constructed; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; Gaussian noise is added to the first sequence, and normalization is performed to obtain a normalized first sequence; the normalized first sequence is input into the linear data projection module for data projection to obtain a linearly projected sequence; the linearly projected sequence is input into the pre-trained language model for reconstruction to obtain a reconstructed sequence and a predicted sequence; based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, a loss function for the pre-trained model for dynamic system modeling is constructed by minimizing the reconstruction loss; finally, the initial pre-trained model for dynamic system modeling is trained according to the loss function to obtain a trained pre-trained model for dynamic system modeling.

[0124] This invention proposes a general encoder concept that combines a pre-trained language model with a dynamic system for data encoding in dynamic modeling. For training data with different dimensions, this invention proposes a linear data projection module for dimension alignment, allowing data of different dimensions to be used as input to the training model simultaneously. For complex systems, this invention can embed the original state into the latent space, thus easily capturing dynamics and improving the accuracy of dynamic system modeling.

[0125] Figure 4 This is a schematic diagram of the structure of a dynamic system sequence representation learning device based on a pre-trained network provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a dynamic system sequence representation learning device based on a pre-trained network can include the above-mentioned... Figure 3 The illustrated dynamic system sequence representation learning device is based on a pre-trained network. Optionally, the dynamic system sequence representation learning device 410 based on a pre-trained network may include a first processor 2001.

[0126] Optionally, the dynamic system sequence representation learning device 410 based on the pre-trained network may also include a memory 2002 and a transceiver 2003.

[0127] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0128] The following is combined with Figure 4 The components of the dynamic system sequence representation learning device 410 based on pre-trained networks are described in detail below:

[0129] The first processor 2001 is the control center of the dynamic system sequence representation learning device 410 based on a pre-trained network. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0130] Optionally, the first processor 2001 can perform various functions of the dynamic system sequence representation learning device 410 based on the pre-trained network by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0131] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0132] In a specific implementation, as one example, the dynamic system sequence representation learning device 410 based on a pre-trained network may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0133] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0134] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected to the interface circuit of the dynamic system sequence representation learning device 410 based on a pre-trained network. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0135] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0136] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0137] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the learning device 410 based on the dynamic system sequence representation of the pre-trained network. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0138] It should be noted that, Figure 4 The structure of the dynamic system sequence representation learning device 410 based on the pre-trained network shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown, or combine some components, or have different component arrangements.

[0139] Furthermore, the technical effect of the dynamic system sequence representation learning device 410 based on the pre-trained network can be referred to the technical effect of the dynamic system sequence representation learning method based on the pre-trained network described in the above method embodiments, and will not be repeated here.

[0140] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0141] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0142] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0143] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0144] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0145] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0146] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0147] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0148] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A dynamic system sequence representation learning method based on pre-trained networks, characterized in that, The method includes: S1. Obtain observation data of the dynamic system; the observation data includes: observed values ​​of objects in the dynamic system; Among them, the dynamic system is the transportation system; The transportation system generates multiple sets of observation data, represented as follows: Each of them For different parameters, including the number of objects in the system and the sequence length; M S Indicates the number of groups; N represents m Observations of each object; ε m Represents the interactions between objects; the observed values ​​of the objects are represented as T m V represents the length of the time series. s Dimensions representing the state of a dynamic system; S2. Divide the observed values ​​into a first sequence and a second sequence; S3. Construct an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; S4. Add Gaussian noise to the first sequence and normalize it to obtain the normalized first sequence; input the normalized first sequence into the linear data projection module for data projection to obtain the linearly projected sequence. S5. Input the linearly projected sequence into the pre-trained language model for reconstruction to obtain the reconstructed sequence and the predicted sequence; The pre-trained language model includes: a convolutional layer, an encoder, and a decoder; The convolutional layer is used to maintain local semantic information and adjust the dimension of the input state to the dimension of the encoder input. The encoder is used to encode the input sequence; The decoder includes two linear layers: a first linear layer and a second linear layer, used to decode the hidden features to obtain the reconstructed sequence; Specifically, step S5 involves inputting the linearly projected sequence into the pre-trained language model for reconstruction to obtain the reconstructed sequence and the predicted sequence, including: S51. Input the linearly projected sequence into the pre-trained language model, and process it through convolutional layers and encoders to obtain the reconstructed sequence. S52. Input the reconstructed sequence into the first linear layer and the second linear layer simultaneously for calculation. The reconstructed sequence is obtained through the processing of the first linear layer, and the predicted sequence is obtained through the processing of the second linear layer. S6. Based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence, construct a loss function for the pre-trained model of dynamic system modeling by minimizing the reconstruction loss; S7. Train the initial pre-trained model for dynamic system modeling according to the loss function to obtain a trained pre-trained model for dynamic system modeling. In this process, the parameters of a pre-trained model used for modeling a dynamic system are used as the initial parameters for fine-tuning. Modeling is performed on a specific dynamic system to obtain the ODE network representation of that specific dynamic system; where the specific dynamic system is a traffic system.

2. The dynamic system sequence representation learning method based on pre-trained networks according to claim 1, characterized in that, The observation data also includes the interactions between objects in the dynamic system.

3. The dynamic system sequence representation learning method based on pre-trained networks according to claim 1, characterized in that, The pre-trained model for the initial dynamic system modeling described in S3 includes a linear data projection module and a pre-trained language model, as well as a lexicalization module.

4. The dynamic system sequence representation learning method based on pre-trained networks according to claim 3, characterized in that, Before the step in S4 where the normalized first sequence is input into the linear data projection module for data projection to obtain the linearly projected sequence, the method further includes: The first sequence after normalization is input into the lexicalization module and divided to obtain multiple subsequences.

5. A dynamic system sequence representation learning device based on a pre-trained network, wherein the dynamic system sequence representation learning device based on a pre-trained network is used to implement the dynamic system sequence representation learning method based on a pre-trained network as described in any one of claims 1-4, characterized in that, The device includes: The first acquisition unit is used to acquire observation data of the dynamic system; the observation data includes: the observation values ​​of objects in the dynamic system; A partitioning unit is used to divide the observed values ​​into a first sequence and a second sequence; The first building unit is used to build an initial pre-trained model for dynamic system modeling; the initial pre-trained model for dynamic system modeling includes: a linear data projection module and a pre-trained language model; The processing unit is used to add Gaussian noise to the first sequence, and obtain the normalized first sequence through normalization processing; the normalized first sequence is input into the linear data projection module for data projection to obtain the linearly projected sequence. The second acquisition unit is used to input the linearly projected sequence into the pre-trained language model for reconstruction, and obtain the reconstructed sequence and the predicted sequence. The second construction unit is used to construct a loss function for a pre-trained model of dynamic system modeling by minimizing the reconstruction loss based on the reconstructed sequence, the predicted sequence, the first sequence, and the second sequence. The training unit is used to train the initial pre-trained model of the dynamic system modeling according to the loss function, so as to obtain the trained pre-trained model of the dynamic system modeling.

6. The dynamic system sequence representation learning device based on a pre-trained network according to claim 5, characterized in that, The observation data also includes the interactions between objects in the dynamic system.

7. A dynamic system sequence representation learning device based on a pre-trained network, characterized in that, The dynamic system sequence representation learning device based on pre-trained networks includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 4.

Images