AI model training method and device and AI model reasoning method and device

Abstract

The invention provides an AI model training method, reasoning method and device, and the AI model training method comprises the steps: obtaining a training data set, wherein the training data set comprises a plurality of pieces of first time sequence sample data; obtaining a plurality of first mask sequences, the plurality of first mask sequences being obtained by performing multiple times of frequency domain mask operations on the first time sequence sample data, the multiple times of frequency domain mask operations covering frequency domain data corresponding to the time sequence sample data being different in frequency component; taking the plurality of first mask sequences as input of a to-be-trained A I model, and outputting first reconstruction time sequence data; and based on the first reconstruction time sequence data and the first time sequence sample data corresponding to the first reconstruction time sequence data, adjusting a weight parameter of the to-be-trained A I model to obtain a pre-trained A I model. According to the method, a multi-frequency domain mask reconstruction mode is adopted, so that the model has very strong information extraction capability, and the prediction precision of the model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a training method, inference method and apparatus for an AI model. Background Technology

[0002] Time series forecasting has wide applications in various fields, including energy, transportation, economics, weather, and medicine. However, many problems still exist in specific application scenarios. For example, the insufficient information extraction capability of time series forecasting models leads to poor predictive performance. Summary of the Invention

[0003] The embodiments of this application provide a training method, inference method, and apparatus for an AI model. By employing a multi-frequency domain mask reconstruction method, the model has a strong information extraction capability, thereby improving the prediction accuracy of the model.

[0004] In a first aspect, this application provides a method for training an AI model, comprising: acquiring a training dataset, the training dataset including multiple first time-series sample data; acquiring multiple first mask sequences, the multiple first mask sequences being obtained by performing multiple frequency domain masking operations on the first time-series sample data, wherein the frequency components masked by the multiple frequency domain masking operations on the frequency domain data corresponding to the time-series sample data are different; using the multiple first mask sequences as input to the AI ​​model to be trained, and outputting first reconstructed time-series data; and adjusting the weight parameters of the AI ​​model to be trained based on the first reconstructed time-series data and its corresponding first time-series sample data to obtain a pre-trained AI model.

[0005] The AI ​​model training method provided in this application disrupts the time-series sample data in the frequency domain by employing multiple frequency domain masking reconstruction. This forces the model to reconstruct the original time-series sample data based on the disrupted data, thereby giving the AI ​​model's backbone a strong information extraction capability and improving the prediction accuracy of downstream tasks. In other words, the frequency domain masking operation performed on the time-series sample data includes: first, converting the time-series sample data to the frequency domain; then, partially masking this data to obtain partially masked (disrupted) frequency domain data; and finally, converting the partially masked data back to the time domain to obtain time-series sample data disrupted in the frequency domain. The model then reconstructs the disrupted data (i.e., the sample data after the frequency domain masking operation), compares the reconstructed data with the original time-series sample data to determine the loss, and adjusts the model's parameters based on this loss. This allows the model to learn the features of the time-series sample data, improving its feature extraction capability.

[0006] In one possible implementation, obtaining multiple first mask sequences can be achieved by: converting the first time-series sample data to the frequency domain to obtain frequency domain data; performing a masking operation on the frequency domain data to obtain frequency domain mask data; converting the frequency domain mask data to the time domain to obtain a first mask sequence; and repeating the above operation multiple times on the first time-series sample data to obtain multiple first mask sequences.

[0007] By using multiple frequency domain masking operations, time series data is reconstructed from different information perspectives, enabling the backbone network of the AI ​​model to learn the pattern information of the time series data under different frequency components. This gives the backbone network of the AI ​​model a powerful information extraction capability, helps the AI ​​model achieve better prediction accuracy, and alleviates the problem of local cold start.

[0008] In another possible implementation, a specific implementation of performing a masking operation on the frequency domain data to obtain frequency domain masked data is as follows: generate a partitioning threshold, and divide the frequency domain data into high-frequency part data and low-frequency part data based on the partitioning threshold; perform a masking operation on the high-frequency part data or the low-frequency part data in the frequency domain data to obtain frequency domain masked data.

[0009] Optionally, the partitioning threshold is randomly generated, and / or the high-frequency or low-frequency data in the frequency domain data to be masked is randomly determined. In this way, by randomly generating the partitioning threshold, the frequency domain data is randomly divided into high-frequency and low-frequency data, and then the masking operation is randomly performed on the high-frequency or low-frequency data, thereby increasing the robustness of the backbone network of the AI ​​model.

[0010] In another possible implementation, the AI ​​model to be trained includes a first backbone network layer and a first reconstruction head. A specific implementation of using multiple first mask sequences as input to the AI ​​model to output reconstructed temporal data is as follows: inputting multiple first mask sequences into the first backbone network layer to obtain a first feature vector; inputting the first feature vector into the first reconstruction head to output first reconstructed temporal data. A loss value is calculated using the reconstructed temporal data and the input temporal sample data, and then the weight parameters of the first backbone network and the first reconstruction head are adjusted based on this loss value.

[0011] In another possible implementation, the backbone network layer includes an encoder layer; a specific implementation of inputting multiple first mask sequences into the first backbone network layer to obtain a first feature vector is as follows: the multiple first mask sequences are used as input to the encoder layer, and multiple encoded vectors are output; based on the multiple encoded vectors, the first feature vector is obtained.

[0012] By using multiple frequency domain mask reconstruction operations, the encoder learns the patterns under different frequency components of the time series data, giving it a powerful information extraction capability.

[0013] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are fused to obtain the first feature vector. By fusing the information of multiple encoding vectors, a first feature vector is obtained. Subsequently, a reconstructed time series data is obtained through this first feature vector. Then, the loss is calculated based on the reconstructed time series data and the input time series sample data to obtain a loss value. The weight parameters of the encoder and the reconstruction head are adjusted based on the loss value to increase training efficiency.

[0014] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are determined as multiple first encoding feature vectors, that is, the first encoding feature vectors include a set of encoding vectors. Subsequently, multiple reconstructed time series data are obtained through these multiple first feature vectors. Then, the loss is calculated based on these multiple reconstructed time series data and the input time series sample data to obtain multiple loss values. The weight parameters of the encoder and reconstruction head are adjusted based on these multiple loss values.

[0015] In another possible implementation, the training dataset includes first temporal sample data from multiple domains; the AI ​​model to be trained includes a first cue layer, a second backbone network layer, and a second reconstruction head; a specific implementation of using multiple first mask sequences as input to the AI ​​model to be trained and outputting reconstructed temporal data is as follows: the first temporal sample data is used as input to the first cue layer to obtain a first cue matrix, which represents the domain characteristics of the domain to which the first temporal sample data belongs; the first cue matrix is ​​fused with multiple first vectors to obtain multiple first fused vectors, which are either multiple vectors after the multiple first mask sequences have been processed by the second backbone network layer, or multiple vectors before the multiple first mask sequences have been processed by the second backbone network layer; based on the multiple first fused vectors, a second feature vector is obtained; the second feature vector is input to the second reconstruction head to output reconstructed temporal data.

[0016] The training method for the AI ​​model provided in this application uses multi-domain time series data for learning during the pre-training process, thereby learning the common features of time series between different domains. The most relevant feature information of the current domain can be obtained through the first cue matrix, further improving the performance of the AI ​​model.

[0017] In another possible implementation, the first prompt layer includes a first prompt information library and a first input layer. The prompt information library stores multiple prompt vectors, which represent domain features of multiple domains. A specific implementation of using the first time-series sample data as input to the first prompt layer to obtain the first prompt matrix is ​​as follows: the first time-series sample data is processed by the first input layer to obtain a first input vector; based on the first input vector, several prompt vectors are matched from the prompt information library to obtain several prompt vectors; based on the several prompt vectors, the first prompt matrix is ​​obtained.

[0018] By learning domain knowledge from time-series data across multiple domains during pre-training, a generalizable cue information library is learned. This cue library includes multiple cue vectors that represent domain features across multiple domains. When the AI ​​model makes predictions, it provides domain knowledge about the time-series data, helping the AI ​​model improve its prediction accuracy.

[0019] In another possible implementation, a specific implementation of adjusting the weight parameters of the AI ​​model to be trained based on the first reconstructed time-series data and its corresponding first time-series sample data is as follows: determine the first loss value based on the first reconstructed time-series data and its corresponding first time-series sample data; and adjust the weight parameters of the second reconstruction head, the weight parameters of the second backbone network, the weight parameters of multiple prompt vectors in the first prompt information library, and the weight parameters of the first input layer based on the first loss value.

[0020] In another possible implementation, the training dataset also includes multiple labeled second-time-series sample sequences; the AI ​​model to be trained also includes a prediction head, which is trained and optimized based on the multiple labeled second-time-series sample sequences. This provides few-shot learning or zero-shot learning capabilities, addressing the cold-start problem in scenarios with no samples.

[0021] In another possible implementation, the AI ​​model training method provided in this application further includes: acquiring a training dataset for a target business scenario, wherein the training dataset for the target business scenario includes multiple labeled third-time-series sample data under the target business scenario; and fine-tuning the pre-trained AI model based on the training dataset for the target business scenario to obtain a trained AI model. That is, after pre-training is completed, the pre-trained AI model is fine-tuned to ensure the prediction accuracy of the AI ​​model under the target business scenario.

[0022] In another possible implementation, the pre-trained AI model includes a second cue layer, a third cue layer, a third backbone network, a third reconstruction head, and a second prediction head. For the training dataset of the target business scenario, the pre-trained AI model is fine-tuned to obtain a trained AI model. One specific implementation involves performing multiple frequency domain masking operations on the third time-series sample data to obtain multiple second mask sequences. The second and third cue matrices are then fused to obtain the target cue matrix. The second cue matrix is ​​obtained based on the processing of the third time-series sample sequences by the second cue layer, representing the domain characteristics of the third time-series sample sequences. The third cue matrix is ​​obtained based on the third cue layer, representing the characteristics of the target business scenario. The target cue matrix is ​​then combined with multiple... The second vector is fused to obtain multiple second fused vectors. These multiple second vectors are either multiple vectors resulting from the processing of multiple second mask sequences by the third backbone network layer, or multiple vectors resulting from the processing of multiple second mask sequences by the third backbone network layer. Based on these multiple second fused vectors, a third feature vector is obtained. The third feature vector is used as input to the third reconstruction head and the second prediction head, respectively, to obtain second reconstruction time-series data and prediction data. Based on the third time-series sample sequence and its corresponding second reconstruction time-series data, and the prediction data and its corresponding labels for the second reconstruction time-series data, a second loss value and a third loss value are determined. Based on the second loss value and the third loss value, the weight parameters of the third prompting layer, the third backbone network, the third reconstruction head, and the second prediction head are adjusted.

[0023] In the fine-tuning stage, this application obtains a second prompt matrix driven by the currently input time-series sample data, and obtains specific prompt information (i.e., a third prompt matrix) related to the target business scenario based on efficient fine-tuning. The second and third prompt matrices are then fused to obtain a prompt matrix that simultaneously possesses domain generalization and domain-specific features. By enhancing the prompt matrix with specific features under the target business scenario, the prediction accuracy of the AI ​​model in downstream tasks is further improved.

[0024] In another possible implementation, the third cue layer includes a third cue matrix, which is updated and optimized during the fine-tuning of the pre-trained AI model based on a training dataset tailored to the target business scenario. In other words, the third cue matrix in the third cue layer is learned by the AI ​​model during the fine-tuning phase, acquiring domain-specific information specific to the target business scenario.

[0025] Secondly, this application also provides an inference method for an AI model, including acquiring time series data to be predicted; using the time series data to be predicted as input to the AI ​​model and outputting a prediction result, wherein the AI ​​model is trained based on the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect.

[0026] The AI ​​model trained using the training method provided in this application is used for inference and prediction. On the one hand, during the pre-training stage, multiple frequency domain masking operations enable the backbone network to learn the patterns of time-series data under different frequency components, giving the backbone network a strong information extraction capability. At the same time, the prompt learning method of this application is used to learn generalization features of different domains during the pre-training stage. On the other hand, during the fine-tuning stage, specific domain features of the target business scenario are learned. These specific domain features are used to enhance the prompt information learned during the pre-training stage, further improving the prediction accuracy of the AI ​​model in the target business scenario and enhancing the performance of the AI ​​model in the target business scenario.

[0027] Thirdly, this application also provides a training device for an AI model, including a first acquisition module, a frequency domain masking module, a reconstruction module, and a training module. The first acquisition module acquires a training dataset, which includes multiple first time-series sample data. The frequency domain masking module acquires multiple first mask sequences, obtained by performing multiple frequency domain masking operations on the first time-series sample data. The multiple frequency domain masking operations mask different frequency components of the corresponding frequency domain data of the time-series sample data. The reconstruction module uses the multiple first mask sequences as input to the AI ​​model to be trained and outputs first reconstructed time-series data. The training module adjusts the weight parameters of the AI ​​model to be trained based on the first reconstructed time-series data and its corresponding first time-series sample data to obtain a pre-trained AI model.

[0028] In one possible implementation, the frequency domain masking module is specifically used to: convert the first time-series sample data to the frequency domain to obtain frequency domain data; perform a masking operation on the frequency domain data to obtain frequency domain mask data; convert the frequency domain mask data to the time domain to obtain a first mask sequence; and repeat the above operation multiple times on the first time-series sample data to obtain multiple first mask sequences.

[0029] In another possible implementation, a specific implementation of performing a masking operation on the frequency domain data to obtain frequency domain masked data is as follows: generate a partitioning threshold, and divide the frequency domain data into high-frequency part data and low-frequency part data based on the partitioning threshold; perform a masking operation on the high-frequency part data or the low-frequency part data in the frequency domain data to obtain frequency domain masked data.

[0030] Optionally, the partitioning threshold is randomly generated, and / or the high-frequency or low-frequency data in the frequency domain data to be masked is randomly determined. In this way, by randomly generating the partitioning threshold, the frequency domain data is randomly divided into high-frequency and low-frequency data, and then the masking operation is randomly performed on the high-frequency or low-frequency data, thereby increasing the robustness of the backbone network of the AI ​​model.

[0031] In another possible implementation, the AI ​​model to be trained includes a first backbone network layer and a first reconstruction head. The reconstruction module is specifically used to: input multiple first mask sequences into the first backbone network layer to obtain a first feature vector; input the first feature vector into the first reconstruction head to output first reconstructed temporal data. A loss value is calculated using the reconstructed temporal data and the input temporal sample data, and then the weight parameters of the first backbone network and the first reconstruction head are adjusted based on this loss value.

[0032] In another possible implementation, the backbone network layer includes an encoder layer; a specific implementation of inputting multiple first mask sequences into the first backbone network layer to obtain a first feature vector is as follows: the multiple first mask sequences are used as input to the encoder layer, and multiple encoded vectors are output; based on the multiple encoded vectors, the first feature vector is obtained.

[0033] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are fused to obtain the first feature vector. By fusing the information of multiple encoding vectors, a first feature vector is obtained. Subsequently, a reconstructed time series data is obtained through this first feature vector. Then, the loss is calculated based on the reconstructed time series data and the input time series sample data to obtain a loss value. The weight parameters of the encoder and the reconstruction head are adjusted based on the loss value to increase training efficiency.

[0034] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are determined as multiple first encoding feature vectors, that is, the first encoding feature vectors include a set of encoding vectors. Subsequently, multiple reconstructed time series data are obtained through these multiple first feature vectors. Then, the loss is calculated based on these multiple reconstructed time series data and the input time series sample data to obtain multiple loss values. The weight parameters of the encoder and reconstruction head are adjusted based on these multiple loss values.

[0035] In another possible implementation, the training dataset includes first time-series sample data from multiple domains; the AI ​​model to be trained includes a first cue layer, a second backbone network layer, and a second reconstruction head; the reconstruction module is specifically used to: take the first time-series sample data as input to the first cue layer to obtain a first cue matrix, the first cue matrix representing the domain features of the domain to which the first time-series sample data belongs; fuse the first cue matrix with multiple first vectors to obtain multiple first fused vectors, the multiple first vectors being multiple vectors after multiple first mask sequences have been processed by the second backbone network layer, or the multiple first vectors being multiple vectors before multiple first mask sequences have been processed by the second backbone network layer; based on the multiple first fused vectors, obtain a second feature vector; input the second feature vector into the second reconstruction head, and output reconstructed time-series data.

[0036] In another possible implementation, the first prompt layer includes a first prompt information library and a first input layer. The prompt information library stores multiple prompt vectors, which represent domain features of multiple domains. A specific implementation of using the first time-series sample data as input to the first prompt layer to obtain the first prompt matrix is ​​as follows: the first time-series sample data is processed by the first input layer to obtain a first input vector; based on the first input vector, several prompt vectors are matched from the prompt information library to obtain several prompt vectors; based on the several prompt vectors, the first prompt matrix is ​​obtained.

[0037] In another possible implementation, the training dataset also includes multiple labeled second time-series sample sequences; the AI ​​model to be trained also includes a prediction head, which is trained and optimized based on the multiple labeled second time-series sample sequences.

[0038] In another possible implementation, the first acquisition module is also used to acquire a training dataset for the target business scenario, which includes multiple labeled third time-series sample data under the target business scenario; the training device for the AI ​​model provided in this application also includes a fine-tuning module, which is used to fine-tune the pre-trained AI model based on the training dataset for the target business scenario to obtain the trained AI model.

[0039] In another possible implementation, the pre-trained AI model includes a second cue layer, a third cue layer, a third backbone network, a third reconstruction head, and a second prediction head. The fine-tuning module is specifically used for: performing multiple frequency domain masking operations on the third time-series sample data to obtain multiple second mask sequences; fusing the second and third cue matrices to obtain a target cue matrix, where the second cue matrix is ​​obtained based on the processing of the third time-series sample sequences by the second cue layer, representing the domain characteristics of the third time-series sample sequences; and the third cue matrix, obtained based on the third cue layer, representing the characteristics of the target business scenario; and fusing the target cue matrix with multiple second vectors to obtain multiple second fused vectors. Multiple second vectors are either multiple vectors resulting from the processing of multiple second mask sequences by the third backbone network layer, or multiple vectors before the processing of multiple second mask sequences by the third backbone network layer. A third feature vector is obtained based on these multiple second fusion vectors. The third feature vector is used as input to the third reconstruction head and the second prediction head, respectively, to obtain second reconstruction time-series data and prediction data. A second loss value and a third loss value are determined based on the third time-series sample sequence and its corresponding second reconstruction time-series data, and the prediction data and its corresponding labels for the second reconstruction time-series data. The weight parameters of the third prompting layer, the third backbone network, the third reconstruction head, and the second prediction head are adjusted based on the second loss value and the third loss value.

[0040] In another possible implementation, the third cue layer includes a third cue matrix, which is updated and optimized during the fine-tuning of the pre-trained AI model based on a training dataset tailored to the target business scenario.

[0041] Fourthly, this application also provides an inference device for an AI model, including a second acquisition module and an inference module, wherein the second acquisition module is used to acquire time series data to be predicted; the inference module is used to take the time series data to be predicted as input to the AI ​​model and output a prediction result, wherein the AI ​​model is trained based on the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect.

[0042] Fifthly, embodiments of this application provide a computing device including a memory and a processor. The memory stores instructions that, when executed by the processor, cause the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect, and / or the inference method of the AI ​​model described in the second aspect to be implemented.

[0043] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect, and / or the inference method of the AI ​​model in the second aspect to be implemented.

[0044] In a seventh aspect, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect, and / or the inference method of the AI ​​model in the second aspect.

[0045] Eighthly, embodiments of this application also provide a chip including at least one processor and a communication interface, the processor being used to execute the training method of the AI ​​model described in the first aspect or any possible implementation of the first aspect, and / or the inference method of the AI ​​model in the second aspect. Attached Figure Description

[0046] Figure 1 This diagram illustrates a basic framework for artificial intelligence.

[0047] Figure 2 A system architecture diagram of the sample processing system provided in this application embodiment;

[0048] Figure 3 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application;

[0049] Figure 4 A flowchart illustrating a training method for an AI model provided in an embodiment of this application;

[0050] Figure 5 A schematic diagram illustrating the implementation of the frequency domain masking method provided in an embodiment of this application is shown;

[0051] Figure 6 This illustration shows a schematic diagram of how an AI model processes input time-series sample data in a pre-training state, according to an embodiment of this application.

[0052] Figure 7 This illustration shows another AI model from an embodiment of this application processing input time-series sample data in a pre-training state;

[0053] Figure 8 This illustration shows a schematic diagram of how an AI model processes input time-series sample data in a pre-training state, according to an embodiment of this application.

[0054] Figure 9 This illustration shows a schematic diagram of the processing of input time-series sample data A by an AI model in the prediction state according to an embodiment of this application;

[0055] Figure 10 This illustration shows a schematic diagram of the AI ​​model trained according to an embodiment of this application, illustrating the process of processing the input time-series data during the prediction phase.

[0056] Figure 11 This illustration shows a schematic diagram of how an AI model processes input time-series sample data in a pre-training state, according to an embodiment of this application.

[0057] Figure 12 The illustration shows a schematic diagram of the implementation of the prompting learning method provided in the embodiment of this application during the pre-training stage;

[0058] Figure 13 This illustration shows a schematic diagram of the AI ​​model's processing of input time-series data during the inference phase after training, as provided in an embodiment of this application.

[0059] Figure 14 This illustration shows a schematic diagram of the implementation of cue learning in the fine-tuning stage provided in an embodiment of this application;

[0060] Figure 15 A flowchart illustrating the inference method of the AI ​​model provided in this application embodiment;

[0061] Figure 16 A schematic diagram of the structure of an AI model training device provided in an embodiment of this application;

[0062] Figure 17 This is a schematic diagram of the structure of an AI model inference device provided in an embodiment of this application;

[0063] Figure 18 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0064] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0065] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such processes, methods, systems, products, or apparatus.

[0066] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0067] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0068] First, the overall workflow of the artificial intelligence system will be described. Please refer to [link / reference]. Figure 1 , Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0069] The above-mentioned artificial intelligence framework is elaborated below from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technical means), to the industrial ecosystem of the system. From the perspective of the IT value chain, the main framework of artificial intelligence includes an infrastructure layer, a data layer, a data processing layer, a general capability layer (achieving general capabilities through data processing results, such as translation, text analysis, computer vision processing, speech recognition, and image recognition), and the topmost application layer, which encapsulates the overall artificial intelligence solution, productizing intelligent information decision-making and realizing its practical application.

[0070] (1) Infrastructure:

[0071] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the external world is achieved through sensors; computing power is provided by intelligent chips, including but not limited to hardware acceleration chips such as central processing units (CPUs), neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). The basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0072] (2) Data

[0073] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0074] (3) Data processing

[0075] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0076] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0077] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0078] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0079] (4) General ability

[0080] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0081] (5) Smart Products and Industry Applications

[0082] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0083] The AI ​​model training method provided in this application can be applied to the training of AI models. The training of AI models includes a pre-training stage and a fine-tuning stage. In the pre-training stage, through multiple frequency domain mask reconstruction operations, the backbone network of the AI ​​model learns the patterns under different frequency components of time-series data, enabling the backbone network to have a strong information extraction capability. At the same time, through a learnable cue information base, the generalization features of different domains are learned in the pre-training stage. In the fine-tuning stage, a learnable cue matrix for the training dataset under the target business scenario is set, and the knowledge related to the target business scenario is learned. This cue matrix is ​​then fused with the cue matrix learned in the pre-training stage to obtain a cue matrix that has both domain generalization knowledge and domain-specific knowledge, thereby improving the prediction accuracy of the AI ​​model in downstream tasks.

[0084] The AI ​​model training method provided in this application embodiment can be applied to train various types of AI models, improve the training effect of AI models, and thus improve the prediction accuracy of the trained AI models.

[0085] For example, the training method of the AI ​​model provided in this application embodiment can train the detection type AI model based on the training dataset under the detection task. The trained AI model can be applied to the detection task of time series data. For example, the AI ​​model can detect whether the business is abnormal based on the time series data of the business.

[0086] For example, the training method of the AI ​​model provided in this application embodiment can train the AI ​​model of the prediction type based on the training dataset under the prediction task. The trained AI model can be applied to the prediction task of time series data. For example, the AI ​​model can predict future time series data based on the generated time series data.

[0087] For example, the training method of the AI ​​model provided in this application embodiment can train the AI ​​model of the classification type based on the training dataset under the classification task. The trained AI model can be applied to the classification task of time series data. For example, the AI ​​model can classify the operation status of the business according to the time series data of the business.

[0088] In other words, the training method for the AI ​​model provided in this application embodiment can be used to train any type of AI model. The trained AI model can perform corresponding tasks, such as detection, classification and prediction of time series data.

[0089] When AI models are applied to prediction tasks based on time-series data, they can include, but are not limited to, prediction tasks such as photovoltaic power generation forecasting, load forecasting, weather forecasting, wind power generation forecasting, network traffic forecasting, and commodity sales volume forecasting.

[0090] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 2 The sample processing system provided in the embodiments of this application will be described below, see appendix. Figure 2 The data acquisition device 260 is used to collect training data. In this embodiment of the application, the training data is the data used when distilling the student model.

[0091] After collecting the training data, the data acquisition device 260 stores the training data in the database 230. The training device 220 performs knowledge distillation on the student model based on the teacher model based on the training data maintained in the database 230 to obtain the target model / rule 201.

[0092] In practical applications, the training data maintained in database 230 may not all come from the data acquisition device 260; it may also be received from other devices. Furthermore, it should be noted that training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained in database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0093] The target model / rule 201 trained using training device 220 can be applied to different systems or devices, such as... Figure 2 In the execution device 210, the execution device 210 can be a terminal, such as a mobile phone, tablet computer, laptop computer, augmented reality (AR), virtual reality (VR), wearable device, smart robot, vehicle terminal, etc., or it can be a server or cloud, etc.

[0094] Specifically, the training device 220 can deploy the trained AI model to the execution device 210.

[0095] exist Figure 2 In this embodiment, the execution device 210 configures the input / output (I / O) interface 212 for exchanging data with external devices. The user can input data to the I / O interface 212 through the client device 240. In this embodiment, the data may include timing data input by the user.

[0096] Preprocessing modules 213 and 214 are used to preprocess the input data (such as timing data input by the user) received from the I / O interface 212. It should be understood that preprocessing modules 213 and 214 may be absent, or only one preprocessing module may be used. When preprocessing modules 213 and 214 are absent, the calculation module 211 can directly process the input data.

[0097] The calculation module 211 is used to perform calculations and other related processing on the data input from the preprocessing module 213 and the preprocessing module 214 according to the target model / rule 201 mentioned above.

[0098] During the preprocessing of input data by the execution device 210, or during the calculation and related processing of the calculation module 211 of the execution device 210, the execution device 210 can call the data, code, etc. of the database storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0099] Finally, I / O interface 212 returns the processing results (such as time-series data for a future period) to client device 240 for the user. It should be understood that the target model / rule 201 is different for different time-series data processing tasks (including classification, detection, and prediction tasks), and the processing results are correspondingly different as well.

[0100] It is worth noting that the training device 220 can generate a target model / rule 201 corresponding to different downstream systems. This target model / rule 201 can achieve the above-mentioned goals or complete the above-mentioned tasks, thereby providing the user with the required results. It should also be noted that the training device 220 can generate corresponding preprocessing models for the target model / rule 201 corresponding to different downstream systems, such as the preprocessing models corresponding to preprocessing modules 213 and / or 214.

[0101] exist Figure 2 In the scenario shown, the user can manually specify the data to be input into the execution device 210 (e.g., input the time series data to be predicted), for example, by operating through the interface provided by the I / O interface 212. Alternatively, the client device 240 can automatically input data into the I / O interface 212 (e.g., input the time series data to be predicted) and obtain the results. If the client device 240 requires user authorization to automatically input data, the user can set the corresponding permissions in the client device 240. The user can view the results output by the execution device 210 on the client device 240 (e.g., output time series data for a future period), which can be presented in various forms such as display, sound, or animation. The client device 240 can also act as a data acquisition terminal, collecting data such as... Figure 2 The input data (time series data) of the input I / O interface 212 and the output results of the target model / rule 201 are stored in the database 230 as new sample data.

[0102] It is worth noting that Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application, and the positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0103] Figure 3 This is a schematic diagram of the hardware structure of a chip provided in an embodiment of this application. The chip includes a neural network processor (NPU) 30. This chip can be configured as follows: Figure 2In the execution device 210 shown, the chip is used to perform the calculations of the calculation module 211. This chip can also be located in, for example... Figure 2 The training device 220 shown is used to complete the training work of the training device 220 and output the target model / rule 201.

[0104] The training method of the AI ​​model in this application embodiment can be executed in the arithmetic circuit 303 and / or vector calculation unit 307 in the neural network processor 30, thereby obtaining an AI model with high prediction accuracy.

[0105] The following is a brief introduction to the various modules and units in the neural network processor 30.

[0106] The neural network processor 30 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor 30 is the arithmetic circuit 303. When the neural network processor 30 is working, the controller 304 in the neural network processor 30 can control the arithmetic circuit 303 to retrieve matrix data from the memory and perform multiplication operations.

[0107] In some implementations, the arithmetic circuit 303 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0108] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0109] The vector computation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 307 can be used for network computation in non-convolutional / non-fully connected layers (FC) of neural networks, such as pooling, batch normalization, and local response normalization.

[0110] In some implementations, vector computation unit 307 can store the processed output vector into unified buffer 306. For example, vector computation unit 307 can apply a nonlinear function to the output of arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0111] The unified memory 306 is used to store input data and output data.

[0112] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the Direct Memory Access Controller (DMAC) 305, the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0113] BIU stands for Bus Interface Unit 310, which is used to enable interaction between the main CPU, DMAC and instruction fetch memory 309 via a bus.

[0114] The instruction fetch buffer 309, which is connected to the controller 304, is used to store the instructions used by the controller 304.

[0115] The controller 304 is used to call the instructions cached in the instruction memory 309 to control the operation of the computing accelerator.

[0116] Generally, the unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 can all be on-chip memories. The external memory of the NPU can be memory located outside the NPU itself, such as double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory. It should be understood that... Figure 3 The chip hardware structure shown is merely illustrative and is not intended to limit the scope of this application.

[0117] To facilitate understanding of the solutions in the embodiments of this application, the technical terms involved in this document will be explained first.

[0118] In training a neural network, the loss function is crucial because we want the network's output to be as close as possible to the desired predicted value. This is achieved by comparing the network's current prediction with the target value and updating the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, where parameters are pre-configured for each layer). For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network can predict the target value or a value very close to it. Therefore, it's necessary to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the neural network becomes a process of minimizing this loss.

[0119] Masking reconstruction (reconstruction): This includes masking operations and reconstruction operations. Masking operations refer to the destruction of the original sequence data. This destruction is generally manifested in covering a certain proportion of the original sequence. In the time dimension, the masked part can be assigned a value of 0, or the masked part can be turned into a learnable mask token. In the frequency domain dimension, certain frequency components can be masked. Reconstruction operations refer to the process of encoding the original sequence through a neural network and then decoding it back to the original sequence. Generally, the original sequence can be destroyed and then restored, or it can be restored directly without destruction. Masking reconstruction specifically refers to the reconstruction after destroying the original sequence, in order to enable the encoder to learn complete representation information.

[0120] Pre-training: Generally refers to the process of pre-training a model with massive amounts of multi-domain data before performing downstream tasks, thereby obtaining a basic model that has learned the corresponding knowledge.

[0121] Fine-tuning refers to adjusting the model parameters using a certain amount of current task scenario data for a specified downstream task scenario, in order to enable the model to learn certain information contained in the current task and thus improve the accuracy of the current task. There are many methods for adjusting model parameters, which will not be listed one by one.

[0122] Zero-shot learning refers to using a pre-trained model directly without needing to fine-tune it using current scenario data, thereby enabling rapid business deployment.

[0123] Hint learning: Inputting certain auxiliary data helps the model acquire more information and further improve its ability to solve the current task. This concept is generalized from the field of natural language processing (NLP). Based on current mainstream views, it can be classified according to interpretability into hard prompts and soft prompts. Hard prompts tend to use discrete templates or task text descriptions (tokens can be matched one-to-one in the vocabulary). Soft prompts are composed of continuous, uninterpretable tokens. They are updated more during fine-tuning without updating the pre-trained model parameters, enabling the model to learn professional knowledge of the current task domain and enhance its ability to solve downstream tasks.

[0124] In the relevant time series forecasting task solutions, a number of modeling strategies have emerged that optimize model architecture or learning methods. However, they are essentially still based on the case-by-case modeling method, that is, for each task scenario, the model is trained based on the data of the current task scenario to obtain a prediction model that is strongly related to the current business scenario.

[0125] For example, the first related solution, which can be called the FEDFormer solution, is similar to the current mainstream Form-Based models. FEDFormer relies on the Attention mechanism to capture point-wise relationships and utilizes the time-domain to frequency-domain transformation to convert enhancement and Attention operations to the frequency domain. This not only results in good prediction performance but also significantly reduces the model's time complexity. The main structure of FEDFormer adopts an encoder-decoder structure, with the number of encoders and decoders being variable and can be customized as needed. Internally, it consists of four sub-modules:

[0126] The frequency-enhanced block (FEB) transforms information learned in the time domain into information learned in the frequency domain. The main idea is that information in the time domain is redundant, while in the frequency domain, a few frequency components can effectively describe the current time series. Therefore, transforming the learning of the time series in the time domain into the learning of a few frequency components in the frequency domain allows for the effective capture of relevant and useful information. Its main process is as follows: projection from time domain to frequency domain --> sampling of frequency components --> learning in the frequency domain --> frequency domain completion --> projection back to the time domain.

[0127] Frequency Enhanced Attention (FEA): The core idea of ​​this module is similar to that of the FEB module. Its main process is still: projection from the time domain to the frequency domain --> frequency component sampling --> in-frequency learning --> frequency domain completion --> projection back to the time domain. The main difference between FEA and FEB lies in the input and in-frequency learning parts. The FEB module only receives information from one of the original input / encoder / decoder as input for learning, while the FEA module receives information from both the encoder and decoder as input for learning. In the in-frequency learning part, the FEA module performs attention operations in the frequency domain, rather than the simple linear operations of FEB.

[0128] The core of the MOE Decomp module is to decompose the input signal into a seasonal component and a trend component. In the encoder, the trend component is discarded, the seasonal component is passed to the next layer for learning, and finally passed to the decoder. In the decoder, the trend component is retained, and after multiple layers are added together, it is applied to the output result.

[0129] Feed Forward: This module learns features at the feature level. Simply put, it can be understood as a multi-layer perception (MLP) layer, which fuses information to generate new signals.

[0130] FEDFormer performs random sampling in the frequency domain. This operation significantly reduces the length of the input vector, thereby lowering computational complexity. Furthermore, the above discussion is based on Fourier transform, and wavelet transform possesses similar properties. Therefore, converting the Fourier transform to a wavelet transform can be considered a variant of FEDFormer. Fourier bases are global, while wavelet bases are local.

[0131] This scheme performs random sampling in the frequency domain in the hope of reducing computational complexity by reducing the length of the input vector. However, this sampling is inevitably lossy to the input information. Although a trade-off can be made between computational performance and final model accuracy through the sampling rate, this trade-off still requires manual intervention to optimize hyperparameters. Another drawback of this scheme is that, due to the certain complexity of the model, it requires full downstream task data to drive case-by-case modeling and learning as the basis for achieving the corresponding accuracy target.

[0132] The second related solution, which can be called TTMs (tiny time mixers), is based on the idea of ​​pre-training on massive amounts of multi-domain time series data, and then fine-tuning the model based on specific downstream tasks, so as to achieve better prediction performance while starting with a small number of samples.

[0133] The modeling core of TTMs is an encoder-only architecture, with the network architecture mainly composed of MLPs. The pre-training stage mainly consists of TTM Backbone and TTM Decoder. In the fine-tuning stage, an Optional ExogenousMixer module can be added. Its main process is as follows:

[0134] Data preprocessing: For datasets collected from multiple domains, two problems exist: first, each dataset has a different resolution (i.e., inconsistent sampling frequency); second, for multivariate time series datasets, each dataset may have different dimensions. To address these problems, for the first issue, high-resolution data is downsampled to balance the proportion of datasets with different resolutions as much as possible. For the second issue, channel-independent modeling is performed, that is, each multivariate time series dataset is decomposed into single-dimensional time series for modeling.

[0135] Pre-training process: During pre-training, no information exchange occurs between channels, so single-channel modeling is performed. First, the raw data is standardized and patched. Then, for the current data resolution, corresponding resolution prompts (soft prompts, which can be learned and are optional) are obtained. After concatenation, the data enters the TTM Backbone module, which performs information exchange learning. This module consists of L layers of TTM Blocks, each containing M layers of TSMixer Blocks. The TTM Backbone recombines the patch length in multiple levels, based on multi-scale modeling, which is more conducive to extracting time dimension information at different resolutions. At the same time, each Mixer Block performs inter-patch and intra-patch information exchange to enhance information extraction capabilities. After completing the learning in the TTM Backbone module, the TTM Decoder module is used. The model architecture of this module is the same as that of the TTM Backbone, but with fewer layers, resulting in smaller module parameters. After information updates, the data enters the Forecast Linear Head module and undergoes de-standardization to complete the prediction training task.

[0136] Downstream task fine-tuning process: After acquiring the current downstream task data, the model is fine-tuned based on a certain amount of data. During this process, the process remains consistent with pre-training, but the model parameters of the TTM Backbone module are fixed and not adjusted, while the TTM Decoder module performs parameter fine-tuning. At the same time, according to the needs of downstream business, an Inter channelmixer information interaction module can be added to this module. This module performs information interaction between multiple channels to enhance the prediction capability of downstream tasks. Additionally, according to business needs, an Optional Exogenous Mixer module can be inserted between the Forecast Linear head and the data inverse standard version module. This module can insert covariate information for the time point to be predicted, which can further enhance prediction performance.

[0137] This solution is based on a pure time-series modeling method. Although it has established a learning paradigm of pre-training + fine-tuning, its information extraction capability is insufficient and it does not show excellent performance, so its business application value is not great.

[0138] In summary, in the first related scheme, frequency domain modeling offers another perspective on time series data. However, its drawback lies in the fact that it simply performs a time-frequency domain conversion, reuses the frequency domain information using time domain modeling methods, and then converts it back to the time domain. This modeling approach still relies on training with the entire dataset and cannot address the issue of rapid site startup under conditions of few or no samples. While the second related scheme integrates the pre-training + fine-tuning paradigm, its information extraction method is relatively simple. From the perspective of time domain modeling alone, it does not perform comprehensive information extraction effectively, resulting in insufficient prediction performance to support the rapid startup of downstream tasks.

[0139] To address the aforementioned issues, this application proposes a training method and apparatus for an AI model. This AI model can be applied to time-series data processing tasks (such as classification, detection, and prediction). By employing a multi-frequency domain mask reconstruction method, the model possesses strong information extraction capabilities, thereby improving the prediction accuracy of the AI ​​model.

[0140] Taking the training of an AI model for a prediction task of time series data as an example, this application introduces the training method and apparatus of the AI ​​model provided in the embodiments. The training of AI models for other time series data processing tasks (such as classification tasks or detection tasks) is similar, with the only difference being the different training datasets. For specific implementation, refer to the training process of the AI ​​model for a prediction task of processing time series data. For the sake of brevity, the embodiments of this application will not be described in detail.

[0141] The training method, inference method, and apparatus of the AI ​​model provided in the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0142] Figure 4 This is a flowchart illustrating a training method for an AI model provided in an embodiment of this application. The method can be... Figure 2 The training device 220 in the system implements multi-frequency domain mask reconstruction to improve the information extraction capability of the AI ​​model, thereby improving the prediction accuracy of the AI ​​model. For example... Figure 4 As shown, the training method for the AI ​​model provided in this application embodiment includes at least steps S401 to S404.

[0143] In step S401, a training dataset is obtained, which includes multiple time-series sample data.

[0144] The training device retrieves a training dataset from the database, which includes multiple time-series sample data. Optionally, these multiple time-series sample data can encompass multiple domains, such as energy, transportation, and natural and social data.

[0145] Optionally, the training dataset can be a preprocessed version of a collected dataset. For example, using the open-source dataset Monash, which covers multiple fields such as energy, transportation, nature, and society, the dataset can be split into one-dimensional data to address the different data dimensions across different fields. Furthermore, since the sampling frequencies of datasets from different fields may vary, partial downsampling can be used to enhance the overall distribution of the dataset. For instance, for time-series data within the same field, the data can be segmented according to a certain length to obtain multiple time-series sample data of length L. For example, segmenting the data according to 512 time points yields multiple time-series sample data of length 512, thus obtaining multiple time-series sample data.

[0146] In step S402, multiple mask sequences are obtained, which are obtained by performing multiple frequency domain masking operations on the time-series sample data.

[0147] After obtaining a training dataset containing multiple time-series samples, multiple frequency domain masking operations are performed on any one time-series sample in the training dataset. Each frequency domain masking operation masks different frequency components of the corresponding frequency domain data. Taking the multiple frequency domain masking operations on time-series sample data A as an example, first, a time-frequency conversion algorithm is used to convert time-series sample data A from the time domain to the frequency domain, obtaining frequency domain data a. Then, a masking operation is performed on frequency domain data a to obtain frequency domain masked data a. `(For example, a mask operation is performed on a portion of the frequency components in the frequency domain data a to obtain the frequency domain data with the masked frequency components.) The frequency domain mask data is converted to the time domain using a time-frequency conversion algorithm to obtain the mask sequence A'. The above operation is repeated multiple times on the time-series sample data A to obtain multiple mask sequences A'.

[0148] Alternatively, a frequency domain masking operation can be performed only once on the time-series sample data to obtain a mask sequence, thus reducing the number of masking operations performed during training and reducing the computational cost of training.

[0149] Multiple frequency domain masking operations can also be performed on time series sample data to obtain multiple mask sequences, which can be used to reconstruct time series sample data from different information perspectives, learn the patterns of different frequencies in the time series, and enable the backbone network of the AI ​​model to have a powerful information extraction capability.

[0150] It should be noted that the embodiments of this application do not impose specific limitations on the time-frequency conversion algorithm used to convert time-series sample data from the time domain to the frequency domain. Any suitable time-frequency conversion algorithm can be used, such as Fourier transform, selected discrete cosine transform, wavelet transform, etc.

[0151] In another example, to increase the robustness of the trained AI model, embodiments of this application provide a frequency domain masking method, which can further increase the robustness of the trained AI model.

[0152] Figure 5 A schematic diagram illustrating the implementation of the frequency domain masking method provided in an embodiment of this application is shown. Figure 5 As shown, firstly, time-frequency conversion is used to transform the time-series sample data A from the time domain to the frequency domain, obtaining frequency domain data a. Then, a threshold is randomly generated, and this threshold is used to divide the frequency domain data a into high-frequency and low-frequency components. Next, either the high-frequency or low-frequency component is randomly selected for masking, and a masking operation is performed on the high-frequency or low-frequency component. Based on the currently unmasked high-frequency or low-frequency component, time-frequency conversion is used to restore the time domain from the frequency domain. The resulting time series is the mask sequence A', which represents a mask destruction operation performed at the frequency domain level. This process is repeated multiple times to obtain multiple sequences of mask destruction operations, i.e., Figure 5 The time-series data Mask 1, Mask 2, Mask 3…Mask N are used to obtain multiple mask sequences A'. The frequency domain data corresponding to each mask sequence A' is different. Here, the meaning of the frequency domain data corresponding to the mask sequence is that after converting the time-series sample data A from the time domain to the frequency domain, a portion of the frequency domain data is masked, resulting in the unmasked frequency domain data, such as… Figure 5The mask sequence 1, mask sequence 2, mask sequence 3... mask sequence N represents the unmasked frequency domain data before time-frequency conversion. Figure 5 It can be seen that because the frequency domain data obtained by converting time series sample data is masked in different ways, the unmasked frequency domain data is also different. As a result, the frequency domain data corresponding to each mask sequence in the mask sequence 1, mask sequence 2, mask sequence 3... mask sequence N are different. This is beneficial to reconstruct the time series data from different information perspectives in combination with subsequent data reconstruction. This allows the backbone network of the AI ​​model to learn the pattern information of the time series data under different frequency components, giving the backbone network of the AI ​​model a powerful information extraction capability.

[0153] It should be noted that, Figure 5 The illustrated method is merely one feasible frequency domain masking method and does not constitute a limitation on the embodiments of this application. For example, when performing a masking operation on frequency domain data a, the masking operation can be performed on a randomly determined high-frequency part or low-frequency part in a certain proportion. For instance, the high-frequency part of frequency domain data a can be randomly selected for masking operation, and half of the high-frequency part of frequency domain data a can be masked to obtain the masked frequency domain data a. ` That is, only half of the high-frequency part of the mask is masked, instead of... Figure 5 The high-frequency portion of the code is masked.

[0154] In step S403, multiple mask sequences are used as input to the AI ​​model to be trained, and reconstructed time series data is output.

[0155] Multiple mask sequences are used as input to the AI ​​model to be trained, and the reconstructed time series data is obtained after processing by each neural network layer in the AI ​​model to be trained.

[0156] Taking the AI ​​model to be trained as a transformer architecture as an example, this paper introduces the process of the AI ​​model processing the input time-series sample data in the pre-training state.

[0157] Figure 6 This illustration shows a schematic diagram of how an AI model, according to an embodiment of this application, processes input time-series sample data in a pre-training state. For example... Figure 6 As shown, the pre-trained AI model includes a patchify and projection layer, an encoder layer, and a decoder layer. Temporal data A undergoes multiple frequency domain masking operations to obtain multiple mask sequences A'. These mask sequences A' are then processed by the patchify and projection layer, several stacked encoders in the encoder layer, and several stacked decoders in the decoder layer, resulting in multiple reconstructed temporal data sets. A .

[0158] Figure 7This illustration shows another AI model from an embodiment of this application processing input time-series sample data in a pre-training state. For example... Figure 7 As shown, the pre-trained AI model includes a patchify and projection layer, an encoder layer, a decoder layer, and a reconstruction head. Temporal data A undergoes multiple frequency domain masking operations to obtain multiple mask sequences A'. These mask sequences A' are then processed by the patchify and projection layer, several stacked encoders in the encoder layer, several stacked decoders in the decoder layer, and the reconstruction head, respectively, to obtain multiple reconstructed temporal data. A .

[0159] In another example, to increase the training efficiency of the AI ​​model, the encoding vectors of multiple mask sequences are fused during the pre-training process of the AI ​​model, and then a reconstructed time series data is output through the processing of the decoder and / or reconstruction head.

[0160] Figure 8 This illustration shows a schematic diagram of how an AI model, according to an embodiment of this application, processes input time-series sample data in a pre-training state. For example... Figure 8 As shown, multiple mask sequences are processed by the encoder layer, outputting multiple encoded vectors. These encoded vectors are then used as input to the aggregator layer, which fuses the encoder vectors. The fused encoded vector is then input to the decoder for decoding, yielding a decoded vector. This decoded vector is then processed by the reconstruction head to output a single piece of reconstructed timing data. A .

[0161] In step S404, the weight parameters of the AI ​​model to be trained are adjusted based on the reconstructed time series data and its corresponding time series sample data to obtain the pre-trained AI model.

[0162] The pre-trained AI model obtains reconstructed time-series sample data by inferring and predicting the masked data after multiple frequency domain masking operations. Then, the loss value of the loss function can be calculated based on the reconstructed time-series sample data and the input time-series sample data. Based on this loss value, the parameters of the AI ​​model are adjusted and iterated repeatedly until the conditions for pre-training completion are met (such as reaching the set number of training iterations or training convergence). The pre-trained AI model is then obtained.

[0163] against Figure 6 or Figure 7The pre-training scheme shown, which involves multiple mask sequences without fusion processing, performs multiple frequency domain masking operations on any time-series sample data in the training dataset. Taking time-series sample data A as an example, multiple mask sequences are obtained by processing the multiple mask sequences through the AI ​​model, resulting in multiple reconstructed time-series data. A Multiple reconstruction time series data A The loss values ​​are calculated separately for each time series sample data A to obtain multiple loss values. Then, the AI ​​model is tuned based on these multiple loss values, that is, the weight parameters of the patchify & projection layer, encoder layer, decoder layer and reconstruction head of the AI ​​model are adjusted.

[0164] against Figure 8 The pre-training scheme shown is a scheme that requires fusion processing after encoding multiple mask sequences. This scheme involves performing multiple frequency domain masking operations on the time-series sample data A to obtain multiple mask sequences A'. These multiple mask sequences A' are then processed by an AI model to output a reconstructed time-series data. A Reconstructing time series data A The loss value is calculated using the time-series sample data A. Then, the AI ​​model is tuned based on this loss value, specifically by adjusting the weight parameters of the patchify & projection layer, encoder layer, fusion layer, decoder layer, and reconstruction head of the AI ​​model.

[0165] In another example, the pre-trained AI model in this embodiment of the application also includes a prediction head, which provides the AI ​​model with zero-shot learning capability through pre-training of the prediction head. The prediction head learns the mapping from the representation of historical input data to future data.

[0166] At this point, the training dataset also includes labeled time-series sample data, the labels of which are related to the downstream task type of the AI ​​model; for example, when the downstream task of the AI ​​model is a classification task, the label of the time-series sample data is the true category; when the downstream task of the AI ​​model is a detection task, the label of the time-series sample data is the anomaly label; when the downstream task of the AI ​​model is a prediction task, the label of the time-series sample data is the business data value for a future period of time.

[0167] Figure 9 This illustration shows a schematic diagram of the processing of input time-series sample data A by an AI model in the prediction state, according to an embodiment of this application. Figure 9As shown, taking labeled time-series sample data B as an example, multiple frequency domain masking operations are performed on the time-series sample data B to obtain multiple mask sequences B'. Then, the multiple mask sequences B' are processed by the patchify & projection layer to obtain multiple representation vectors. The multiple representation vectors are encoded by the encoder layer to obtain multiple encoded vectors. The multiple encoded vectors are fused to obtain a fused encoded vector. Figure 9 (Not shown in the diagram; of course, in some other examples, fusion processing can be omitted, and multiple encoded vectors can be input into subsequent neural network layers for processing separately.) Finally, the fused encoded vectors are processed by the reconstruction encoder layer and reconstruction head to obtain the reconstructed temporal data. A The predicted temporal data B is obtained through processing with the prediction decoder layer and prediction head. Based on the reconstructed temporal data... B The loss is calculated based on the time-series sample data B to obtain loss value 1. The loss is calculated based on the predicted time-series data B and the label to obtain loss value 2. The AI ​​model in the pre-trained state is tuned based on loss value 1 and loss value 2, that is, the weight parameters of the patchify & projection layer, encoder layer, fusion layer, reconstruction decoder layer, reconstruction head and prediction head of the AI ​​model are adjusted.

[0168] Depend on Figure 9 It can be seen that the weight parameters of the prediction decoder layer are the same as those of the reconstruction decoder layer, that is, the prediction decoder layer and the reconstruction decoder layer share the same parameters. During pre-training, only the weight parameters of the reconstruction decoder layer need to be adjusted, which increases training efficiency.

[0169] Of course, in some other examples, the weight parameters of the predictor decoder layer can also be independent and participate in the pre-training of the AI ​​model, that is, the parameters will also be adjusted during the pre-training process of the AI ​​model.

[0170] To further improve the performance of the AI ​​model in downstream tasks, after the AI ​​model is pre-trained, it will be fine-tuned using some business data from the downstream tasks.

[0171] Fine-tuning an AI model is similar to the pre-training process. For example, first, business data from downstream task scenarios is acquired, and this data is preprocessed to obtain multiple labeled time-series sample data. Then, any one of these labeled time-series sample data is fine-tuned... Figure 9The processing is illustrated below. Taking time-series sample data C as an example, multiple frequency domain masking operations are performed on the time-series sample data C to obtain multiple mask sequences C'. Then, the multiple mask sequences C' are processed by the patchify & projection layer to obtain multiple representation vectors. The multiple representation vectors are encoded by the encoder layer to obtain multiple encoded vectors. The multiple encoded vectors are fused to obtain a fused encoded vector. Finally, the fused encoded vector is processed by the reconstruction encoder layer + reconstruction head to obtain the reconstructed time-series data. C The predicted temporal data C is obtained through processing the prediction decoder layer and prediction head. Based on the reconstructed temporal data... C The loss is calculated using the time-series sample data C, resulting in loss value 3. The loss is then calculated using the predicted time-series data C and the labels, resulting in loss value 4. Based on loss values ​​3 and 4, the AI ​​model in the fine-tuned state is tuned, specifically by adjusting the weight parameters of the patchify & projection layer, encoder layer, fusion layer, reconstruction decoder layer, reconstruction head, and prediction head of the AI ​​model.

[0172] It is understandable that during the prediction phase of the AI ​​model, there is no need to perform multiple frequency domain mask reconstruction operations (i.e., including multiple frequency domain mask operations, as well as subsequent processing and reconstruction of the mask data by the encoder layer, decoder layer, and reconstruction head to obtain reconstructed time-series data). In other words, the AI ​​model in the prediction state does not have a reconstruction decoder and reconstruction head structure.

[0173] Figure 10 This diagram illustrates the processing of input time-series data by the AI ​​model trained according to an embodiment of this application during the prediction phase. Figure 10 As shown, the predictive AI model includes a segmentation and mapping layer, an encoder layer, a decoder layer, and a prediction head. The time-series data generated by specific business processes in downstream tasks is used as input to the AI ​​model. The time-series data is first processed by the segmentation and mapping layer to output a representation vector. This representation vector is then input to the encoder layer to obtain an encoded vector. The decoder layer decodes this encoded vector to obtain a decoded vector. Finally, this decoded vector is used as input to the prediction head to output predicted time-series data. This predicted time-series data indicates the business data values ​​generated by the specific business process in the future.

[0174] In this embodiment, the encoder learns the patterns of different frequency components of the time-series data through multiple frequency domain mask reconstruction operations, thereby enabling the encoder to have a powerful information extraction capability and thus achieve good prediction accuracy of the trained AI model.

[0175] In some other embodiments, this application also provides a prompt learning method to perform prompt learning during the pre-training stage, further improving the prediction accuracy of the AI ​​model. Specifically, during the pre-training process, multi-domain data modeling needs to consider the extraction of corresponding domain knowledge. In this application embodiment, the characteristics of multi-domain pre-training data can be learned through a codebook to obtain a generalizable prompt codebook, which can then add prompt information of that domain during the AI ​​model's inference process, thereby improving the inference and prediction accuracy of the AI ​​model.

[0176] Figure 11 This illustration shows a schematic diagram of how an AI model, according to an embodiment of this application, processes input time-series sample data in a pre-training state. For example... Figure 11 As shown, any time-series sample data D in the training dataset undergoes the following processing: First, the time-series sample data D is processed through multiple frequency domain masking and segmentation mapping operations to obtain multiple representation vectors, and the mapping, clustering, and sharding operations in the first cue layer are used to obtain the first cue matrix; then, the first cue matrix is ​​fused with the multiple representation vectors (e.g., concatenated) to obtain multiple fused vectors to enhance the information of the original time-series sample data D. The multiple fused vectors are input into the encoder for encoding to obtain multiple encoded vectors. The multiple encoded vectors are then fused to obtain a fused encoded vector. Finally, the fused encoded vector is processed by the reconstruction encoder layer and reconstruction head to obtain the reconstructed time-series data. D The predicted temporal data D is obtained through processing with the prediction decoder layer and prediction head. Based on the reconstructed temporal data... D The loss is calculated based on the time-series sample data D, resulting in a loss value of 5. The loss is then calculated based on the predicted time-series data D and the labels, resulting in a loss value of 6. Based on the loss values ​​5 and 6, the pre-trained AI model is tuned, specifically by adjusting the weight parameters of the first cue layer, the segmentation and mapping layer, the encoder layer, the fusion layer, the reconstruction decoder layer, the reconstruction head, and the prediction head.

[0177] In another example, the first cue matrix can also be fused at the encoder output. That is, the first cue matrix is ​​fused with multiple encoded vectors output by the encoder after processing the representation vectors of multiple mask sequences to obtain multiple fused encoded vectors. Then, the multiple fused encoded vectors are aggregated into a feature vector, which is then input to the reconstruction decoder layer and the prediction decoder layer for subsequent processing.

[0178] Figure 12 This diagram illustrates the implementation of the prompting learning method provided in this application during the pre-training phase. For example... Figure 12As shown, the cue layer contains a register codebook, which stores multiple cue vectors representing domain characteristics from multiple domains. For example, the register codebook stores multiple cue vectors e1, e2, e3…e H Each cue vector represents a domain feature, meaning each cue vector is used to provide a generalized description of domain knowledge. Of course, in other examples, a cue vector can represent domain features from multiple domains, or multiple cue vectors can represent domain features from a single domain.

[0179] In one example, the initial multiple cue vectors in the register codebook can be derived from the representation vectors X of temporal sample data from multiple domains. e Clustering is used to obtain the representation vector X of time-series sample data from multiple domains. e Clustering is performed to obtain H clusters. The center vector of each cluster is used as the initial multiple cue vectors e1, e2, e3...e in the register codebook. H .

[0180] Then, the multiple cue vectors in the register codebook are updated and learned through cue learning.

[0181] See also Figure 12 Time series sample data is mapped into a representation vector X through a mapping layer. e Then the Cluster layer performs the following operations:

[0182]

[0183] That is, matching is performed from the register codebook (also known as the hint information base or hint codebook) to obtain the representation vector X. e The most relevant hint vector (which in some other examples may also be the representation vector X) e The most relevant multiple cue vectors are then used to segment the cue vectors to obtain the first cue matrix X. u .

[0184] Optionally, a similarity metric can be used to match the representation vector X from the register codebook. e The top k most relevant cue vectors, that is, the top k most relevant cue vectors in the register codebook that are most relevant to the representation vector X. e The K most similar cue vectors are those in the register codebook that are most similar to the representation vector X. e The top k most relevant cue vectors.

[0185] The first prompt matrix X uMultiple representation vectors from multiple mask sequences are concatenated to obtain multiple fusion vectors, which are used to enhance the information of the original time-series sample data D. These fusion vectors are then input into an encoder for encoding, resulting in multiple encoded vectors. These encoded vectors are then fused to obtain a fused encoded vector. Finally, the fused encoded vector is processed by the reconstruction encoder layer and reconstruction head to obtain the reconstructed time-series data. D The predicted temporal data D is obtained through processing with the prediction decoder layer and prediction head. Based on the reconstructed temporal data... D The loss is calculated using the time-series sample data D to obtain loss value 5. The loss is calculated using the predicted time-series data D and the label to obtain loss value 6. The weight parameters of the mapping layer are adjusted based on loss value 5 and loss value 6, and multiple cue vectors in the register codebook are also adjusted to obtain a cue codebook that can represent the features of various domains in multiple domains.

[0186] In the pre-training stage, the AI ​​model is trained through multiple frequency domain mask reconstructions and cue learning. In the prediction stage, the first cue matrix, which represents the domain characteristics of the input time-series data, is obtained by matching the input time-series data from the register codebook. The first cue matrix is ​​then concatenated with the input time-series data to enhance the time-series data and improve the prediction accuracy of the AI ​​model.

[0187] It is easy to understand that the trained AI model does not need to perform multi-frequency domain mask reconstruction and cue learning in the prediction and inference state. In other words, the AI ​​model in the inference state does not include multi-frequency domain mask operations, reconstruction decoder and reconstruction head.

[0188] Figure 13 This illustration shows a schematic diagram of the processing of input time-series data by the AI ​​model after training, as provided in an embodiment of this application, during the inference phase. Figure 13 As shown, the time-series data is used as input to the prompting layer and the segmentation mapping layer, respectively, to obtain the first prompting matrix and the representation vector. The first prompting matrix and the representation vector are concatenated to obtain the concatenated vector (of course, in other examples, the first prompting matrix can also be concatenated with the encoder vector after the representation vector is encoded by the encoder layer after the encoder layer). That is, the prompting information is used to enhance the time-series data. The concatenated vector is then input to the encoder layer for encoding to obtain the encoded vector. The encoded vector is processed by the prediction decoder layer and the prediction head to output the predicted time-series data. This predicted time-series data indicates the business data values ​​generated by the business in the future.

[0189] Considering that the multiple domains of the sample data included in the training dataset cannot cover all domains, the multiple domain features in the register codebook obtained through the prompts learned in the pre-training stage cannot cover all domain features. For the input time series data of a specific downstream task, the register codebook may not contain the domain features of the domain to which the input time series data belongs. In other words, during the prediction and inference process, the first prompt matrix matched from the prompt codebook may not be able to represent the domain features of the domain of the input time series data, which will affect the prediction accuracy of the AI ​​model.

[0190] Therefore, in another example, the cue learning provided in this application embodiment also includes cue learning in a fine-tuning stage. Through cue learning in the fine-tuning stage, specific information of the current task domain is learned. The specific information of the current task domain and the domain features learned in the register codebook are fused together to inject new domain knowledge into the original time series data, thereby improving the prediction accuracy of the AI ​​model for specific business scenarios.

[0191] Figure 14 This illustration shows a schematic diagram of the implementation of cue learning during the fine-tuning stage provided in an embodiment of this application. For example... Figure 14 As shown, the register codebook learned in the pre-training stage is frozen (or you can choose not to freeze it and participate in fine-tuning). It does not participate in fine-tuning. Multiple time-series sample data are generated using business data from the current business scenario. Multiple time-series sample data from the current business scenario are used for prompt learning to obtain the domain features of the current business scenario.

[0192] The time series sample data E is any one of multiple time series sample data in the current business scenario. Taking the time series sample data E as an example, we will introduce the processing process of the fine-tuned AI model on the time series sample data.

[0193] like Figure 14 As shown, the time-series sample data E is processed by a mapping layer to obtain the representation vector X. e The representation vector X e The Select layer matches the register codebook to obtain the representation vector X. e The top k most relevant cue vectors are used to derive the first cue matrix. The Select layer processing is as follows:

[0194]

[0195] By multiplying learnable vectors u and v into matrices, a low-rank matrix with a size relative to the first cue matrix is ​​obtained. This matrix is ​​then fused with the first cue matrix, for example, by performing a Hadamard product, to obtain the target cue matrix X. rThe target cue matrix is ​​subsequently concatenated with the representation vectors of multiple mask sequences obtained from the time-series sample data E after multiple frequency domain masking operations, resulting in multiple fused vectors to enhance the information of the original time-series sample data E. These fused vectors are then input into the encoder layer for encoding, yielding multiple encoded vectors. These encoded vectors are then fused to obtain a fused encoded vector. Finally, the fused encoded vector is processed by the reconstruction encoder layer and reconstruction head to obtain the reconstructed time-series data. E The predicted temporal data E is obtained through processing with the prediction decoder layer and prediction head. Based on the reconstructed temporal data... E The loss is calculated based on the time series sample data E, resulting in a loss value of 7. The loss is then calculated based on the predicted time series data E and the label, resulting in a loss value of 8. The weight parameters of the mapping layer, as well as the vectors u and v, are adjusted based on the loss values ​​7 and 8 to obtain the domain knowledge features that can characterize the domain to which the time series sample data E belongs.

[0196] In this way, by learning generalized features from multiple domains during the pre-training phase, and injecting specific domain knowledge for downstream tasks during the fine-tuning phase, efficient downstream fine-tuning can be achieved by fusing generalized domain feature information (domains that have appeared in pre-training) and specific additional feature information, thereby further enhancing the AI ​​model's prediction accuracy for downstream tasks.

[0197] The following specific application examples illustrate the practical implementation of the solution provided in this application. It can be divided into pre-training state, fine-tuning state, and inference state.

[0198] pre-training state

[0199] Multi-source data preprocessing: Using the collected open-source dataset Monash, which covers multiple fields such as energy, transportation, nature, and society, and considering the different data dimensions across different fields, the datasets were split into one-dimensional data for pre-training. Furthermore, because the sampling frequencies of each dataset are not entirely the same, partial downsampling was used to enhance the overall distribution of the pre-training dataset, resulting in a total of 358 million time-point data points. This data was then segmented to construct training samples, and for a specific one-dimensional time series dataset R... M The prediction task is defined as predicting future data of a certain length based on historical data of a certain length. Based on this requirement, the samples are divided according to the historical length L (512 in this embodiment) and the prediction length T (720 in this embodiment). That is, the sample length (512+720=1232, with a time step of 1 for sliding, to construct multiple samples). Similarly, any single-dimensional time series data is processed in this way, thereby obtaining a massive amount of pre-trained samples based on multi-domain data.

[0200] Multi-frequency domain mask reconstruction: This is part of the overall model training task, obtaining the corresponding sample sequence R. L First, the sample sequence R is converted through time-frequency transformation. L The process is transformed to the frequency domain, and a threshold λ is randomly generated to divide the entire frequency domain components into high-frequency and low-frequency parts. Then, a random number is used to determine whether to mask the high-frequency or low-frequency part (this masking operation removes the selected frequency component). Based on the currently unmasked high-frequency or low-frequency part, time-frequency transformation is performed to restore the time domain from the frequency domain. The resulting time series has undergone masking destruction at the frequency domain level. This process is repeated n (e.g., 10 times) times for the original input sequence to obtain multiple sequences of masking destruction operations. This sequence is fused with the cue matrix obtained from subsequent cue layers. The processed sequence is then sequentially encoded by an encoder, and the n codes are fused. The original input sequence R is then processed based on the fused code. L The reconstructed training task.

[0201] Prompt Codebook Generalization: During the pre-training phase, a generalized prompt codebook is learned from multi-domain pre-training data. Based on the prompt codebook, a prompt matrix representing the domain knowledge corresponding to the current data is obtained to enhance the information of the original data input. In this process, a learnable Prompt Codebook of size M and dimension N (in this example, M is 128 and N is 512) is constructed (i.e., 128*512 in size). The input is the original historical time series data R. L The driver, through neural networks and similarity metrics, extracts the top-k (8 in this example) most relevant prompts from the prompt codebook for the current data or the current domain, forming a prompt matrix. This prompt matrix is ​​then concatenated with the sequence data after masking and corruption, and input into the encoder for information interaction. Simultaneously, the prompt codebook is continuously trained and optimized based on the current pre-trained dataset, thereby enabling the prompt codebook to generalize the description of knowledge in different domains and abstract a generalized prompt codebook.

[0202] The prediction task was designed to provide zero-shot learning capabilities for historical input data R. L Representation learning to future data R T The mapping is performed by a neural network, i.e., the prediction head mentioned above. For example, the prediction head can be an MLP network.

[0203] To further improve the performance of the pre-trained model in downstream tasks, it is necessary to fine-tune the pre-trained model based on a subset of data from the downstream task scenarios. The fine-tuning process is basically the same as the pre-training process. Taking the load prediction scenario data as an example, it contains one year's load data for a certain location, with a time granularity of 15 minutes. In this embodiment, it is necessary to predict the load data for 96 future locations based on the load data of 512 historical locations. The main process is as follows:

[0204] Downstream task data preprocessing: The training set, validation set, and test set are divided into a 6:2:2 ratio. The samples are split according to the historical length L (512 in this embodiment) and the prediction length T (96 in this embodiment). The sample set is constructed sequentially, i.e., the sample length (512+96=608, sliding with a time step of 1 to construct multiple samples). At the same time, in order to verify the few-shot learning ability, the last 50% of the training set is selected for few-shot fine-tuning learning verification. The effect is also verified based on the full training set.

[0205] Multi-frequency domain mask reconstruction: After the downstream task data is preprocessed, the multi-frequency domain mask reconstruction task is used as a fine-tuning task. This process is consistent with the pre-training process, and will not be described in detail here for the sake of simplicity.

[0206] Hint codebook generalization: The fine-tuning state differs slightly from the pre-training state. The hint codebook learned in the pre-training state can be either fixed and not involved in fine-tuning or not fixed and involved in fine-tuning. At the same time, based on the current business scenario, a learnable network is provided to extract specific information of the current task domain. In this embodiment, a low-rank matrix of the same size as the hint matrix is ​​obtained by multiplying the learnable vectors u and v matrices, and a Hadamard product is performed with the hint matrix selected in the codebook. During the fine-tuning process, by fusing with the generalized domain knowledge hint matrix obtained from the hint codebook, new domain knowledge is injected into the original time series, thereby improving the model's ability to handle specific business scenarios.

[0207] Prediction task construction: This process is the same as the pre-training process. If the prediction length is consistent with the pre-training length, it is not necessary to replace the prediction head; otherwise, the prediction head size needs to be replaced according to the new business requirements. For example, in this embodiment, the prediction length is inconsistent with the prediction length during pre-training, and the prediction head output size needs to be replaced from 720 to 96.

[0208] Inference state: After obtaining the high-precision prediction model through the fine-tuning state, it is deployed and then real-time prediction is performed based on online input business data. In the inference state process, the data does not need to undergo multiple frequency domain masking operations, but the extraction of prompt codebook information is retained.

[0209] The AI ​​model training method provided in this application uses multiple frequency domain mask reconstruction operations to enable the encoder to learn patterns under different frequency components of time-series data, thus giving the encoder a powerful information extraction capability. At the same time, based on cue codebook generalization learning, it learns generalization features of different domains in the pre-training stage, and adds specific domain information in the fine-tuning / inference state. Through this information enhancement, the performance of the AI ​​model in downstream tasks is further improved.

[0210] To verify the performance of the AI ​​model trained by the training method provided in this application embodiment, experiments were conducted at a Huawei power load site with 50% few-shot and 100% full-shot results. These results were compared with other existing solutions. From the accuracy metric, this application embodiment achieved state-of-the-art (SOTA) performance at 50% few-shot, as shown in Table 1.

[0211]

[0212] Table 1

[0213] To further verify the training scheme provided in this application embodiment, fine-tuning was performed using different downstream task datasets. Validation was conducted on open-source datasets such as Weather, ETTm2, ETTm1, and ETTh2 (few-shot learning on 50% of the training dataset outperformed the current best case-by-case modeling method), demonstrating that the AI ​​model trained in this application embodiment can be extended to various multivariate time series prediction tasks such as influenza epidemics, traffic, meteorology, power, and transformer oil temperature. Table 2 shows the performance of the AI ​​model trained in this application embodiment (which can be referred to as ROSE) on multiple datasets, proving its excellent performance.

[0214]

[0215] Table 2

[0216] This application provides an AI model inference method, which uses the AI ​​model trained by the training method provided in this application to achieve accurate prediction of downstream tasks.

[0217] Figure 15 This is a flowchart illustrating the inference method of the AI ​​model provided in an embodiment of this application. The method can be... Figure 2 The execution device 210 performs the execution, and the computing module 211 of the execution device 210 is equipped with a training device 220 that has trained an AI model according to the AI ​​model training method provided in the embodiments of this application. Figure 15As shown, the inference method of the AI ​​model provided in this application embodiment includes at least steps S1501 to S1502.

[0218] In step S1501, the time series data to be predicted is obtained.

[0219] Users can input the time-series data to be predicted to the I / O interface 212 through the client device 240. The time-series data to be predicted can be the time-series data formed by the business data values ​​generated by the target business within a certain period of time. For example, if the current application scenario of the AI ​​model is a load prediction task for site A, the time-series data to be predicted is: historical load data generated from 512 points in site A up to the current time with a time granularity of 15 minutes. The historical load data generated at these 512 time points constitute the time-series data to be predicted.

[0220] In step S1502, the time series data to be predicted is used as the input to the AI ​​model, and the prediction result is output.

[0221] The time series data to be predicted is input into the AI ​​model, and the AI ​​model outputs a prediction result after inference. The prediction result indicates the business data value within a future period. The AI ​​model is trained using the training method of the AI ​​model provided in the embodiments of this application.

[0222] For example, if the time series data to be predicted consists of historical load data generated from 512 points of site A to the present, then the predicted time series data output by the AI ​​model indicates the load data to be generated by site A in the next 96 points. It is understandable that the prediction time length of the predicted time series data output by the AI ​​model is related to the prediction head trained by the AI ​​model during the training / fine-tuning process.

[0223] For details on how AI models process the input time-series data to be predicted, please refer to [the relevant documentation / reference]. Figure 10 and Figure 13 The description.

[0224] The AI ​​model inference method provided in this application employs an AI model that, on the one hand, uses multiple frequency domain masking reconstruction during the training phase. This involves randomly masking high-frequency or low-frequency signals in the time series, setting different thresholds, performing multiple frequency domain masks, and then aggregating them after inputting them into the encoder. This operation allows for the reconstruction of the time series from different information perspectives, enabling the learning of different frequency patterns and giving the encoder a powerful information extraction capability, resulting in better prediction accuracy for the AI ​​model. On the other hand, through cue learning during the pre-training phase, highly generalizable and multi-domain-specific cue information (i.e., cue codebook) is obtained. During the fine-tuning / inference phase, the current domain data drives the acquisition of the corresponding discrete cue matrix, and specific task-related cue information is obtained based on efficient fine-tuning. This results in a cue matrix that simultaneously possesses generalization and specific features, enhancing the prediction capability of downstream tasks and further increasing the prediction accuracy of the AI ​​model.

[0225] Based on the same concept as the aforementioned embodiment of an AI model training method, this application also provides an AI model training device 1600, which can train an AI model with high prediction accuracy. The AI ​​model training device 1600 includes components for implementing... Figure 4-14 The units or modules in the training method of the AI ​​model shown.

[0226] Figure 16 This is a schematic diagram of the structure of an AI model training device provided in an embodiment of this application. Figure 16 As shown, the training device 1600 for the AI ​​model includes a first acquisition module 1601, a frequency domain masking module 1602, a reconstruction module 1603, and a training module 1604. The first acquisition module 1601 acquires a training dataset, which includes multiple first time-series sample data. The frequency domain masking module 1602 acquires multiple first masking sequences, obtained by performing multiple frequency domain masking operations on the first time-series sample data. The multiple frequency domain masking operations mask different frequency components of the corresponding frequency domain data of the time-series sample data. The reconstruction module 1603 uses the multiple first masking sequences as input to the AI ​​model to be trained and outputs first reconstructed time-series data. The training module 1604 adjusts the weight parameters of the AI ​​model to be trained based on the first reconstructed time-series data and its corresponding first time-series sample data to obtain a pre-trained AI model.

[0227] In one possible implementation, the frequency domain masking module 1602 is specifically used to: convert the first time-series sample data to the frequency domain to obtain frequency domain data; perform a masking operation on the frequency domain data to obtain frequency domain mask data; convert the frequency domain mask data to the time domain to obtain a first mask sequence; and repeat the above operation multiple times on the first time-series sample data to obtain multiple first mask sequences.

[0228] In another possible implementation, a specific implementation of performing a masking operation on the frequency domain data to obtain frequency domain masked data is as follows: generate a partitioning threshold, and divide the frequency domain data into high-frequency part data and low-frequency part data based on the partitioning threshold; perform a masking operation on the high-frequency part data or the low-frequency part data in the frequency domain data to obtain frequency domain masked data.

[0229] Optionally, the partitioning threshold is randomly generated, and / or the high-frequency or low-frequency data in the frequency domain data to be masked is randomly determined. In this way, by randomly generating the partitioning threshold, the frequency domain data is randomly divided into high-frequency and low-frequency data, and then the masking operation is randomly performed on the high-frequency or low-frequency data, thereby increasing the robustness of the backbone network of the AI ​​model.

[0230] In another possible implementation, the AI ​​model to be trained includes a first backbone network layer and a first reconstruction head; the reconstruction module 1603 is specifically used to: input multiple first mask sequences into the first backbone network layer to obtain a first feature vector; input the first feature vector into the first reconstruction head to output first reconstructed temporal data. A loss value is calculated using the reconstructed temporal data and the input temporal sample data, and then the weight parameters of the first backbone network and the first reconstruction head are adjusted based on the loss value.

[0231] In another possible implementation, the backbone network layer includes an encoder layer; a specific implementation of inputting multiple first mask sequences into the first backbone network layer to obtain a first feature vector is as follows: the multiple first mask sequences are used as input to the encoder layer, and multiple encoded vectors are output; based on the multiple encoded vectors, the first feature vector is obtained.

[0232] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are fused to obtain the first feature vector. By fusing the information of multiple encoding vectors, a first feature vector is obtained. Subsequently, a reconstructed time series data is obtained through this first feature vector. Then, the loss is calculated based on the reconstructed time series data and the input time series sample data to obtain a loss value. The weight parameters of the encoder and the reconstruction head are adjusted based on the loss value to increase training efficiency.

[0233] In another possible implementation, a specific way to obtain the first feature vector based on multiple encoding vectors is as follows: multiple encoding vectors are determined as multiple first encoding feature vectors, that is, the first encoding feature vectors include a set of encoding vectors. Subsequently, multiple reconstructed time series data are obtained through these multiple first feature vectors. Then, the loss is calculated based on these multiple reconstructed time series data and the input time series sample data to obtain multiple loss values. The weight parameters of the encoder and the reconstruction head are adjusted based on these multiple loss values.

[0234] In another possible implementation, the training dataset includes first time-series sample data from multiple domains; the AI ​​model to be trained includes a first cue layer, a second backbone network layer, and a second reconstruction head; the reconstruction module 1603 is specifically used to: take the first time-series sample data as input to the first cue layer to obtain a first cue matrix, the first cue matrix representing the domain features of the domain to which the first time-series sample data belongs; fuse the first cue matrix with multiple first vectors to obtain multiple first fused vectors, the multiple first vectors being multiple vectors after multiple first mask sequences have been processed by the second backbone network layer, or the multiple first vectors being multiple vectors before multiple first mask sequences have been processed by the second backbone network layer; obtain a second feature vector based on the multiple first fused vectors; input the second feature vector into the second reconstruction head to output reconstructed time-series data.

[0235] In another possible implementation, the first prompt layer includes a first prompt information library and a first input layer. The prompt information library stores multiple prompt vectors, which represent domain features of multiple domains. A specific implementation of using the first time-series sample data as input to the first prompt layer to obtain the first prompt matrix is ​​as follows: the first time-series sample data is processed by the first input layer to obtain a first input vector; based on the first input vector, several prompt vectors are matched from the prompt information library to obtain several prompt vectors; based on the several prompt vectors, the first prompt matrix is ​​obtained.

[0236] In another possible implementation, the training dataset also includes multiple labeled second time-series sample sequences; the AI ​​model to be trained also includes a prediction head, which is trained and optimized based on the multiple labeled second time-series sample sequences.

[0237] In another possible implementation, the first acquisition module 1601 is further used to acquire a training dataset for the target business scenario, which includes multiple labeled third time-series sample data under the target business scenario; the training device for the AI ​​model provided in this application also includes a fine-tuning module 1605, which is used to fine-tune the pre-trained AI model based on the training dataset for the target business scenario to obtain the trained AI model.

[0238] In another possible implementation, the pre-trained AI model includes a second cue layer, a third cue layer, a third backbone network, a third reconstruction head, and a second prediction head. The fine-tuning module 1605 is specifically used for: performing multiple frequency domain masking operations on the third time-series sample data to obtain multiple second mask sequences; fusing the second and third cue matrices to obtain a target cue matrix, where the second cue matrix is ​​obtained based on the processing of the third time-series sample sequences by the second cue layer, representing the domain characteristics of the third time-series sample sequences; and the third cue matrix is ​​obtained based on the third cue layer, representing the characteristics of the target business scenario; and fusing the target cue matrix with multiple second vectors to obtain multiple second fused vectors. Multiple second vectors are either multiple vectors resulting from the processing of multiple second mask sequences by the third backbone network layer, or multiple vectors before the processing of multiple second mask sequences by the third backbone network layer. A third feature vector is obtained based on these multiple second fusion vectors. The third feature vector is used as input to the third reconstruction head and the second prediction head, respectively, to obtain second reconstruction temporal data and prediction data. A second loss value and a third loss value are determined based on the third temporal sample sequence and its corresponding second reconstruction temporal data, and the prediction data and its corresponding labels for the second reconstruction temporal data, respectively. The weight parameters of the third prompting layer, the third backbone network, the third reconstruction head, and the second prediction head are adjusted based on the second loss value and the third loss value.

[0239] In another possible implementation, the third cue layer includes a third cue matrix, which is updated and optimized during the fine-tuning of the pre-trained AI model based on a training dataset tailored to the target business scenario.

[0240] The AI ​​model training apparatus 1600 according to the embodiments of this application can correspond to executing the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the AI ​​model training apparatus 1600 are respectively for implementing Figure 4-14 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0241] Based on the same concept as the aforementioned embodiment of an AI model inference method, this application also provides an AI model inference device 1700, which can achieve accurate prediction of downstream tasks. The AI ​​model inference device 1700 includes components for implementing... Figure 15 The units or modules of each step in the reasoning method of the AI ​​model shown.

[0242] Figure 17 This is a schematic diagram of the structure of an AI model inference device provided in an embodiment of this application. Figure 17As shown, the inference device 1700 of the AI ​​model includes a second acquisition module 1701 and an inference module 1702. The second acquisition module 1701 is used to acquire time series data to be predicted; the inference module 1702 is used to take the time series data to be predicted as input to the AI ​​model and output the prediction result. The AI ​​model is trained based on the training method of the AI ​​model provided in the embodiments of this application.

[0243] The AI ​​model inference device 1700 according to the embodiments of this application can correspond to executing the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the AI ​​model inference device 1700 are respectively for implementing Figure 15 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0244] This application embodiment also provides a computing device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 4-15 The method described.

[0245] Figure 18 A schematic diagram of the structure of a computing device provided in an embodiment of this application.

[0246] like Figure 18 As shown, the computing device 1800 includes at least one processor 1801, a memory 1802, and a communication interface 1803. The processor 1801, memory 1802, and communication interface 1803 are communicatively connected, which can be achieved via a wired (e.g., bus) or wireless connection. The communication interface 1803 is used to send and / or receive data from other devices. The memory 1802 stores computer instructions, which the processor 1801 executes to perform the methods described in the preceding method embodiments, thereby improving the training quality of the AI ​​model and increasing the prediction accuracy of the trained AI model in downstream prediction tasks.

[0247] It should be understood that, in the embodiments of this application, the processor 1801 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0248] The memory 1802 may include read-only memory and random access memory, and provides instructions and data to the processor 1801. The memory 1802 may also include non-volatile random access memory. Optionally, the random access memory may be, for example, high bandwidth memory (HBM).

[0249] The memory 1802 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0250] It should be understood that the computing device 1800 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 4-15 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0251] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0252] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0253] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0254] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0255] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0256] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for training an AI model, characterized in that, include: Obtain a training dataset, which includes multiple first time-series sample data; Multiple first mask sequences are obtained by performing multiple frequency domain masking operations on the first time-series sample data. The multiple frequency domain masking operations mask different frequency components of the frequency domain data corresponding to the time-series sample data. The plurality of first mask sequences are used as input to the AI ​​model to be trained, and the first reconstruction time series data is output. Based on the first reconstructed time series data and its corresponding first time series sample data, the weight parameters of the AI ​​model to be trained are adjusted to obtain a pre-trained AI model.

2. The method according to claim 1, characterized in that, The acquisition of multiple first mask sequences includes: The first time-series sample data is converted to the frequency domain to obtain the frequency domain data; Perform a masking operation on the frequency domain data to obtain frequency domain masked data; The frequency domain mask data is converted to the time domain to obtain the first mask sequence; The above operation is repeated multiple times on the first time-series sample data to obtain the plurality of first mask sequences.

3. The method according to claim 2, characterized in that, The step of performing a masking operation on the frequency domain data to obtain frequency domain masked data includes: Generate a partitioning threshold, and divide the frequency domain data into high-frequency data and low-frequency data based on the partitioning threshold; A masking operation is performed on the high-frequency portion of the frequency domain data or the low-frequency portion of the frequency domain data to obtain the frequency domain mask data.

4. The method according to claim 3, characterized in that, The division threshold is randomly generated, and / or the high-frequency data or the low-frequency data in the frequency domain data to which the masking operation is performed is randomly determined.

5. The method according to any one of claims 1-4, characterized in that, The AI ​​model to be trained includes a first backbone network layer and a first reconstruction head; The step of using the plurality of first mask sequences as input to the AI ​​model to be trained and outputting reconstructed time-series data includes: The plurality of first mask sequences are input into the first backbone network layer to obtain the first feature vector; The first feature vector is input into the first reconstruction head, and the first reconstruction time series data is output.

6. The method according to claim 5, characterized in that, The backbone network layer includes an encoder layer; The step of inputting the plurality of first mask sequences into the first backbone network layer to obtain the first feature vector includes: The plurality of first mask sequences are used as input to the encoder layer, and a plurality of encoded vectors are output. The first feature vector is obtained based on the multiple encoded vectors.

7. The method according to claim 6, characterized in that, The process of obtaining the first feature vector based on the plurality of encoded vectors includes: The multiple encoded vectors are fused to obtain the first feature vector.

8. The method according to claim 1, characterized in that, The training dataset includes first-time-series sample data from multiple domains; The AI ​​model to be trained includes a first cueing layer, a second backbone network layer, and a second reconstruction head. The step of using the plurality of first mask sequences as input to the AI ​​model to be trained and outputting reconstructed time-series data includes: The first time-series sample data is used as the input to the first prompting layer to obtain a first prompting matrix, which represents the domain characteristics of the domain to which the first time-series sample data belongs. The first prompt matrix is ​​fused with multiple first vectors to obtain multiple first fused vectors. The multiple first vectors are multiple vectors after the multiple first mask sequences are processed by the second backbone network layer, or the multiple first vectors are multiple vectors before the multiple first mask sequences are processed by the second backbone network layer. Based on the multiple first fusion vectors, a second feature vector is obtained; The second feature vector is input into the second reconstruction head, and the reconstructed time series data is output.

9. The method according to claim 8, characterized in that, The first prompt layer includes a first prompt information library and a first input layer. The prompt information library stores multiple prompt vectors, and the multiple prompt vectors represent the domain features of the multiple domains. The step of using the first time-series sample data as input to the first prompting layer to obtain the first prompting matrix includes: The first time-series sample data is processed by the first input layer to obtain the first input vector; Based on the first input vector, several prompt vectors are obtained by matching from the prompt information database; Based on the aforementioned prompt vectors, the first prompt matrix is ​​obtained.

10. The method according to any one of claims 1-9, characterized in that, The training dataset also includes multiple labeled second time-series sample sequences; The AI ​​model to be trained also includes a prediction head, which is trained and optimized based on the multiple labeled second time-series sample sequences.

11. The method according to any one of claims 1-10, characterized in that, Also includes: Obtain a training dataset for the target business scenario, wherein the training dataset for the target business scenario includes multiple labeled third-time-series sample data under the target business scenario; Based on the training dataset for the target business scenario, the pre-trained AI model is fine-tuned to obtain the trained AI model.

12. The method according to claim 11, characterized in that, The pre-trained AI model includes a second prompting layer, a third prompting layer, a third backbone network, a third reconstruction head, and a second prediction head. The training dataset for the target business scenario is used to fine-tune the pre-trained AI model to obtain a fully trained AI model, including: Perform the multiple frequency domain masking operations on the third time-series sample data to obtain multiple second mask sequences; The second prompt matrix and the third prompt matrix are fused to obtain the target prompt matrix. The second prompt matrix is ​​obtained by processing the third time-series sample sequence based on the second prompt layer. The second prompt matrix represents the domain characteristics of the domain to which the third time-series sample sequence belongs. The third prompt matrix is ​​obtained based on the third prompt layer and represents the characteristics of the target business scenario. The target cue matrix is ​​fused with multiple second vectors to obtain multiple second fused vectors. The multiple second vectors are multiple vectors after the multiple second mask sequences are processed by the third backbone network layer, or the multiple second vectors are multiple vectors before the multiple second mask sequences are processed by the third backbone network layer. Based on the multiple second fusion vectors, a third feature vector is obtained; The third feature vector is used as the input to the third reconstruction head and the second prediction head, respectively, to obtain the second reconstruction time series data and prediction data; The second loss value and the third loss value are determined based on the third time series sample sequence and its corresponding second reconstructed time series data, and the predicted data and its corresponding second reconstructed time series data labels, respectively. Based on the second loss value and the third loss value, the weight parameters of the third prompting layer, the third backbone network, the third reconstruction head, and the second prediction head are adjusted.

13. The method according to claim 12, characterized in that, The third prompt layer includes a third prompt matrix, which is updated and optimized during the fine-tuning of the pre-trained AI model based on the training dataset for the target business scenario.

14. A reasoning method for an AI model, characterized in that, include: Obtain the time series data to be predicted; The time series data to be predicted is used as input to the AI ​​model, and the prediction result is output. The AI ​​model is trained based on the method described in any one of claims 1-13.

15. A training device for an AI model, characterized in that, include: The first acquisition module is used to acquire the training dataset, which includes multiple first time-series sample data. The frequency domain masking module acquires multiple first mask sequences, which are obtained by performing multiple frequency domain masking operations on the first time-series sample data. The multiple frequency domain masking operations mask different frequency components of the frequency domain data corresponding to the time-series sample data. The reconstruction module is used to take the multiple first mask sequences as input to the AI ​​model to be trained and output the first reconstruction time series data. The training module is used to adjust the weight parameters of the AI ​​model to be trained based on the first reconstructed time series data and its corresponding first time series sample data, so as to obtain the pre-trained AI model.

16. An inference device for an AI model, characterized in that, include: The second acquisition module is used to acquire the time series data to be predicted. The inference module is used to take the time series data to be predicted as input to the AI ​​model and output the prediction result, wherein the AI ​​model is trained based on the method described in any one of claims 1-13.

17. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-14 to be implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-14 to be implemented.

Images