Deep learning model training method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a deep learning model training method and device, computer equipment and a storage medium. A first decomposition variable and a second decomposition variable corresponding to the initial time sequence data are sequentially loaded into N reversible attention layers in an original neural network model for forward propagation, and an Nth high-layer activation feature and an Nth low-layer activation feature are predicted to complete prediction processing of the initial time sequence data; sequentially loading N reversible attention layers in the original neural network model, carrying out reverse propagation on an Nth high-level activation feature, an Nth low-level activation feature and a target loss value calculated according to the Nth high-level activation feature, and sequentially calculating a high-level inverse solution feature and a low-level inverse solution feature corresponding to each reversible attention layer, therefore, the original neural network model is updated without keeping the loading state of each reversible attention layer, and the applicability of the neural network model applied to the edge equipment is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deep learning model training method, device, computer equipment and storage medium. Background Art

[0002] Edge computing has the characteristics of low processing delay, low risk of data leakage and low data transmission cost. Because it does not rely too much on the network, edge computing can also perform local data processing tasks in a timely manner when the device is offline. Edge computing has been widely used.

[0003] The Transformer model based on the attention mechanism has strong characteristics in both natural language processing and computer vision, but the parameters of a complete Transformer model are extremely large. The common business processing method is that the user enters data on the client, sends the data to be processed to the server through the network, and uses the server's Transformer model to process it. However, the above method has problems such as high latency, dependence on the network and high risk of data leakage. Especially when processing business in the financial field, it cannot meet the low latency and high security characteristics of financial business.

[0004] In the prior art, due to the characteristics of the Transformer model, which has high parameter and structural complexity, high computational complexity, and high energy consumption, edge devices are limited by computing power, memory, and energy consumption, and are unable to directly load the multi-layer neural network of the Transformer model; and because each layer of the neural network of the Transformer model needs to cache the feature representation output by each layer during data transmission, it is used to update the parameters of each layer of the neural network. From the above, it can be seen that the current Transformer model is directly applied to edge devices, and the configuration requirements for edge devices are extremely stringent. Therefore, when the Transformer model is applied to edge devices, the Transformer model has the problem of poor applicability. Summary of the invention

[0005] The embodiments of the present invention provide a deep learning model training method, apparatus, computer equipment and storage medium, which solve the problem of poor applicability of the Transformer model when the Transformer model is applied to edge devices.

[0006] An embodiment of the present invention provides a deep learning model training method, comprising: Get initial time series data; Decomposing the initial time series data to obtain a first decomposition variable and a second decomposition variable; Sequentially load the N reversible attention layers in the original neural network model, perform forward propagation on the first decomposition variable and the second decomposition variable, obtain the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable; obtain the target loss value according to the Nth high-level activation feature; wherein N is a positive integer; Loading N reversible attention layers in the original neural network model in sequence, performing reverse propagation on the target loss value, the Nth high-level activation feature, and the Nth low-level activation feature, and obtaining the first high-level inverse feature corresponding to the Nth high-level activation feature and the first low-level inverse feature corresponding to the Nth low-level activation feature; According to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature, the first layer original parameters in the original neural network model are updated to obtain a target neural network model.

[0007] The embodiment of the present invention also provides a deep learning model training device, including: An initial time series data acquisition module, used to acquire initial time series data; An initial time series data decomposition module, used to decompose the initial time series data to obtain a first decomposition variable and a second decomposition variable; A forward propagation module, used to sequentially load the N reversible attention layers in the original neural network model, perform forward propagation on the first decomposition variable and the second decomposition variable, obtain the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable; obtain the target loss value according to the Nth high-level activation feature; wherein N is a positive integer; A reverse propagation module is used to sequentially load the N reversible attention layers in the original neural network model, reversely propagate the target loss value, the N-th high-level activation feature, and the N-th low-level activation feature, and obtain the first high-level inverse solution feature corresponding to the N-th high-level activation feature and the first low-level inverse solution feature corresponding to the N-th low-level activation feature; The target neural network model acquisition module is used to update the first layer original parameters in the original neural network model according to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature to obtain the target neural network model.

[0008] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned deep learning model training method when executing the computer program.

[0009] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned deep learning model training method.

[0010] The above-mentioned deep learning model training method, device, computer equipment and storage medium, after decomposing the initial time series data into a first decomposition variable and a second decomposition variable, sequentially load the N reversible attention layers in the original neural network model, forward propagate the first decomposition variable and the second decomposition variable, predict the Nth high-level activation feature and the Nth low-level activation feature to complete the prediction processing of the initial time series data; sequentially load the N reversible attention layers in the original neural network model, reverse propagate the Nth high-level activation feature, the Nth low-level activation feature and the target loss value calculated according to the Nth high-level activation feature, sequentially calculate the high-level inverse solution feature and the low-level inverse solution feature corresponding to each reversible attention layer, thereby completing the update of the original neural network model without keeping each reversible attention layer in a loaded state, thereby improving the applicability of the neural network model to edge devices.

[0011] When loading a neural network model (Transformer model) through a common edge device, the reversible neural network layers of the Transformer model can be loaded sequentially through the memory without having to load the entire Transformer model at the same time. This solves the problems of large model size, high computing cost, energy consumption, poor real-time requirements, and high transmission costs when deploying neural network models on edge devices. When the neural network model is deployed on the edge device, it meets the low latency and high security requirements of fraud detection, risk assessment, and personalized finance in the financial field. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0013] Figure 1 is a schematic diagram of an application environment of a deep learning model training method in one embodiment of the present invention; Figure 2 is a flow chart of a deep learning model training method in one embodiment of the present invention; Figure 3 is a flowchart of step S203 of the deep learning model training method in one embodiment of the present invention; Figure 4is a flowchart of step S301 of the deep learning model training method in one embodiment of the present invention; Figure 5 is a flowchart of step S204 of the deep learning model training method in one embodiment of the present invention; Figure 6 is a flowchart of step S501 of the deep learning model training method in one embodiment of the present invention; Figure 7 is a flowchart of step S205 of the deep learning model training method in one embodiment of the present invention; Figure 8 is a flowchart of step S701 of the deep learning model training method in one embodiment of the present invention; Fig. 9 is a schematic diagram of a deep learning model training device in one embodiment of the present invention; Fig.10 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The deep learning model training method provided by the embodiment of the present invention can be applied in Figure 1 In the application environment shown. Figure 1 As shown, the client (computer device) communicates with the server (edge ​​device) through the network. The client, also known as the user end, refers to the program that corresponds to the server (edge ​​device) and provides local services to the client. The client (computer device) includes but is not limited to various personal computers, laptops, smart phones, tablets, cameras and portable wearable devices. The server (edge ​​device) can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0016] The deep learning model training method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the deep learning model training method is applied in a deep learning model training system, and the deep learning model training system includes: Figure 1 The client and server (edge ​​device) shown communicate with each other through a network to implement the deep learning model training method provided in an embodiment of the present invention.

[0017] In this example, in business applications in the financial field, the Transformer model can process and understand complex text data, time series data, and structured data, learn key features and patterns from them, and provide accurate prediction and decision results. The Transformer model on the edge device can achieve fast data processing and instant decision support, providing efficient, personalized, and secure services for financial business.

[0018] Among them, common business applications in the financial field include fraud detection, risk assessment investment decision-making, personalized financial services, etc. The specific applications are as follows: When applied to fraud detection, the Transformer model can learn fraud patterns and abnormal patterns by analyzing large amounts of transaction data and user behavior patterns, and identify potential fraud behaviors in real time, thereby detecting fraud. The Transformer model on edge devices can quickly make predictions and decisions, reducing fraud risks and losses.

[0019] When applied to risk assessment, the Transformer model can analyze the customer's historical data, financial data, market data, etc., learn complex feature associations, complete risk assessment, and provide accurate risk assessment results. The Transformer model on the edge device can process customer data in real time, quickly assess risks, and provide decision support.

[0020] When applied to investment decisions, the Transformer model can analyze a large amount of financial news, market data, and financial statements, learn market patterns and trends, and predict stock price fluctuations and market fluctuations to make investment decisions. The Transformer model on edge devices can provide real-time investment advice and decision support to help investors make more informed decisions.

[0021] When applied to personalized financial services, the Transformer model can analyze customers' transaction history, preferences, and behavior patterns to provide personalized recommendations, customized products and services. The Transformer model on edge devices can instantly generate personalized financial services and recommendations based on customers' real-time data, improving customer satisfaction and loyalty.

[0022] In one embodiment, if Figure 2 As shown, a deep learning model training method is provided, and the method is applied in Figure 1The following steps are used to illustrate the server (edge ​​device) in the example: S201: Acquire initial time series data; S202: Decompose the initial time series data to obtain a first decomposition variable and a second decomposition variable; S203: sequentially load the N reversible attention layers in the original neural network model, perform forward propagation on the first decomposition variable and the second decomposition variable, obtain the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable; obtain the target loss value according to the Nth high-level activation feature; wherein N is a positive integer; S204: sequentially loading the N reversible attention layers in the original neural network model, reversely propagating the target loss value, the Nth high-level activation feature, and the Nth low-level activation feature, and obtaining the first high-level inverse feature corresponding to the Nth high-level activation feature and the first low-level inverse feature corresponding to the Nth low-level activation feature; S205: According to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature, the first layer original parameters in the original neural network model are updated to obtain the target neural network model.

[0023] Among them, this example is a deep learning model, and in particular, it involves a Transformer model based on a reversible attention mechanism module. When the Transformer model is deployed on edge devices, it is suitable for processing various types of sequence data, including text, speech, images, etc. It has broad application potential in the fields of natural language processing, dialogue systems, image processing, audio processing, and recommendation systems. By applying the Transformer model on edge devices, fast, personalized, and privacy-secure data processing and services can be achieved.

[0024] Furthermore, the traditional Transformer model has the problems of poor computational efficiency and large memory footprint. In the prior art, a Transformer model based on a reversible attention mechanism is an improvement on the traditional Transformer model. Since the computational complexity of the attention mechanism is quadratically related to the sequence length, the Transformer model based on the reversible attention mechanism allows the calculation results of the attention weights to be saved in memory for subsequent use by designing the calculation process of the attention weights as a reversible operation, thereby avoiding repeated calculations, especially when processing longer sequences. However, reducing memory footprint only through reverse calculations still cannot guarantee the applicability of the Transformer model based on the reversible attention mechanism on edge devices.

[0025] As an example, in step S201, the server receives the initial time series data transmitted by the client, or receives the initial time series data uploaded locally by the server, and the initial time series data may be the original time series. The initial time series data may be the historical transaction data and repayment behavior of customers, market index, currency exchange rate and interest rate data arranged according to time in the financial field, so as to be used for risk assessment and market trend prediction.

[0026] In this example, if there are N time series data X, and each data is set to L length, then the data set is: Among them, X is the initial time series data, Represents the second data element of the i-th text.

[0027] In this example, especially in the financial field, the customer's historical transaction data and repayment behavior can be used as time series data for credit risk assessment; investment portfolio information can be analyzed as time series data to assess the volatility and correlation of different assets, thereby predicting investment risks; time series data analysis can be used to analyze fluctuations and trends in the financial market; market indices, currency exchange rates, interest rates and other data can be analyzed as time series data to predict market changes and future trends, providing a basis for investment decisions.

[0028] It can be understood that time series data is a collection of data points arranged in chronological order. It is usually a sequence of data values ​​observed at consecutive time points, where each data point corresponds to a specific timestamp. Time series are usually used to describe phenomena or events that change over time, such as daily temperature, monthly sales, stock price history, etc.

[0029] As an example, in step S202, after receiving the time series data, the server decomposes the initial time series data into a first decomposition variable and a second decomposition variable according to a preset channel. In this example, the initial time series data X=[X1,X2], where X1 is the first decomposition variable and X2 is the second decomposition variable.

[0030] In this example, the invertible neural network (INN) used by the Transfomer model based on the reversible attention mechanism module is a basic building block of the network, which is the affine coupling layer promoted by the Real NVP model. The neural network is divided into two parts according to the channel, and the input initial time series data is divided into two parts X1 (first decomposition variable) and X2 (second decomposition variable) according to the channel. Then, the two parts are passed to the residual modules in each layer of the neural network for processing, and the outputs of the two parts are recombined in the last layer.

[0031] As an example, in step S203, after receiving the decomposed first decomposition variable and the second decomposition variable, the server sequentially loads N reversible neural networks by inputting the first decomposition variable and the second decomposition variable into the original neural network model to process the first decomposition variable and the second decomposition variable, and outputs the Nth high-level activation feature and the Nth low-level activation feature. After the Nth high-level activation feature is determined as a predicted value, the loss value between the predicted value and the true value is calculated through the loss function to determine the target loss value of this calculation, which is used to update the model parameters.

[0032] Among them, when loading the next layer of reversible neural network, loading the reversible neural network of this layer will stop.

[0033] It should be noted that the original neural network model consists of N reversible attention layers. Due to the configuration limitations of the server (edge ​​device), multiple reversible attention layers cannot be loaded simultaneously when running the original neural network model. The present invention sequentially loads the corresponding reversible attention layers through the memory to run the entire original neural network model on the server (edge ​​device) to complete the prediction business of the initial time series data.

[0034] Furthermore, after the initial time series data (the first decomposition variable and the second decomposition variable) are input, the first reversible attention layer is loaded to output the output value (the first high-level activation feature and the first low-level activation feature) corresponding to the current reversible attention layer (the first reversible attention layer); the second reversible attention layer is loaded in sequence to receive the output value output by the previous reversible attention layer (the first reversible attention layer), and the output value (the second high-level activation feature and the second low-level activation feature) corresponding to the current reversible attention layer is output; when the last reversible attention layer (the Nth reversible attention layer) is loaded, according to the output value (the N-1th high-level activation feature and the N-1th low-level activation feature) corresponding to the previous reversible attention layer (the N-1th reversible attention layer), the output value (the Nth high-level activation feature and the Nth low-level activation feature) output by the last reversible attention layer is output, and the calculation of the initial input value is completed, and the loss calculation is performed according to the output value output by the last reversible attention layer for model updating.

[0035] Furthermore, in the forward propagation, each reversible attention layer calculates the output value (high-level activation features and low-level activation features) of the current reversible attention layer through the attention mechanism for the input value of the previous layer. The specific formula is as follows: In the forward propagation, X(X1,X2) is the output value of the previous reversible attention layer (the input value of the current reversible attention layer), X1 is the low-level activation feature or the first decomposition variable of the previous layer output (the current layer input), and X2 is the high-level activation feature or the second decomposition variable of the previous layer output (the current layer input); Y(Y1,Y2) is the output value of the current reversible attention layer, Y1 is the low-level activation feature of the current layer output, and Y2 is the high-level activation feature of the current layer output; Attention is the attention calculation, and feedforward is the feedforward operation.

[0036] In this example, the calculation of the target loss value is based on different business requirements, that is, the different correlations between high-level activation features and low-level activation features. Different loss functions are selected to calculate the predicted value and the true value corresponding to the output value. Common loss functions include but are not limited to mean square error (MSE), cross-entropy loss (Cross-Entropy Loss) and absolute error loss (MAE).

[0037] Among them, since the Nth high-level activation features described in the last reversible attention layer (Nth reversible attention layer) of this example include the Nth low-level activation features. Therefore, this example can obtain the target loss value according to the predicted value and the true value corresponding to the Nth high-level activation feature; or obtain the target loss value by respectively obtaining the true value corresponding to the Nth high-level activation feature and the true value corresponding to the Nth low-level activation feature according to the Nth high-level activation feature and the Nth low-level activation feature; and can also obtain the target loss value by fusing the Nth high-level activation feature and the Nth low-level activation feature to obtain the corresponding predicted value.

[0038] As an example, in step S204, after obtaining the target loss value, the server reversely propagates the Nth high-level activation feature and the Nth low-level activation feature, loads them into N reversible neural networks in sequence, reversely processes the Nth high-level activation feature and the Nth low-level activation feature, and outputs the Nth high-level activation feature and the Nth low-level activation feature. After the Nth high-level activation feature is determined as the predicted value, the loss value between the Nth high-level activation feature and the true value is calculated through the loss function to determine the target loss value of this calculation, which is used to update the model parameters. The predicted value is confirmed according to business needs, and can be the evaluation result of the customer's credit risk assessment, the prediction result of the market investment risk forecast; it can be used to predict the future trend of market changes.

[0039] Among them, when loading the next layer of reversible neural network, loading the reversible neural network of this layer will stop.

[0040] It should be noted that in the optimization and update process of the original neural network model, each reversible attention layer needs to calculate the gradient of the reversible attention layer according to the input value (high-level inverse solution features and low-level inverse solution features of the previous layer) and the output value (high-level inverse solution features and low-level inverse solution features of this layer) of the current calculation of the reversible attention layer, combined with the target loss function, to complete the parameter update of the reversible attention layer. When the last reversible attention layer (the first reversible attention layer) in the back propagation process is updated, the entire original neural model is updated. In the existing method, each reversible attention layer updates the parameters by caching the output value and combining it with the target loss value obtained by subsequent calculation. This method requires each reversible attention layer to cache the output value in a temporary cache for use in subsequent layers. According to the chain rule, this application calculates the input value (output value of the previous layer) of each reversible attention layer in sequence during the back propagation process by reverse propagation, so that there is no need to cache the output value of each reversible attention layer, that is, to save cache space through redundant calculation and reduce the computing pressure on the server (edge ​​device).

[0041] Furthermore, after calculating the target loss value, the N-th high-level activation feature and the N-th low-level activation feature are loaded into the N-th reversible attention layer to output the output value (the N-th high-level inverse feature and the N-th low-level inverse feature) corresponding to the current reversible attention layer (the N-th reversible attention layer); the N-1-th reversible attention layer is reversely loaded to receive the output value output by the previous reversible attention layer (the N-th reversible attention layer), and the output value (the N-1-th high-level inverse feature and the N-1-th low-level inverse feature) corresponding to the current reversible attention layer is output; when reversely loaded to the last reversible attention layer (the first reversible attention layer), the output value (the first high-level inverse feature and the first low-level inverse feature) corresponding to the previous reversible attention layer (the second reversible attention layer) is output according to the output value (the second high-level inverse feature and the second low-level inverse feature) received, and the reverse propagation process is completed.

[0042] The forward propagation input value of each reversible attention layer is the same as the reverse propagation output value. For example, the Nth high-level inverse feature is the same as the N-1th high-level activation feature, and the Nth low-level inverse feature is the same as the N-1th low-level activation feature. Through back propagation, the input value of a reversible attention layer (that is, the output value of the previous reversible attention layer) is inversely solved according to the output value of the reversible attention layer.

[0043] Furthermore, in reverse propagation, each reversible attention layer receives the output value of the previous layer (the output value of this layer during forward propagation), and calculates the output value of the reversible attention layer of this layer (the input value of this layer during forward propagation), that is, the high-level inverse solution feature and the low-level inverse solution feature, through the attention mechanism. The specific formula is as follows: During the reverse propagation process, X(X1, X2) is the output value of the reversible attention layer of this layer (in the forward propagation process, it is the input value of the reversible attention layer of this layer), X1 is the output low-level inverse feature or the first decomposition variable of this layer, and X2 is the output high-level inverse feature or the second decomposition variable of this layer; during the reverse propagation process, Y1 and Y2 are the input values ​​of the reversible attention layer of this layer (in the forward propagation process, they are the output values ​​of the reversible attention layer of this layer), Y1 is the input low-level inverse feature or the Nth low-level activation feature of this layer, and Y2 is the input high-level inverse feature or the Nth high-level activation feature of this layer; Attention is attention calculation, and feedforward is feedforward operation.

[0044] As an example, in step S205, after receiving the target loss value, the first high-level inverse feature and the first low-level inverse feature output by the last reversible attention layer (the first reversible attention layer) of reverse propagation, the server obtains the first target gradient value based on the target loss value, the first high-level inverse feature and the first low-level inverse feature for updating the original parameters of the first layer in the original neural network model.

[0045] In this example, the original parameters of the first layer in the original neural network model are not updated only by the target loss value, the first high-level inverse solution feature, and the first low-level inverse solution feature to complete the update of the original neural network model. Instead, in the reverse propagation process, the original parameters corresponding to each reverse neural network layer are updated according to the high-level inverse solution features and low-level inverse solution features corresponding to each reverse neural network layer, combined with the target loss value. When the last reversible attention layer (the first reversible attention layer) of the reverse propagation is completed, the update of the original parameters of the last reversible attention layer (the first reversible attention layer) is completed, and the update of the entire original neural network model is completed.

[0046] In this example, after decomposing the initial time series data into the first decomposition variable and the second decomposition variable, the N reversible attention layers in the original neural network model are loaded in sequence, the first decomposition variable and the second decomposition variable are propagated forward, and the Nth high-level activation feature and the Nth low-level activation feature are predicted to complete the prediction processing of the initial time series data; the N reversible attention layers in the original neural network model are loaded in sequence, the Nth high-level activation feature, the Nth low-level activation feature and the target loss value calculated according to the Nth high-level activation feature are reversely propagated, and the high-level inverse solution feature and the low-level inverse solution feature corresponding to each reversible attention layer are calculated in sequence, so that the original neural network model can be updated without keeping each reversible attention layer loaded, thereby improving the applicability of the neural network model to edge devices. The problems of large model size, high computing cost, energy consumption, poor real-time requirements and high transmission cost when deploying neural network models on edge devices are solved. When the neural network model is deployed on the edge device, the low latency and high security requirements of fraud detection, risk assessment and personalized finance in the financial field are met.

[0047] In one embodiment, if Figure 3 As shown, in step S203, N reversible attention layers in the original neural network model are loaded in sequence, and forward propagation is performed on the first decomposition variable and the second decomposition variable to obtain the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable, including: S301, loading the first reversible attention layer in the original neural network model, performing forward calculation on the first decomposition variable and the second decomposition variable, and outputting the first high-level activation feature corresponding to the first decomposition variable and the first low-level activation feature corresponding to the second decomposition variable; S302, load the i-th reversible attention layer in the original neural network model, perform forward calculation on the i-1th high-level activation feature and the i-1th low-level activation feature output by the i-1th reversible attention layer, and output the i-th high-level activation feature corresponding to the i-1th high-level activation feature and the i-th low-level activation feature corresponding to the i-1th low-level activation feature; wherein i is an integer, and 1 <i<N; S303, load the Nth reversible attention layer in the original neural network model, perform forward calculation on the N-1th high-level activation feature and the N-1th low-level activation feature output by the N-1th reversible attention layer, and output the Nth high-level activation feature corresponding to the N-1th high-level activation feature and the Nth low-level activation feature corresponding to the N-1th low-level activation feature.

[0048] Among them, the forward propagation method is to propagate from the 1st reversible attention layer, the 2nd reversible attention layer, ... the N-1th reversible attention layer to the Nth reversible attention layer in order. During the forward propagation process, if the current layer is the i-th reversible attention layer, the previous layer refers to the i-1th reversible attention layer, and the next layer refers to the i+1th reversible attention layer.

[0049] As an example, in step S301, after the initial time series data is decomposed into the first decomposition variable and the second decomposition variable, the server performs forward propagation on the first decomposition variable and the second decomposition variable, processes the input value (the first decomposition variable and the second decomposition variable) by loading the first reversible attention layer in the original neural network model, and outputs the output value, i.e., the first high-level activation feature corresponding to the first decomposition variable and the first low-level activation feature corresponding to the second decomposition variable, through the forward propagation formula mentioned in the above steps, and uses the output value of the first reversible attention layer (the first high-level activation feature and the first low-level activation feature) as the input value of the next reversible attention layer.

[0050] In this example, after the server forward loads the first reversible attention layer to process the input value of that layer and outputs the output value as the input value of the next reversible attention layer, the server loads the next reversible attention layer and stops loading the current reversible attention layer to save redundant computing power.

[0051] As an example, in step S302, after receiving the i-1th high-level activation feature and the i-1th low-level activation feature output by the i-1th reversible attention layer, the server sequentially loads the i-1th reversible attention layer, performs forward calculations on the i-1th high-level activation feature and the i-1th low-level activation feature output by the i-1th reversible attention layer, and outputs the i-1th high-level activation feature corresponding to the i-1th high-level activation feature and the i-1th low-level activation feature corresponding to the i-1th low-level activation feature as the output value of the i-1th reversible attention layer, that is, as the input value of the i+1th reversible attention layer.

[0052] As an example, in step S303, after receiving the N-1th high-level activation feature and the N-1th low-level activation feature output by the N-1th reversible attention layer, the server sequentially loads the N-1th reversible attention layer, performs forward calculations on the N-1th high-level activation feature and the N-1th low-level activation feature output by the N-1th reversible attention layer, and outputs the N-1th high-level activation feature corresponding to the N-1th high-level activation feature and the N-1th low-level activation feature corresponding to the N-1th low-level activation feature. Since the N-th reversible attention layer is the last reversible attention layer in the original neural network model, the N-th high-level activation feature and the N-th low-level activation feature are used to calculate the target loss value.

[0053] In this example, the server sequentially loads the first reversible attention layer to process the input value of this layer, and after the output value is used as the input value of the next reversible attention layer, the server loads the next reversible attention layer and stops loading the current reversible attention layer to save redundant computing power. Except for the first reversible attention layer, the input value and output value of each reversible attention layer are the output values ​​of the previous layer, so as to complete the prediction of the initial time series data according to business needs.

[0054] In one embodiment, if Figure 4 As shown, in step S301, a first source vector and a first machine vector are input into a deep learning model to obtain a first loss value, including: S401, performing attention calculation on the second decomposition variable to obtain a first forward weighted feature; merging the first forward weighted feature and the first decomposition variable to obtain a first low-level activation feature; S402, performing a feedforward operation on the first low-level activation feature to obtain a first feedforward feature; merging the first feedforward feature and the second decomposition variable to obtain a first high-level activation feature.

[0055] Among them, since two channels are preset in the original neural network according to the characteristics of the reversible neural network, the low-level activation features are processed by the low-level residual network, and the high-level activation features are processed by the high-level residual network.

[0056] It can be understood that during the forward propagation process, each reversible attention layer processes the input value of the reversible attention layer of this layer (the low-level activation features and high-level activation features output by the previous layer, or the first decomposition variable and the second decomposition variable) through the following method, and outputs the low-level activation features and high-level activation features of this layer.

[0057] As an example, in step S401, after the initial time series data is decomposed into a first decomposition variable and a second decomposition variable, the server performs forward propagation on the first decomposition variable and the second decomposition variable. The low-level residual network loaded into the first reversible attention layer processes the first decomposition variable and the second decomposition variable to obtain the first low-level activation feature.

[0058] In this example, during the forward propagation process, the low-level residual network processes the first decomposition variable and the second decomposition variable as follows: Among them, Attention(X2) means to perform attention calculation on X2 (the high-level activation feature or the second decomposition variable of the input of this layer) to obtain the weighted first forward weighted feature. The attention mechanism is used to calculate the correlation between X2 and other positions and to weight different positions in X2. Then, by adding X1 (the low-level activation feature or the first decomposition variable of the input of this layer) to the first forward weighted feature corresponding to attention(X2), X1 is kept unchanged in the calculation of the current layer and added to the attention-weighted X2 to obtain the first low-level activation feature Y1.

[0059] As an example, in step S402, after obtaining the first low-level activation feature, the server uses the high-level residual network in the first reversible attention layer to process the first low-level activation feature and the second decomposition variable to obtain the first high-level activation feature.

[0060] In this example, during the forward propagation process, the high-level residual network processes the first low-level activation feature and the second decomposition variable as follows: Among them, FeedForward(Y1) represents a feedforward operation on Y1 (the first low-level activation feature). The feedforward operation includes a fully connected layer and a subnetwork of an activation function, and performs a nonlinear transformation on Y1 (the first low-level activation feature) to obtain the first forward feedforward feature. Then, by adding X2 (the high-level activation feature or the second decomposition variable input to this layer) to the first forward feedforward feature corresponding to FeedForward(Y1), keeping X2 unchanged in the calculation of the current layer, and adding it to the first forward feedforward feature after the feedforward operation, the first high-level activation feature is obtained.

[0061] In this example, represents the forward propagation calculation function, and Represents the dot product attention mechanism. The formula of the dot product attention mechanism is as follows: Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix.

[0062] In this example, the weighted representation of X2 is calculated through the low-level activation feature or the first decomposition variable X1 of the previous layer output (this layer input) and the high-level activation feature or the second decomposition variable X2 of the previous layer output (this layer input) respectively through the attention mechanism, and it is added to X1 to obtain Y1 (the first low-level activation feature). Then Y2 (the first high-level activation feature) is obtained by performing a feedforward operation on Y1 and adding it to X2. Thus, the calculation process can be used for each layer of the Transformer model to achieve layer-by-layer transformation and encoding of features.

[0063] In one embodiment, if Figure 5 As shown, in step S204, N reversible attention layers in the original neural network model are loaded in sequence, the target loss value, the Nth high-level activation feature, and the Nth low-level activation feature are reversely propagated to obtain the first high-level inverse feature corresponding to the Nth high-level activation feature and the first low-level inverse feature corresponding to the Nth low-level activation feature, including: S501, loading the Nth reversible attention layer in the original neural network model, reversely calculating the Nth high-level activation feature and the Nth low-level activation feature, outputting the Nth high-level inverse feature corresponding to the Nth high-level activation feature and the Nth low-level inverse feature corresponding to the Nth low-level activation feature; updating the Nth layer original parameters in the Nth reversible attention layer according to the target loss value, the Nth high-level inverse feature and the Nth low-level inverse feature, and obtaining the Nth layer target parameters; S502, load the i-th reversible attention layer in the original neural network model, perform reverse calculation on the i+1-th high-level inverse solution feature and the i+1-th low-level inverse solution feature output by the i+1-th reversible attention layer, and output the i-th high-level inverse solution feature corresponding to the i+1-th high-level inverse solution feature and the i-th low-level inverse solution feature corresponding to the i+1-th low-level inverse solution feature; update the i-th layer original parameters in the i-th reversible attention layer according to the target loss value, the i-th high-level inverse solution feature and the i-th low-level inverse solution feature, and obtain the i-th layer target parameters; wherein i is an integer, and 1 <i<N; S503, load the first reversible attention layer in the original neural network model, perform reverse calculation on the second high-level inverse features and the second low-level inverse features output by the second reversible attention layer, and output the first high-level inverse features corresponding to the second high-level inverse features and the first low-level inverse features corresponding to the second low-level inverse features.

[0064] The reverse propagation method is to propagate from the Nth reversible attention layer, the N-1th reversible attention layer, ... the 2nd reversible attention layer to the 1st reversible attention layer in order. In the reverse propagation process, if the current layer is the i-th reversible attention layer, the previous layer refers to the i+1th reversible attention layer, and the next layer refers to the i-1th reversible attention layer.

[0065] It should be noted that in the process of sequential propagation of the N reversible attention layers, the parameters of the reversible attention layers to which they are propagated are updated according to the target loss value and the input and output values ​​of the reversible attention layers of this layer. When the reversible attention layers of the first layer are propagated in reverse, the update of the entire original neural network model is completed. In this example, the high-level residual network in the reversible attention layer is updated by the high-level inverse solution features, and the low-level residual network in the reversible attention layer is updated by the low-level inverse solution features, thereby completing the update of the reversible attention layer of this layer.

[0066] As an example, in step S501, after calculating the target loss value, the server performs reverse propagation on the Nth high-level activation feature and the Nth low-level activation feature, processes the input value (the Nth low-level activation feature and the Nth high-level activation feature) by loading it into the Nth reversible attention layer in the original neural network model, and outputs the output value, i.e., the Nth low-level inverse feature corresponding to the Nth low-level activation feature and the Nth high-level inverse feature corresponding to the Nth high-level activation feature, through the reverse propagation formula mentioned in the above steps, and uses the output value of the Nth reversible attention layer (the Nth low-level inverse feature and the Nth high-level inverse feature) as the input value of the next reversible attention layer.

[0067] In this example, after the server reversely loads the Nth reversible attention layer to process the input value of this layer and outputs the output value as the input value of the next reversible attention layer, the server loads the next reversible attention layer and stops loading the current reversible attention layer to save redundant computing power.

[0068] In this example, the target loss value is propagated backwards at the same time, and the target loss value is a fixed value during propagation. According to the target loss value, the Nth high-level inverse solution feature, and the Nth low-level inverse solution feature, the Nth layer original parameters in the Nth reversible attention layer are updated to obtain the Nth layer target parameters to complete the update of the Nth reversible attention layer.

[0069] As an example, in step S502, after receiving the i+1th high-level activation feature and the i+1th low-level activation feature output by the i+1th reversible attention layer, the server sequentially loads the i+1th reversible attention layer in the original neural network model, performs reverse propagation reverse calculation on the i+1th high-level inverse solution feature and the i+1th low-level inverse solution feature output by the i+1th reversible attention layer, and outputs the i+1th high-level inverse solution feature corresponding to the i+1th high-level inverse solution feature and the i+1th low-level inverse solution feature corresponding to the i+1th low-level inverse solution feature. And according to the target loss value and the i+1th high-level inverse solution feature and the i+1th low-level inverse solution feature output by this layer, the original parameters of the i+1th layer in the i+1th reversible attention layer are updated to obtain the target parameters of the i+1th layer.

[0070] As an example, in step S503, after receiving the second high-level activation feature and the second low-level activation feature output by the second reversible attention layer, the server loads the first reversible attention layer in the original neural network model, performs reverse calculation on the second high-level inverse feature and the second low-level inverse feature output by the second reversible attention layer, and outputs the first high-level inverse feature corresponding to the second high-level inverse feature and the first low-level inverse feature corresponding to the second low-level inverse feature.

[0071] In this example, the first low-level inverse solution feature is equal to the first decomposition variable at the time of input, and the first high-level inverse solution feature is equal to the second decomposition variable at the time of input.

[0072] In this example, the server sequentially loads the input value of the Nth reversible attention layer to process the input value of the layer, and after the output value is used as the input value of the next reversible attention layer, the server reversely loads the next reversible attention layer and stops loading the current reversible attention layer to save redundant computing power. Except for the Nth reversible attention layer, the input value and output value of each reversible attention layer are the output values ​​of the previous layer, so as to complete the prediction of the initial time series data according to business needs.

[0073] In one embodiment, if Figure 6 As shown, in step S501, the Nth reversible attention layer in the original neural network model is loaded, the Nth high-level activation feature and the Nth low-level activation feature are reversely calculated, and the Nth high-level inverse solution feature corresponding to the Nth high-level activation feature and the Nth low-level inverse solution feature corresponding to the Nth low-level activation feature are output, including: S601, performing a feedforward operation on the Nth low-level activation feature to obtain a first reverse feedforward feature; performing a difference calculation on the first reverse feedforward feature and the Nth high-level activation feature to obtain an Nth high-level inverse solution feature; S602, perform attention calculation on the N-th high-level inverse solution feature to obtain the first inverse weighted feature; perform difference calculation on the N-th low-level activation feature and the first inverse weighted feature to obtain the N-th low-level inverse solution feature.

[0074] Among them, since two channels are preset in the original neural network according to the characteristics of the reversible neural network, the low-level inverse features are processed by the low-level residual network, and the high-level inverse features are processed by the high-level residual network.

[0075] It can be understood that during the back propagation process, each reversible attention layer processes the input value of the reversible attention layer of this layer (the low-level inverse features and high-level inverse features output by the previous layer, or the Nth low-level activation features and the Nth high-level activation features) through the following method, and outputs the low-level inverse features and high-level inverse features of this layer.

[0076] As an example, in step S601, after calculating the target loss value, the server performs reverse propagation on the Nth high-level activation feature and the Nth low-level activation feature. The Nth reversible attention layer loaded into the original neural network model performs inverse processing on the Nth high-level activation feature and the Nth low-level activation feature to obtain the Nth high-level inverse solution feature.

[0077] In this example, during the back propagation process, the high-level residual network in the Nth reversible attention layer processes the Nth high-level activation features and the Nth low-level activation features as follows: Among them, FeedForward(Y1) represents the feedforward operation on Y1 (the Nth low-level inverse solution feature). The feedforward operation includes a sub-network of a fully connected layer and an activation function, and performs a nonlinear transformation on Y1 (the Nth low-level activation feature) to obtain the first reverse feedforward feature. Then, the difference between the first reverse feedforward feature and Y2 (the Nth high-level activation feature) is calculated to obtain the Nth high-level inverse solution feature.

[0078] As an example, in step S602, after obtaining the Nth high-level inverse-solved feature, the server uses the high-level residual network in the Nth reversible attention layer to process the Nth high-level inverse-solved feature and the Nth low-level activation feature to obtain the Nth high-level inverse-solved feature.

[0079] In this example, during the back propagation process, the low-level residual network in the Nth reversible attention layer processes the Nth high-level inverse features and the Nth low-level activation features as follows: Among them, Attention(X2) means to perform attention calculation on X2 (the Nth high-level inverse feature) to obtain the weighted first forward weighted feature. Then, the Nth low-level inverse feature is obtained by performing difference calculation on Y1 (the Nth low-level activation feature) and the first inverse weighted feature.

[0080] In this example, during the reverse propagation process, the high-level and low-level features (input values ​​in the forward propagation) are inversely solved to maintain parameter consistency. The activation values ​​restored during the reverse propagation process can ensure that during the parameter update process, the parameters of each layer can correctly reflect the optimization direction of the entire network, thereby providing accurate gradient information and ensuring the stability and convergence of parameter updates. More importantly, this method saves the memory usage of the output values ​​of each reverse neural network layer during the forward propagation process, that is, through the reverse calculation method, the memory usage is reduced, further ensuring the smooth operation of the server (edge ​​device).

[0081] In one embodiment, if Figure 7 As shown, in step S205, according to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature, the first layer original parameters in the original neural network model are updated to obtain the target neural network model, including: S701, performing gradient calculation on the target loss value, the first high-level inverse solution feature, and the first low-level inverse solution feature to obtain a first target gradient value; S702, updating the original parameters of the first layer according to the first target gradient value to obtain the target parameters of the first layer; S703. Based on the first layer target parameters, confirm the target neural network model.

[0082] It should be noted that each reversible neural layer calculates the gradient according to the following formula based on the input value and output value of this layer. The first reversible neural layer is an example and is not limited.

[0083] As an example, in step S701, after reverse propagation to the last layer (the first reversible neural layer), the server performs gradient calculation based on the target loss value, the first high-level inverse feature, and the first low-level inverse feature to obtain the first target gradient value, which is used to update the first original parameter corresponding to the first reversible neural network.

[0084] In this example, the formula for obtaining the first target gradient value is as follows: W is the weight parameter corresponding to the attention mechanism of the first reversible neural layer, X is the output value of the original parameters of the first layer (the first high-level inverse feature and the first low-level inverse feature), Y is the input value of the original parameters of the first layer (the second high-level inverse feature and the second low-level inverse feature), T represents the transpose calculation, and L is the target loss value.

[0085] Among them, ∂L / ∂W represents the gradient of L (loss function) to W (weight parameter), which is also the first target gradient value, and the first target gradient value is used to update W (weight parameter). ∂L / ∂Y represents the gradient of L (target loss value) to the input value of the first reversible neural layer (the second high-level inverse feature and the second low-level inverse feature) through back propagation. X^T represents the transposition of the output value of the original parameter of the first layer (the first high-level inverse feature and the first low-level inverse feature) to calculate the gradient of L (loss function) to W (weight parameter).

[0086] As an example, in step S702, after obtaining the first target gradient value, the server updates the original parameters of the first layer using the first target gradient value, thereby obtaining updated first layer target parameters and completing the parameter update of the first reversible neural layer.

[0087] In this example, the original parameter row update formula for layer 1 is as follows: θ_new = θ_old - learning_rate *∂L / ∂W θ_new is the updated target parameter of layer 1, θ_old is the original parameter of layer 1, and learning_rate is the learning rate.

[0088] In this example, learning_rate is a hyperparameter that controls the step size of parameter updates. For example, you can use optimization algorithms such as momentum, RMSprop, or Adam to update the model parameters.

[0089] As an example, in step S703, after the server updates the first reversible neural layer, since the first reversible neural layer is the last layer in the reverse propagation, the server completes the original parameters of the first layer of each reversible attention layer in the entire original neural network model, thereby obtaining the target neural network model.

[0090] It should be noted that each reversible attention layer is updated according to the above formula. In the reverse propagation, when the corresponding reversible attention layer is loaded, the parameter update of the reversible attention layer of this layer is completed. The first reversible attention layer is only used for example.

[0091] In this example, through back propagation, the gradient value of each reversible attention layer can be used to update the parameters of the corresponding layer, thereby optimizing the performance of the model. Back propagation is the core method of gradient calculation and parameter update in deep learning, which enables the model to automatically learn and adjust parameters to minimize the loss function. This solves the problem of updating each reversible attention layer when loading, and does not require the server (edge ​​device) to load each reversible attention layer at the same time.

[0092] In one embodiment, if Figure 8 As shown, in step S701, gradient calculation is performed on the target loss value, the first high-level inverse solution feature, and the first low-level inverse solution feature to obtain the first target gradient value, including: S801, performing gradient calculation on the target loss value and the first high-level inverse solution feature to obtain the first high-level gradient value; updating the high-level original parameters according to the first high-level gradient value to obtain the high-level target parameters; S802, performing gradient calculation on the target loss value and the first low-layer inverse solution feature to obtain the first low-layer gradient value; updating the low-layer original parameters according to the first low-layer gradient value to obtain the low-layer target parameters; S803: Acquire the first layer target parameters based on the high-level target parameters and the low-level target parameters.

[0093] Among them, the original parameters of the first layer include the original high-level parameters corresponding to the high-level residual network and the original low-level parameters corresponding to the low-level residual network.

[0094] As an example, in step S801, when the first reversible attention layer loaded by the server obtains the first high-level inverse feature, it performs gradient calculation on the target loss value and the first high-level inverse feature to obtain the first high-level gradient value, and updates the high-level original parameters according to the above formula and the first high-level gradient value to obtain the high-level target parameters and complete the update of the high-level residual network.

[0095] As an example, in step S802, when the first reversible attention layer loaded by the server obtains the first low-layer inverse feature, it performs gradient calculation on the target loss value and the first low-layer inverse feature, obtains the first low-layer gradient value, and updates the low-layer original parameters according to the above formula and the first low-layer gradient value, obtains the low-layer target parameters, and completes the update of the low-layer residual network.

[0096] As an example, in step S803, after the server has updated the low-layer residual network and the high-layer residual network respectively, that is, after obtaining the high-layer target parameters and the low-layer target parameters, it completes the update of the first reversible attention layer and obtains the first-layer target parameters.

[0097] In this example, the original parameters corresponding to each reversible attention layer include the original high-level parameters corresponding to the high-level residual network and the original low-level parameters corresponding to the low-level residual network, that is, the high-level residual network is updated using the high-level inverse-solved features, and the low-level residual network is updated using the low-level inverse-solved features to complete the final parameter update.

[0098] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0099] In one embodiment, a deep learning model training device is provided, and the deep learning model training device corresponds one-to-one to the deep learning model training method in the above embodiment. Fig. 9 As shown, the deep learning model training device includes an initial time series data acquisition module 901, an initial time series data decomposition module 902, a forward propagation module 903, a reverse propagation module 904 and a target neural network model acquisition module 905. The functional modules are described in detail as follows: An initial time series data acquisition module 901 is used to acquire initial time series data; An initial time series data decomposition module 902 is used to decompose the initial time series data to obtain a first decomposition variable and a second decomposition variable; The forward propagation module 903 is used to sequentially load the N reversible attention layers in the original neural network model, perform forward propagation on the first decomposition variable and the second decomposition variable, obtain the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable; obtain the target loss value according to the Nth high-level activation feature; wherein N is a positive integer; A reverse propagation module 904 is used to sequentially load the N reversible attention layers in the original neural network model, reversely propagate the target loss value, the Nth high-level activation feature, and the Nth low-level activation feature, and obtain the first high-level inverse solution feature corresponding to the Nth high-level activation feature and the first low-level inverse solution feature corresponding to the Nth low-level activation feature; The target neural network model acquisition module 905 is used to update the first layer original parameters in the original neural network model according to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature to obtain the target neural network model.

[0100] In one embodiment, the forward propagation module 903 includes: The first forward propagation unit is used to load the first reversible attention layer in the original neural network model, perform forward calculation on the first decomposition variable and the second decomposition variable, and output the first high-level activation feature corresponding to the first decomposition variable and the first low-level activation feature corresponding to the second decomposition variable; The i-th forward propagation unit is used to load the i-th reversible attention layer in the original neural network model, perform forward calculations on the i-1th high-level activation feature and the i-1th low-level activation feature output by the i-1th reversible attention layer, and output the i-th high-level activation feature corresponding to the i-1th high-level activation feature and the i-th low-level activation feature corresponding to the i-1th low-level activation feature; where i is an integer, and 1 <i<N; The Nth forward propagation unit is used to load the Nth reversible attention layer in the original neural network model, perform forward calculations on the N-1th high-level activation features and the N-1th low-level activation features output by the N-1th reversible attention layer, and output the Nth high-level activation features corresponding to the N-1th high-level activation features and the Nth low-level activation features corresponding to the N-1th low-level activation features.

[0101] In one embodiment, the first forward propagation unit includes: The low-level activation feature acquisition subunit is used to perform attention calculation on the second decomposition variable to obtain the first forward weighted feature; merge the first forward weighted feature and the first decomposition variable to obtain the first low-level activation feature; The high-level activation feature acquisition subunit is used to perform a feedforward operation on the first low-level activation feature to obtain a first feedforward feature; and to merge the first feedforward feature and the second decomposition variable to obtain a first high-level activation feature.

[0102] In one embodiment, the reverse propagation module 904 includes: The Nth reverse propagation unit is used to load the Nth reversible attention layer in the original neural network model, perform reverse calculation on the Nth high-level activation feature and the Nth low-level activation feature, and output the Nth high-level inverse solution feature corresponding to the Nth high-level activation feature and the Nth low-level inverse solution feature corresponding to the Nth low-level activation feature; according to the target loss value, the Nth high-level inverse solution feature and the Nth low-level inverse solution feature, the Nth layer original parameters in the Nth reversible attention layer are updated to obtain the Nth layer target parameters; The i-th reverse propagation unit is used to load the i-th reversible attention layer in the original neural network model, perform reverse calculation on the i+1-th high-level inverse solution feature and the i+1-th low-level inverse solution feature output by the i+1-th reversible attention layer, and output the i-th high-level inverse solution feature corresponding to the i+1-th high-level inverse solution feature and the i-th low-level inverse solution feature corresponding to the i+1-th low-level inverse solution feature; according to the target loss value, the i-th high-level inverse solution feature and the i-th low-level inverse solution feature, the i-th original parameter of the i-th reversible attention layer is updated to obtain the i-th target parameter; wherein i is an integer, and 1 <i<N; The first reverse propagation unit is used to load the first reversible attention layer in the original neural network model, perform reverse calculation on the second high-level inverse features and the second low-level inverse features output by the second reversible attention layer, and output the first high-level inverse features corresponding to the second high-level inverse features and the first low-level inverse features corresponding to the second low-level inverse features.

[0103] In one embodiment, the Nth reverse propagation unit includes: The high-level inverse feature acquisition subunit is used to perform a feedforward operation on the N-th low-level activation feature to obtain a first reverse feedforward feature; perform a difference calculation on the first reverse feedforward feature and the N-th high-level activation feature to obtain an N-th high-level inverse feature; The low-level inverse feature acquisition subunit is used to perform attention calculation on the N-th high-level inverse feature to obtain the first inverse weighted feature; perform difference calculation on the N-th low-level activation feature and the first inverse weighted feature to obtain the N-th low-level inverse feature.

[0104] In one embodiment, the target neural network model acquisition module 905 includes: A target gradient value acquisition unit, used for performing gradient calculation on a target loss value, a first high-level inverse solution feature, and a first low-level inverse solution feature to obtain a first target gradient value; A target parameter acquisition unit, used for updating the original parameters of the first layer according to the first target gradient value to obtain the target parameters of the first layer; The target neural network model confirmation unit is used to confirm the target neural network model based on the first layer target parameters.

[0105] In one embodiment, the target gradient value obtaining unit includes: The first layer original parameters include original high-layer parameters and original low-layer parameters; The high-level target parameter acquisition subunit is used to perform gradient calculation on the target loss value and the first high-level inverse solution feature to obtain the first high-level gradient value; according to the first high-level gradient value, the high-level original parameters are updated to obtain the high-level target parameters; The low-level target parameter acquisition subunit is used to perform gradient calculation on the target loss value and the first low-level inverse solution feature to obtain the first low-level gradient value; according to the first low-level gradient value, the low-level original parameters are updated to obtain the low-level target parameters; The layer target parameter acquisition subunit is used to acquire the first layer target parameters based on the high-level target parameters and the low-level target parameters.

[0106] For the specific limitations of the deep learning model training device, please refer to the limitations of the deep learning model training method above, which will not be repeated here. Each module in the above-mentioned deep learning model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server (edge ​​device), and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to execute the data used or generated in the process of the deep learning model training method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a deep learning model training method is implemented.

[0108] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the deep learning model training method in the above embodiment is implemented, for example Figure 2 S201-S205 as shown, or Figures 3 to 8As shown in, to avoid repetition, it will not be repeated here. Alternatively, when the processor executes the computer program, the functions of each module / unit in this embodiment of the deep learning model training device are realized, such as Fig. 9 The deep learning model training device shown includes the functions of an initial time series data acquisition module 901, an initial time series data decomposition module 902, a forward propagation module 903, a reverse propagation module 904 and a target neural network model acquisition module 905, which will not be described here to avoid repetition.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the deep learning model training method in the above embodiment is implemented, for example Figure 2 S201-S205 as shown, or Figures 3 to 8 Alternatively, when the computer program is executed by the processor, the functions of each module / unit in the embodiment of the deep learning model training device are realized, for example Fig. 9 The deep learning model training device shown includes the functions of an initial time series data acquisition module 901, an initial time series data decomposition module 902, a forward propagation module 903, a reverse propagation module 904 and a target neural network model acquisition module 905, which will not be described here to avoid repetition.

[0110] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (RambuS) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A training method for a deep learning model, characterized in that: include: Get initial time series data; Decomposing the initial time series data to obtain a first decomposition variable and a second decomposition variable; Loading N reversible attention layers in the original neural network model in sequence, forward propagating the first decomposition variable and the second decomposition variable, and obtaining the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable; Obtaining a target loss value according to the Nth high-level activation feature; wherein N is a positive integer; Loading N reversible attention layers in the original neural network model in sequence, performing reverse propagation on the target loss value, the Nth high-level activation feature, and the Nth low-level activation feature, and obtaining the first high-level inverse feature corresponding to the Nth high-level activation feature and the first low-level inverse feature corresponding to the Nth low-level activation feature; According to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature, the first layer original parameters in the original neural network model are updated to obtain a target neural network model.

2. The deep learning model training method according to claim 1, characterized in that: The N reversible attention layers in the original neural network model are sequentially loaded, the first decomposition variable and the second decomposition variable are forward propagated, and the Nth high-level activation feature corresponding to the first decomposition variable and the Nth low-level activation feature corresponding to the second decomposition variable are obtained, including: Loading the first reversible attention layer in the original neural network model, performing forward calculation on the first decomposition variable and the second decomposition variable, and outputting the first high-level activation feature corresponding to the first decomposition variable and the first low-level activation feature corresponding to the second decomposition variable; Load the i-th reversible attention layer in the original neural network model, perform forward calculation on the i-1th high-level activation feature and the i-1th low-level activation feature output by the i-1th reversible attention layer, and output the i-th high-level activation feature corresponding to the i-1th high-level activation feature and the i-th low-level activation feature corresponding to the i-1th low-level activation feature; where i is an integer, and 1 <i<N; The Nth reversible attention layer in the original neural network model is loaded, and forward calculation is performed on the N-1th high-level activation feature and the N-1th low-level activation feature output by the N-1th reversible attention layer, and the Nth high-level activation feature corresponding to the N-1th high-level activation feature and the Nth low-level activation feature corresponding to the N-1th low-level activation feature are output.

3. The deep learning model training method according to claim 2, characterized in that: The forward calculation of the first decomposition variable and the second decomposition variable to output a first high-level activation feature corresponding to the first decomposition variable and a first low-level activation feature corresponding to the second decomposition variable includes: Performing attention calculation on the second decomposition variable to obtain a first forward weighted feature; merging the first forward weighted feature and the first decomposition variable to obtain a first low-level activation feature; Perform a feedforward operation on the first low-level activation feature to obtain a first feedforward feature; and merge the first feedforward feature and the second decomposition variable to obtain a first high-level activation feature.

4. The deep learning model training method according to claim 1, characterized in that: The N reversible attention layers sequentially loaded into the original neural network model, reversely propagating the target loss value, the N-th high-level activation feature, and the N-th low-level activation feature, and obtaining the first high-level inverse feature corresponding to the N-th high-level activation feature and the first low-level inverse feature corresponding to the N-th low-level activation feature, including: The Nth reversible attention layer in the original neural network model is loaded, the Nth high-level activation feature and the Nth low-level activation feature are reversely calculated, and the Nth high-level inverse feature corresponding to the Nth high-level activation feature and the Nth low-level inverse feature corresponding to the Nth low-level activation feature are output; according to the target loss value, the Nth high-level inverse feature and the Nth low-level inverse feature, the Nth layer original parameter in the Nth reversible attention layer is updated to obtain the Nth layer target parameter; The i-th reversible attention layer in the original neural network model is loaded, and the i+1-th high-level inverse solution feature and the i+1-th low-level inverse solution feature output by the i+1-th reversible attention layer are reversely calculated, and the i-th high-level inverse solution feature corresponding to the i+1-th high-level inverse solution feature and the i-th low-level inverse solution feature corresponding to the i+1-th low-level inverse solution feature are output; according to the target loss value, the i-th high-level inverse solution feature and the i-th low-level inverse solution feature, the i-th layer original parameter in the i-th reversible attention layer is updated to obtain the i-th layer target parameter; wherein i is an integer, and 1 <i<N; The first reversible attention layer in the original neural network model is loaded, and the second high-level inverse-solution features and the second low-level inverse-solution features output by the second reversible attention layer are reversely calculated, and the first high-level inverse-solution features corresponding to the second high-level inverse-solution features and the first low-level inverse-solution features corresponding to the second low-level inverse-solution features are output.

5. The deep learning model training method according to claim 4, characterized in that: The reverse propagation of the Nth high-level activation feature and the Nth low-level activation feature to output the Nth high-level inverse solution feature corresponding to the Nth high-level activation feature and the Nth low-level inverse solution feature corresponding to the Nth low-level activation feature includes: Performing a feedforward operation on the Nth low-level activation feature to obtain a first reverse feedforward feature; performing a difference calculation on the first reverse feedforward feature and the Nth high-level activation feature to obtain an Nth high-level inverse solution feature; Performing attention calculation on the N-th high-level inverse solution feature to obtain a first inverse weighted feature; performing difference calculation on the N-th low-level activation feature and the first inverse weighted feature to obtain an N-th low-level inverse solution feature.

6. The deep learning model training method according to claim 1, characterized in that: The updating of the first layer original parameters in the original neural network model according to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature to obtain the target neural network model includes: Performing gradient calculation on the target loss value, the first high-level inverse solution feature, and the first low-level inverse solution feature to obtain a first target gradient value; According to the first target gradient value, the first layer original parameters are updated to obtain the first layer target parameters; Based on the first layer target parameters, the target neural network model is confirmed.

7. The deep learning model training method according to claim 6, characterized in that: The step of performing gradient calculation on the target loss value, the first high-level inverse solution feature, and the first low-level inverse solution feature to obtain a first target gradient value; According to the first target gradient value, the first layer original parameters are updated to obtain the first layer target parameters, including: The first layer original parameters include original high-layer parameters and original low-layer parameters; Performing gradient calculation on the target loss value and the first high-level inverse solution feature to obtain a first high-level gradient value; updating the high-level original parameters according to the first high-level gradient value to obtain a high-level target parameter; Performing gradient calculation on the target loss value and the first low-layer inverse solution feature to obtain a first low-layer gradient value; updating the low-layer original parameters according to the first low-layer gradient value to obtain a low-layer target parameter; Based on the high-level target parameters and the low-level target parameters, a first-level target parameter is obtained.

8. A deep learning model training device, characterized in that: include: An initial time series data acquisition module, used to acquire initial time series data; An initial time series data decomposition module, used to decompose the initial time series data to obtain a first decomposition variable and a second decomposition variable; A forward propagation module, used to sequentially load N reversible attention layers in the original neural network model, perform forward propagation on the first decomposition variable and the second decomposition variable, and obtain an Nth high-level activation feature corresponding to the first decomposition variable and an Nth low-level activation feature corresponding to the second decomposition variable; Obtaining a target loss value according to the Nth high-level activation feature; wherein N is a positive integer; A reverse propagation module is used to sequentially load the N reversible attention layers in the original neural network model, reversely propagate the target loss value, the N-th high-level activation feature, and the N-th low-level activation feature, and obtain the first high-level inverse solution feature corresponding to the N-th high-level activation feature and the first low-level inverse solution feature corresponding to the N-th low-level activation feature; The target neural network model acquisition module is used to update the first layer original parameters in the original neural network model according to the target loss value, the first high-level inverse solution feature and the first low-level inverse solution feature to obtain the target neural network model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the deep learning model training method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the deep learning model training method as described in any one of claims 1 to 7.

Images