Enterprise asset income time sequence prediction method based on multi-head self-attention mechanism

Abstract

The invention discloses an enterprise asset income time sequence prediction method based on a multi-head self-attention mechanism. The method comprises the following steps: collecting related data of enterprise asset operation; pre-processing the related data to obtain a pre-processed feature vector; converting the preprocessed feature vector into an embedded vector, performing position coding on the embedded vector, and adding the position code and the embedded vector to obtain a model input vector; constructing a Transform model, inputting the Transform model into a multi-layer perceptron regression layer, carrying out nonlinear mapping to a prediction result space, and constructing an enterprise asset income time sequence prediction model; establishing an online learning mechanism to perform fine adjustment on the enterprise asset income time sequence prediction model to obtain a fine-adjusted enterprise asset income time sequence prediction model; and obtaining related data of to-be-tested enterprise asset operation, and inputting the related data into the fine-tuned enterprise asset income time sequence prediction model to obtain an enterprise asset income time sequence prediction result.

Description

Technical Field

[0001] The present invention belongs to the technical field of enterprise asset operation, and particularly relates to a method for predicting the time series of enterprise asset returns based on the multi-head self-attention mechanism. Background Art

[0002] In the process of asset management in modern enterprises, how to accurately predict the operating income of enterprise assets has become a key issue for enhancing enterprise competitiveness and achieving long-term sustainable development. Traditional asset operation income predictions mostly rely on manual experience, simple trend analysis of historical data, or static models for calculation. However, these methods have obvious defects and cannot meet the dynamic and changeable market environment and complex asset operation situations, often resulting in poor accuracy of prediction results and affecting the decision-making effect of enterprises. As a result, the prediction depends on assumed steady-state conditions and cannot truly reflect the changes in the market environment. Therefore, there is an urgent need for a new method that can perform adaptive dynamic prediction of enterprise asset operation income based on market and enterprise operation data, taking into account internal and external factors, so as to improve the accuracy and real-time performance of prediction and help enterprises make more scientific and efficient asset operation decisions. Summary of the Invention

[0003] To solve the above technical problems, the present invention proposes a method for predicting the time series of enterprise asset returns based on the multi-head self-attention mechanism to achieve continuous prediction of enterprise asset operation income.

[0004] To achieve the above object, the present invention provides a method for predicting the time series of enterprise asset returns based on the multi-head self-attention mechanism, including:

[0005] Collect relevant data on enterprise asset operation, where the relevant data includes financial data, market data, policy data, and macroeconomic data;

[0006] Preprocess the relevant data to obtain preprocessed feature vectors;

[0007] Convert the preprocessed feature vectors into embedding vectors, perform positional encoding on the embedding vectors, and add the positional encoding to the embedding vectors to obtain model input vectors;

[0008] Construct a Transformer model, input the Transformer model into a multi-layer perceptron regression layer, perform non-linear mapping to the prediction result space, and construct a time series prediction model for enterprise asset returns;

[0009] Establish an online learning mechanism to fine-tune the time series prediction model for enterprise asset returns to obtain a fine-tuned time series prediction model for enterprise asset returns;

[0010] Obtain relevant data on the asset operation of the enterprise to be measured, input it into the fine-tuned time series prediction model of enterprise asset returns, and obtain the time series prediction results of enterprise asset returns.

[0011] Optionally, converting the preprocessed feature vector into an embedding vector includes:

[0012] E = X′W e + b e ;

[0013] where E is the embedding vector; X′ is the preprocessed feature vector; W e is the embedding matrix; b e is the bias vector.

[0014] Optionally, performing positional encoding on the embedding vector includes:

[0015]

[0016] where pos represents the position in the input sequence X; 2i and 2i + 1 respectively represent the dimension indices; d represents the dimension of the embedding layer.

[0017] Optionally, constructing a Transformer model includes multiple layers of Transformer encoders, and each layer includes a multi-head self-attention mechanism and a feed-forward neural network, specifically including:

[0018] In each encoding layer, perform a linear transformation on the model input vector to obtain a query vector, a key vector, and a value vector;

[0019] By calculating the dot product of the query vector and the key vector, measure the correlation between each feature, and normalize it through the Softmax function to obtain weights;

[0020] Concatenate the outputs of multiple heads and then perform a linear transformation, learn multiple relationships from different subspaces and concatenate them to obtain a concatenation result;

[0021] Perform residual connection and layer normalization on the concatenation result, and obtain the initial Transformer model through a feed-forward neural network and an activation function;

[0022] Stack several layers of the initial Transformer model to obtain the Transformer model.

[0023] Optionally, inputting the Transformer model into a multi-layer perceptron regression layer and performing a non-linear mapping to the prediction result space includes:

[0024] Process the features through a regularization layer, input them into the multi-layer perceptron regression layer to extract and combine the data, and obtain the predicted enterprise asset operation returns for a certain period in the future;

[0025] Multi - task output is achieved by setting different output nodes or independent regression layers.

[0026] Optionally, by calculating the dot product of the query vector and the key vector, the correlation between each feature is measured, and the weights are obtained through normalization by the Softmax function, including:

[0027]

[0028] where Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

[0029] Optionally, obtaining the concatenated result includes:

[0030] MultiHead(Q, K, V) = Concat(head 1 ,..., head n )W O ;

[0031] where Q is the query vector; K is the key vector; V is the value vector; n is the number of heads; W O is the output weight matrix.

[0032] Optionally, performing residual connection and layer normalization on the concatenated result, and through a feed - forward neural network and an activation function, obtaining the initial Transformer model includes:

[0033]

[0034] FFN(Z′ l ) = max(0, Z′ l W 1 +b 1 )W 2 +b 2 ;

[0035] where Z l-1 is the embedding of the input data, Z′ l is the intermediate representation after calculation by the attention mechanism, Z l is the result after completing multi - head attention and the feed - forward neural network, FFN is the feed - forward neural network, W 1 , W 2 and b 1 , b 2 are learnable weights and biases respectively.

[0036] Optionally, obtaining the predicted enterprise asset operation income for a certain period in the future includes:

[0037]

[0038] Among them, W 3 and W 4 are the weight matrices of the first and second layers respectively; b 3 and b 4 are the bias vectors of the first and second layers respectively, is the predicted asset operation income, and h is the predicted feature output by the first-layer perceptron.

[0039] Optionally, multi-task output is achieved by setting different output nodes or an independent regression layer, including: short-term income prediction:

[0040]

[0041] Medium-term income prediction:

[0042]

[0043] Long-term income prediction:

[0044]

[0045] Optionally, as a technical feature that can be added, establishing an online learning mechanism to fine-tune the enterprise asset income time series prediction model includes:

[0046]

[0047] Among them, θ is the model parameter, η is the learning rate, L is the loss function, is the predicted value of the new data, and y new is the true value of the new data.

[0048] Technical effects of the present invention: The present invention discloses an enterprise asset income time series prediction method based on a multi-head self-attention mechanism. According to multi-dimensional data related to enterprise asset operation, feature engineering such as feature selection and normalization is performed to form structured data suitable for input into the Transformer model; based on the self-attention mechanism, the dependence relationships between various factors affecting operation income in enterprise asset operation are identified, and the non-linearity and time series in the data are processed; through multi-layer perceptron regression prediction, post-processing is performed on the model output, and the prediction model is adaptively and dynamically adjusted according to the latest market and operation information to achieve continuous prediction of enterprise asset operation income. Description of the Drawings

[0049] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0050] Figure 1Schematic flowchart of a method for predicting the time series of enterprise asset returns based on the multi-head self-attention mechanism according to an embodiment of the present invention;

[0051] Figure 2 Schematic flowchart of regression prediction and multi-task output according to an embodiment of the present invention. Detailed implementation manners

[0052] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0053] It should be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0054] As Figure 1 shown, an embodiment of the present invention provides a method for predicting the time series of enterprise asset returns based on the multi-head self-attention mechanism, including:

[0055] Collect relevant data on the operation of enterprise assets, where the relevant data includes financial data, market data, policy data, and macroeconomic data. Among them, enterprise financial data: revenue, cost, net profit, production capacity, asset usage, equipment operation efficiency, etc.; external market data: market demand, industry growth rate, market share of competitors, etc.; policy data includes: industry policies issued by the government, tax policies, industry subsidy amounts, government support intensity, etc.; macroeconomic data: GDP growth rate, inflation rate, interest rate, exchange rate, etc.; perform quantization processing on the policy data, specifically including: direct quantization: directly quantize structured data such as tax rate changes and industry subsidy amounts; text quantization: for unstructured text data such as industry policies and government support policies, perform BERT sentiment analysis on it. If the analysis shows a positive impact, assign a positive real value, and if the analysis shows a negative impact, assign a negative real value;

[0056] Preprocess the relevant data to obtain preprocessed feature vectors;

[0057] Specifically, clean, interpolate, and de-duplicate the relevant data to form the feature X = [x 1 , x 2 , …, x n input to the model, where x i represents the i-th feature;

[0058] Perform normalization processing on the features, specifically expressed as:

[0059]

[0060] where x imin and x imax are the minimum and maximum values of the i-th feature respectively, and the processed feature is denoted as X'.

[0061] Convert the preprocessed feature vector into an embedding vector, perform positional encoding on the embedding vector, and add the positional encoding to the embedding vector to obtain the model input vector;

[0062] Construct a Transformer model, input the Transformer model into a multi-layer perceptron regression layer, perform non-linear mapping to the prediction result space, and construct an enterprise asset return time series prediction model;

[0063] Establish an online learning mechanism to fine-tune the enterprise asset return time series prediction model to obtain a fine-tuned enterprise asset return time series prediction model;

[0064] Obtain the relevant data of the enterprise asset operation to be measured, input it into the fine-tuned enterprise asset return time series prediction model, and obtain the enterprise asset return time series prediction result.

[0065] Furthermore, converting the preprocessed feature vector into an embedding vector includes:

[0066] Map each feature to a high-dimensional space through a linear transformation, specifically as follows:

[0067] E = X'W e + b e ;

[0068] where E is the embedding vector; X' is the preprocessed feature vector; W e is the embedding matrix; b e is the bias vector.

[0069] Furthermore, performing positional encoding on the embedding vector includes:

[0070] Use sine and cosine functions to perform positional encoding on the embedding vector, specifically expressed as:

[0071]

[0072] where pos represents the position in the input sequence X; 2i, 2i + 1 respectively represent the dimension indices; d represents the dimension of the embedding layer.

[0073] Add the positional encoding to the embedding vector to obtain the final input vector to ensure that the subsequent model can recognize the data order at different time steps, thereby capturing the time series characteristics of asset operation returns:

[0074] Z0 = E + PE.

[0075] Furthermore, constructing the Transformer model includes multiple layers of Transformer encoders, each layer including a multi-head self-attention mechanism and a feed-forward neural network, specifically including:

[0076] In each encoder layer, first, the input vector is weighted by the multi-head self-attention mechanism. The calculation steps of the multi-head self-attention are as follows:

[0077] Linear transformation: The input vector Zl-1 is respectively mapped to a query vector Q (Query), a key vector K (Key), and a value vector V (Value), specifically as follows:

[0078]

[0079] Among them, W Q , W K , W V are respectively parameter matrices for generating Q, K, and V.

[0080] Multi-head attention mechanism: The attention mechanism is used to calculate the influence degree of various operating factors on the enterprise asset operation income at the current time step. By calculating the dot product of the query vector and the key vector, the correlation between each feature is measured, and the weights are obtained through normalization by the Softmax function. The calculation formula is:

[0081]

[0082] Among them, Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

[0083] Multi-head merging: The outputs of multiple heads are concatenated and then linearly transformed to learn multiple relationships from different subspaces and concatenate them to enhance the ability to capture complex dependencies. The specific expression is as follows:

[0084] MultiHead(Q, K, V) = Concat(head 1 ,..., head n )W O ;

[0085] Among them, Q is the query vector; K is the key vector; V is the value vector; n is the number of heads; W O is the output weight matrix.

[0086] Perform residual connection and layer normalization, specifically expressed as follows:

[0087]

[0088] Among them, Z l-1 is the embedding of the input data, and Z′ l is the intermediate representation calculated by the attention mechanism. Z l is the result after completing the multi-head attention and the feed-forward neural network. FFN is the feed-forward neural network.

[0089] The feed-forward neural network consists of two fully connected layers and uses the activation function ReLU, specifically as follows:

[0090] FFN(Z′ l ) = max(0, Z′ l W 1 + b 1 )W 2 + b 2 ;

[0091] Among them, W 1 , W 2 and b 1 , b 2 are the learnable weights and biases respectively.

[0092] Stack the encoder layers L times, and the final encoder output is Z L .

[0093] Furthermore, as Figure 2 shown, input the Transformer model into the multi-layer perceptron regression layer to perform non-linear mapping to the prediction result space, including:

[0094] Process the features through the regularization layer, input them into the multi-layer perceptron regression layer to extract and combine the data, and obtain the predicted enterprise asset operation income for a certain period in the future:

[0095]

[0096] Among them, W 3 , W 4 are the weight matrices of the first and second layers respectively; b 3 , b 4 are the bias vectors of the first and second layers respectively, is the predicted asset operation income, and h is the predicted feature output by the first layer perceptron.

[0097] Achieve multi-task output by setting different output nodes or independent regression layers, specifically as follows:

[0098] Short-term income prediction:

[0099]

[0100] Medium-term income prediction:

[0101]

[0102] Long-term revenue prediction:

[0103]

[0104] In response to the real-time and adaptability requirements of the prediction model, an online learning mechanism is established to enable the model to adaptively update dynamically for new data. The specific steps are as follows:

[0105] New data is input into the model in real time through the data pipeline. After passing through the same preprocessing process as the training data, a new input vector X′ is generated new 。

[0106] Use an online optimization algorithm to fine-tune the model parameters to adapt to the characteristics of the new data. The specific update formula is as follows:

[0107]

[0108] where θ is the model parameter, η is the learning rate, L is the loss function, is the predicted value of the new data, y new is the true value of the new data.

[0109] The present invention provides a method for feature extraction and preprocessing of multi-dimensional enterprise asset operation influencing factors for enterprise financial, market and policy data; an embedding vector generation and time series position encoding method based on Transformer to ensure that the model can effectively process and understand the time series characteristics of enterprise operation data, and customize a multi-head self-attention mechanism to specifically apply the attention scores to the importance weight assignment of the factors affecting enterprise asset operation revenue, enhancing the model's ability to capture complex dependencies. Establish a regression prediction framework with multi-task output to support parallel prediction of short-term, medium-term and long-term revenues, meet diverse enterprise management needs, and integrate an online learning mechanism to achieve real-time dynamic update of model parameters, ensuring the real-time and adaptability of predictions.

[0110] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for predicting enterprise asset returns based on a multi-head self-attention mechanism, characterized in that: include: Collecting data related to the asset operation of the enterprise, wherein the relevant data includes financial data, market data, policy data and macroeconomic data; Preprocessing the relevant data to obtain a preprocessed feature vector; Converting the preprocessed feature vector into an embedding vector, performing position encoding on the embedding vector, and adding the position encoding to the embedding vector to obtain a model input vector; Constructing a Transformer model, inputting the Transformer model into a multi-layer perceptron regression layer, performing nonlinear mapping to a prediction result space, and constructing a time series prediction model for enterprise asset returns; Establishing an online learning mechanism to fine-tune the enterprise asset return time series forecasting model to obtain a fine-tuned enterprise asset return time series forecasting model; Obtain relevant data on the asset operation of the enterprise to be tested, input it into the fine-tuned enterprise asset return time series forecasting model, and obtain the enterprise asset return time series forecasting results.

2. The enterprise asset return time series prediction method based on the multi-head self-attention mechanism according to claim 1 is characterized in that: Converting the preprocessed feature vector into an embedding vector includes: E=X′W e +b e ; Where E is the embedding vector; X′ is the preprocessed feature vector; W e is the embedding matrix; b e is the bias vector.

3. The enterprise asset return time series prediction method based on the multi-head self-attention mechanism according to claim 1 is characterized in that: Position encoding the embedding vector includes: Among them, pos represents the position in the input sequence X; 2i, 2i+1 represent dimension indices respectively; d represents the dimension of the embedding layer.

4. The enterprise asset return time series prediction method based on the multi-head self-attention mechanism according to claim 1 is characterized in that: The Transformer model consists of multiple layers of Transformer encoders, each of which includes a multi-head self-attention mechanism and a feedforward neural network, specifically: In each encoding layer, the model input vector is linearly transformed to obtain a query vector, a key vector, and a value vector; The correlation between the features is measured by calculating the dot product of the query vector and the key vector, and the weight is obtained by normalizing it through the Softmax function; The outputs of multiple heads are concatenated and linearly transformed, and multiple relationships are learned from different subspaces and concatenated to obtain the concatenated result; Performing residual connection and layer normalization on the splicing results, and obtaining an initial Transformer model through a feedforward neural network and an activation function; Several layers are stacked on the initial Transformer model to obtain a Transformer model.

5. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 1, characterized in that: Inputting the Transformer model into the multi-layer perceptron regression layer and performing nonlinear mapping to the prediction result space includes: The features are processed through the regularization layer and input into the multi-layer perceptron regression layer to extract and combine data to obtain the predicted enterprise asset operating income for a certain period of time in the future; Multi-task output is achieved by setting different output nodes or independent regression layers.

6. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 4, characterized in that: By calculating the dot product of the query vector and the key vector, the correlation between the features is measured, and the weights are obtained by normalizing them through the Softmax function: Among them, Q is the query vector; K is the key vector; V is the value vector; d k is the dimension of the key vector.

7. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 4, characterized in that: The splicing results include: MultiHead(Q,K,V)=Concat(head1,...,head n )W O ; Where Q is the query vector; K is the key vector; V is the value vector; n is the number of heads; W O is the output weight matrix.

8. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 4, characterized in that: The splicing results are subjected to residual connection and layer normalization, and the initial Transformer model is obtained through a feedforward neural network and an activation function, including: FFN(Z′ l )=max(0,Z′ l W1+b1)W2+b2; Among them, Z l-1 is the embedding of the input data, Z′ l is the intermediate representation calculated by the attention mechanism, Z l It is the result after completing multi-head attention and feedforward neural network. FFN is a feedforward neural network, W1, W2 and b1, b2 are learnable weights and biases respectively.

9. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 5, characterized in that: The predicted operating income of corporate assets for a certain period of time in the future includes: Among them, W3 and W4 are the weight matrices of the first and second layers respectively; b3 and b4 are the bias vectors of the first and second layers respectively. is the predicted asset operating income, and h is the predicted feature output by the first layer of perceptron.

10. The method for predicting enterprise asset returns based on a multi-head self-attention mechanism as claimed in claim 5, characterized in that: By setting different output nodes or independent regression layers, multi-task outputs are achieved, including: Short-term profit prediction: Mid-term earnings forecast: Long-term earnings forecast:

Images