Multi-modal data time sequence target prediction method and system based on BLT-BiLSTM

Abstract

The invention discloses a BLT-BiLSTM-based multi-modal data time sequence target prediction method and system, and is applied to the technical field of multi-modal data fusion prediction. The method comprises the following steps: acquiring structured data, text data and audio data; performing text extraction and emotion recognition on the audio data by using an LLM model to generate audio text data, performing word vector feature extraction on the audio text data and the text data by using a Chine-BERT-wwm model, and selecting word vectors with good recognition capability; inputting the audio text features, the text features and the structured data features into a Transform model, and outputting features with a good prediction effect; and taking the fused features as an input set, and inputting the input set into a BiLSTM model for prediction. According to the method, a traditional audio, text and structured data multi-modal feature hard splicing fusion mode is optimized, a Transform model is used for feature fusion, and the accuracy of time series data prediction by using BiLSTM is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data fusion prediction, and more specifically to a multimodal data time series target prediction method and system based on BLT-BiLSTM. Background Art

[0002] In today's era of information explosion, we are surrounded by various information sources, from traditional newspapers, magazines, television and radio to modern Internet, social media, blogs and forums, and information has become more accessible than ever before. The amount of data generated globally every day is huge. According to some estimates, millions of bytes of information are created and shared every minute, covering all aspects. The breadth and depth of information are increasing, and we can quickly find information on almost any topic through search engines. At the same time, with the development of artificial intelligence and machine learning technologies, information screening and personalized recommendations have become more intelligent, helping us to filter out the most relevant and valuable content from massive data. Multimodal technology involves the fusion of multiple modal perceptions of different information storage carriers such as text, pictures, videos and audio to achieve cognition and understanding of these information. The importance of this technology lies in that it can significantly improve the efficiency and accuracy of data processing and understanding, enabling artificial intelligence to better understand the complex information of the human world. Multimodal learning refers to the use of multiple modal data (such as images, text, audio, etc.) in the same learning task to enhance the learning effect of the model. Compared with single-modal learning, multimodal learning can utilize multiple information sources and provide more comprehensive feature representation. The fusion of multimodal data can be performed at the data layer, feature layer and decision layer, and each fusion method has its specific application scenarios and advantages. For multimodal data, by integrating data from different modes, such as text, images, audio, etc., a more comprehensive insight is provided for the prediction of the model, thereby improving the accuracy and robustness of the prediction results. Especially in the face of dynamic changes or noise interference, the fusion of multimodal information is particularly important. However, the current target prediction still uses single-modal data. Therefore, how to provide a multimodal data time series target prediction method and system based on BLT-BiLSTM is a problem that technicians in this field urgently need to solve. Summary of the invention

[0003] In view of this, the present invention provides a multimodal data time series target prediction method and system based on BLT-BiLSTM, which makes up for the shortcoming of the incompleteness of the current prediction of only single-modal information, and solves the problem of increasing noise and increasing computing cost by directly concatenating multi-source heterogeneous data.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A multimodal data time series target prediction method based on BLT-BiLSTM, comprising the following steps:

[0006] S1. Obtain the required structured information data, text information data, and audio information data to obtain the original data set;

[0007] S2. Use the LLM model to convert the audio information in the original data set into audio text information composed of text and emotions; use the Chinese-BERT-wwm model to extract word vector features from the audio text information and the text information data in the original data set, retain all extracted text feature vectors, and use the features of the audio text information and the features of the text information data as a text feature data set;

[0008] S3, use the Transformer model to perform feature soft fusion screening on the text feature dataset and the structured information data in the original dataset, and retain the screened features as the prediction dataset;

[0009] S4. Arrange all feature vectors in the prediction data set by date, use the BiLSTM model to predict the target value of the time series, and obtain the final target prediction value.

[0010] Optionally, S1 specifically includes: obtaining feature information related to the prediction of the target prediction object, including structured information data, text information data, audio information data and the corresponding information generation date, wherein the audio information data contains emotional information and text information of the unsplit audio, and arranging the feature information in ascending order with the generation date as the timestamp, and combining each group of feature information with the generation date into a sample.

[0011] Optionally, in S2, the LLM model is used for the audio information data in the original data set to convert the audio information therein into audio text information composed of text and emotions. Specifically, the LLM model is first pre-trained to prepare the corresponding audio information data and emotional information. The emotional information is first input into the LLM model in the form of text, and then the audio information data is input into the LLM model. After repeated pre-training, each audio information data in the original data set is input into the LLM model as an input. Finally, the LLM model outputs the emotional information and text information of the audio information data.

[0012] Optionally, the Chinese-BERT-wwm model is used in S2 to extract word vector features from the audio text information and the text information data in the original data set. Specifically, CLS and SEP are filled on both sides of the text information, where CLS corresponds to the word vector of the first word of the input text information, the first neuron in the output layer is used to predict the category of the text information, and SEP corresponds to the word vector of the last word in the input text information, which is used to segment different sentences; then the filled text information sequence is input into the RoBERTa encoder, and the resulting hidden layer is S:

[0013] S=enc(CLS,w1,w2,w3,...,w i ,SEP)

[0014] Where enc(·) represents the RoBERTa encoder, w represents a piece of text information, i represents the number of text information, and S∈R i×1024 , only extract the CLS part as the feature word vector of the text and record it as e:

[0015] e=S [CLS]

[0016] The features of the audio text information and the features of the text information data are taken as a text feature data set, and the date is used as a timestamp. A matching method is used to combine the extracted text feature data set and the date into a text feature set sample.

[0017] Optional, S3 is:

[0018] S31, embedding the features of structured information data, audio text information, and text information data to obtain input X embed ;

[0019] S32, X embed Input into the multi-head attention mechanism in the Transformer model. At the beginning of the weight matrix W calculation, the vector matrix W corresponding to the query matrix Q, the key matrix K and the value matrix V is generated using uniform distribution initialization Q , W K and W V ;

[0020] S33, matrix W Q , W K and W V With input X embed Linear transformation calculation is performed to obtain the query matrix Q, key matrix K and value matrix V;

[0021] S34, calculating the dot product of each query matrix Q and the key matrix K to generate an original matching score matrix M;

[0022] S35, using the original matching score matrix M divided by Scaling is performed to obtain the attention score, and the attention score is normalized using softmax to obtain the attention weight. The attention weight distribution result is multiplied by the numerical matrix V to obtain the attention output Attention(Q,K,V);

[0023] S36, by calculating the single-layer attention output Attention (Q i ,K i ,V i ), i self-attention layers are calculated in parallel, and then all the self-attention layer output values ​​are connected in series. The attention mechanism of each single head is head i ;

[0024] S37, the output head of the single-layer self-attention mechanism i Splice them together and generate the final output MultiHead (Q, K, V) through a linear change;

[0025] S38, multi-head attention output MultiHead(Q,K,V) and sub-layer input X embed Get the residual connection Add;

[0026] S39, perform layer normalization on the added Add to obtain output output attn ;

[0027] S310, the forward feedback network realizes the output of the input through two linear changes and a nonlinear activation function ReLU attn The nonlinear transformation of the structured information data features, the audio text information features and the text information data features outputs a 512-dimensional feature vector Feature.

[0028] Optionally, S4 is specifically:

[0029] Set the size of a specific rolling window window_size as the time step t state and provide it to the BiLSTM model. Set the time step to 1, that is, in the BiLSTM model, only one date feature is moved down each cycle. Set the rolling window window_size to 50, use the 50-day fused extracted feature vector data to correspond to the target signal of the 51st date, and merge the 50-day fused extracted feature vector data as the input x at the time step t state. t :

[0030] x t =[Feature1,...,Feature T ]

[0031] Feature T represents the characteristic vector within the Tth date, T is 50;

[0032] The feature information saved by the forward LSTM obtained by the BiLSTM model at each time step t under each window_size is: The feature information saved by the backward LSTM at each time step t under each window_size is Finally, the feature information h at time step t is calculated t

[0033]

[0034] Where W h and W' t are the forward and reverse weight matrices in the BiLSTM network model, respectively, h is the bias parameter, the output dimension of the BiLSTM model is finally set to the dimension of the prediction target, and the final result is the prediction result.

[0035] A multimodal data time series target prediction system based on BLT-BiLSTM executes the above-mentioned multimodal data time series target prediction method based on BLT-BiLSTM, including: a data acquisition module, used to obtain required structured information data, text information data, and audio information data; a data processing module, used to process the audio information data and the text information data; a feature fusion module, used to perform feature soft fusion screening on the text feature data set and the structured information data in the original data set; and a target prediction module, used to predict the target value of the time series to obtain the final target prediction value.

[0036] It can be seen from the above technical solution that compared with the prior art, the present invention provides a multimodal data time series target prediction method and system based on BLT-BiLSTM, which has the following beneficial effects: the present invention obtains structured data, text information data and audio information data, ensures the accuracy and multimodality of the information, and compared with the single use of multimodal data, the multi-source data feature set has better recognition performance in the application of the same model; LLM is used to process and extract audio information data to obtain audio emotion information and audio text information, which can simulate information more comprehensively, and Transformer is used for feature fusion extraction, which takes into account the relationship between features, reduces noise while fusing, and has better prediction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 It is a flow chart of the multimodal data time series target prediction method of the present invention;

[0039] Figure 2 It is a schematic diagram of the multimodal data time series target prediction method of the present invention;

[0040] Figure 3 4 is a flow chart of a stock prediction method in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The embodiment of the present invention discloses a multimodal data time series target prediction method based on BLT-BiLSTM, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0043] S1. Obtain the required structured information data, text information data, and audio information data to obtain the original data set;

[0044] S2. Use the LLM model to convert the audio information in the original data set into audio text information composed of text and emotions; use the Chinese-BERT-wwm model to extract word vector features from the audio text information and the text information data in the original data set, retain all extracted text feature vectors, and use the features of the audio text information and the features of the text information data as a text feature data set;

[0045] S3, use the Transformer model to perform feature soft fusion screening on the text feature dataset and the structured information data in the original dataset, and retain the screened features as the prediction dataset;

[0046] S4. Arrange all feature vectors in the prediction data set by date, use the BiLSTM model to predict the target value of the time series, and obtain the final target prediction value.

[0047] Furthermore, S1 is specifically: obtaining feature information related to the prediction of the target prediction object, including structured information data, text information data, audio information data and the corresponding information generation date, wherein the audio information data contains the emotional information and text information of the unsplit audio, and arranging the feature information in ascending order with the generation date as the timestamp, and combining each group of feature information with the generation date into a sample.

[0048] Furthermore, in S2, the LLM model is used for the audio information data in the original data set to convert the audio information therein into audio text information composed of text and emotion. Specifically, the LLM model is first pre-trained to prepare the corresponding audio information data and emotion information. The emotion information is first input into the LLM model in the form of text, and then the audio information data is input into the LLM model. After repeated pre-training, each audio information data in the original data set is input into the LLM model as an input. Finally, the LLM model outputs the emotion information and text information of the audio information data.

[0049] Furthermore, the Chinese-BERT-wwm model is used in S2 to extract word vector features from the audio text information and the text information data in the original data set. Specifically, CLS and SEP are filled on both sides of the text information, where CLS corresponds to the word vector of the first word of the input text information, and the first neuron in the output layer is used to predict the category of the text information. SEP corresponds to the word vector of the last word in the input text information, which is used to segment different sentences. Then, the filled text information sequence is input into the RoBERTa encoder, and the resulting hidden layer is S:

[0050] S=enc(CLS,w1,w2,w3,...,w i ,SEP)

[0051] Where enc(·) represents the RoBERTa encoder, w represents a piece of text information, i represents the number of text information, and S∈R i×1024 , only extract the CLS part as the feature word vector of the text and record it as e:

[0052] e=S [CLS]

[0053] The features of the audio text information and the features of the text information data are taken as a text feature data set, and the date is used as a timestamp. A matching method is used to combine the extracted text feature data set and the date into a text feature set sample.

[0054] Furthermore, S3 is specifically:

[0055] S31, embedding the features of structured information data, audio text information, and text information data to obtain input X embed :

[0056] X embed =(X+e1)·E

[0057] Where X is the structured information data feature, e1 is the text feature dataset, which includes the features of audio text information and the features of text information data, and each feature is mapped through the embedding matrix E;

[0058] S32, X embed Input into the multi-head attention mechanism in the Transformer model. At the beginning of the weight matrix W calculation, the vector matrix W corresponding to the query matrix Q, the key matrix K and the value matrix V is generated using uniform distribution initialization Q , W K and W V :

[0059]

[0060] Among them, W is the initialization weight matrix, and three matrices are generated as W Q , W K and W V , n is the number of features in the input sample;

[0061] S33, matrix W Q , W K and W V With input X embed Linear transformation calculations are performed to obtain the query matrix Q, key matrix K, and value matrix V:

[0062] Q=X embed W Q

[0063] K=X embed W K

[0064] V=X embed W V

[0065] S34. Calculate the dot product of each query matrix Q and key matrix K to generate the original matching score matrix M:

[0066] M=Q·K T

[0067] In the formula, K T represents the transpose of the key matrix;

[0068] S35, using the original matching score matrix M divided by Scaling is performed to obtain the attention score, and the attention score is normalized using softmax to obtain the attention weight. The attention weight distribution result is multiplied by the numerical matrix V to obtain the attention output Attention(Q,K,V):

[0069]

[0070] Where, d k is the dimension of the key matrix;

[0071] S36, by calculating the single-layer attention output Attention (Q i ,K i ,V i ), i self-attention layers are calculated in parallel, and then all the self-attention layer output values ​​are connected in series. The attention mechanism of each single head is head i :

[0072]

[0073] In the formula, Q i represents the query matrix of the i-th self-attention layer, K i represents the key matrix of the i-th self-attention layer, M i represents the matching score matrix of the i-th layer of attention, dim represents the dimension of the attention head, V i A numerical matrix representing the i-th self-attention layer;

[0074] S37, the output head of the single-layer self-attention mechanism i Spliced ​​together, the final output MultiHead (Q, K, V) is generated through a linear transformation:

[0075] MultiHead(Q,K,V)=Concat(head1,...,head i )W o

[0076] Where W o is the weight matrix of output changes;

[0077] S38, after the structured information features and text information features in the sample are output by the multi-head attention, in order to prevent the diffusion of values ​​from decreasing as the weight value increases due to the network layer being too deep, the residual connection (Residual Connection) and layer normalization (LayerNormalization) are applied between the multi-head attention layer and the feedforward neural network layer, and the multi-head attention output MultiHead (Q, K, V) and the sub-layer input X embed Get the residual connection Add:

[0078] Add = X embed +MultiHead(Q,K,V)

[0079] S39, perform layer normalization on the added Add to obtain output output attn :

[0080] output attn =LayerNorm(Add)

[0081] Each sub-model in the encoder of S310 and Transformer contains a fully connected feed-forward neural network (FFN). The feed-forward network realizes the output of the input through two linear changes and a nonlinear activation function ReLU. attn Nonlinear transformation of structured information data features, audio text information features and text information data features, outputting a 512-dimensional feature vector Feature:

[0082] Feature = W2·ReLU(W1·output attn +b1)+b2

[0083] Where W1 is a linearly transformed weight matrix with a shape of (512, 2048), W2 is a linearly transformed weight matrix with a shape of (2048, 512), b1 and b2 are bias vectors, and ReLU is an activation function applied to the output of the first linear transformation;

[0084] The 1057-dimensional feature vector generates a 512-dimensional feature after passing through the 6-layer Transformer Encoder layer. The date is used as the timestamp, and the matching method is used to match the 512-dimensional feature vector extracted by Transformer fusion with the date as the final data set required for training.

[0085] Furthermore, S4 is specifically:

[0086] The BiLSTM network model contains two LSTM networks running in reverse order. The size of a specific rolling window window_size is set as the time step t state and provided to the BiLSTM model. The time step is set to 1, that is, in the BiLSTM model, only one date feature is moved down each cycle. The rolling window window_size is set to 50, and the feature vector data extracted after fusion of 50 dates is used to correspond to the target signal of the 51st date. The feature vector data extracted after fusion of 50 days is merged as the input x at the time step t state. t :

[0087] x t =[Feature1,...,Feature T ]

[0088] Feature T represents the characteristic vector within the Tth date, T is 50;

[0089] In the embodiment of the present invention, the BiLSTM model is mainly composed of an input gate i t , hidden neuron cells c t , forget gate f t and output gate o t Specifically, the forget gate is composed of the hidden state h at the last moment. t-1 and the current input x t The activation function (sigmoid) is used to calculate f t Used to represent the neuron cell state c at the previous moment t-1 The information retained in the input gate is divided into two layers. The first layer passes the hidden state h at the previous moment. t-1 and the current input x t Perform activation function (sigmoid) calculation to get i t , which represents the information stored after the input at the current time step t. The second layer uses the hidden state h at the previous moment. t-1 and the current input x t Perform activation function (tanh) calculation to obtain new candidate neuron cells state, indicating that the missing attribute information is added and supplemented, and finally the first layer result i t and candidate memory cells Calculate the output c of the input gate t , at this time, by changing the neuron cell state c in the previous time step t-1 Updated to the new neuron cell state c tThe internal cell information state is updated and finally output as the external information state through the output gate. The output gate also consists of two layers. The first layer determines the information that can be transmitted in the neuron cell state by sigmoid calculation, and the second layer processes the neuron cell state at the time step t by tanh calculation to obtain the final external state hidden layer h t The calculation process is as follows:

[0090] f t =sigmoid(W f ·[h t-1 ,x t ]+b f )

[0091] i t =sigmoid(W i ·[h t-1 ,x t ]+b i )

[0092]

[0093] o t =sigmoid(W o ·[h t-1 ,x t ]+b o )

[0094] h t =o t ·tanh(c t ),

[0095] Where W f , W i , W c and W o Represents the weight matrices of the intermediate variables of the forget gate, input gate, and output gate, respectively, b f 、b i 、b c and b o Represent the bias parameters of the forget gate, input gate and output gate respectively.

[0096] The feature information saved by the forward LSTM obtained by the BiLSTM model at each time step t under each window_size is: The feature information saved by the backward LSTM at each time step t under each window_size is Finally, the feature information h at time step t is calculated t

[0097]

[0098] Where W h and W' t are the forward and reverse weight matrices in the BiLSTM network model, respectively, h is the bias parameter, the output dimension of the BiLSTM model is finally set to the dimension of the prediction target, and the final result is the prediction result.

[0099] Further, in one embodiment of the present invention, the multimodal data time series target prediction method in the embodiment of the present invention is used to conduct experiments, and the prediction process is as follows: Figure 3 As shown, specifically:

[0100] Real stock data are collected from the A-share financial market. The data sample size includes 1,219 stock quantitative indicators, 22,542 stock news headline data, and 2,136 audio information data.

[0101] In order to compare the difference between using single-modal data and using the method of the embodiment of the present invention, three sets of training data, namely, basic quantitative indicator data, basic quantitative indicator data combined with technical indicator data as total quantitative indicators, and news headline text data, were selected for comparison with the data set after feature fusion using the BLT method of the present invention. The judgment indicators used included accuracy, Matthews correlation coefficient, and F1-score.

[0102] In order to have a more intuitive understanding of the fusion of stock quantitative index data, news headline text data and audio information, the data samples used are shown in Table 1:

[0103] Table 1

[0104] date <![CDATA[x1]]> <![CDATA[x2]]> <![CDATA[x3]]> <![CDATA[x ... ]]> <![CDATA[x 512 ]]> y 2019 / 1 / 2 0.1424189 0.0245546 0.491711 ... 0.0612054 0 2019 / 1 / 3 0.301795 0.1746840 0.785921 ... 0.0784110 0 2019 / 1 / 4 0.2807834 0.1575846 0.569665 ... -0.070987 1 2019 / 1 / 7 0.0585752 0.2384614 0.422731 ... -0.369630 1 2019 / 1 / 8 0.4583945 0.2281403 0.536410 ... -0.161663 1 2019 / 1 / 9 0.0692194 0.0036530 0.569342 ... -0.192228 0 2019 / 1 / 10 -0.201974 0.193661 0.570046 ... -0.529561 0 2019 / 1 / 11 0.3469714 0.1286998 0.528682 ... -0.218514 0 2019 / 1 / 14 0.4656852 0.1274757 0.394709 ... -0.036533 1

[0105] In Table 1, x1...x 512 The 512 feature information corresponding to the quantitative indicator features of the stock, the news headline text data and the audio text information are extracted through Transformer fusion. y represents the rise and fall trend of the stock (y=1 means the stock rises, y=0 means the stock falls or does not change), and date represents the transaction date;

[0106] The training samples and the test samples are divided into 7:3 ratios. The results of the method proposed in the embodiment of the present invention and the results of the comparative method are shown in Table 2:

[0107] Table 2

[0108]

[0109] It can be seen that the BiLSTM model achieved the best prediction results after using the BT method to integrate the total quantitative indicators, news headline text features and audio information features. At the same time, through comparison, it was found that the prediction effect of the model trained using the calculated technical indicators and basic quantitative indicators was better than that using only basic quantitative indicators. The addition of news headline text also significantly improved the prediction results.

[0110] In order to compare the effect of using Transformer to extract features after feature fusion, which is better than directly using basic quantitative indicators, news title text and audio information features, the present invention is compared with the direct use of total quantitative indicators, news text title features and audio information features. The judgment indicators used include accuracy (ACC), Matthews correlation coefficient (MCC) and F1-score. The comparison results are shown in Table 3:

[0111] Table 3

[0112] Models ACC MCC F1-score BL-BiLSTM.SUM 56.34 0.5726 53.44 BLT-BiLSTM 61.07 0.5831 58.31

[0113] It can be seen that the BiLSTM model achieved the best prediction results after using the total quantitative index, news title text features and audio information features fused by the BLT method in the embodiment of the present invention. It is superior to traditional feature splicing in feature fusion, and its accuracy is improved by about 5%.

[0114] and Figure 1 Corresponding to the method described, an embodiment of the present invention further discloses a multimodal data time series target prediction system based on BLT-BiLSTM, which executes the above-mentioned multimodal data time series target prediction method based on BLT-BiLSTM, including: a data acquisition module, used to obtain the required structured information data, text information data, and audio information data; a data processing module, used to process the audio information data and the text information data; a feature fusion module, used to perform feature soft fusion screening on the text feature data set and the structured information data in the original data set; a target prediction module, used to predict the target value of the time series to obtain the final target prediction value.

[0115] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0116] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal data time series target prediction method based on BLT-BiLSTM, characterized in that: The following steps are involved: S1. Obtain the required structured information data, text information data, and audio information data to obtain the original data set; S2. Use the LLM model to convert the audio information in the original data set into audio text information composed of text and emotions; use the Chinese-BERT-wwm model to extract word vector features from the audio text information and the text information data in the original data set, retain all extracted text feature vectors, and use the features of the audio text information and the features of the text information data as a text feature data set; S3. Use the Transformer model to perform soft fusion screening on the text feature dataset and the structured information data in the original dataset, and retain the screened features as the prediction dataset; S4. Arrange all feature vectors in the prediction data set by date, use the BiLSTM model to predict the target value of the time series, and obtain the final target prediction value.

2. According to a multimodal data time series target prediction method based on BLT-BiLSTM in claim 1, it is characterized in that: S1 is specifically: obtaining feature information related to the prediction of the target prediction object, including structured information data, text information data, audio information data and the corresponding information generation date, wherein the audio information data contains the emotional information and text information of the unsplit audio, and arranging the feature information in ascending order with the generation date as the timestamp, and combining each group of feature information with the generation date into a sample.

3. The multimodal data time series target prediction method based on BLT-BiLSTM according to claim 1, characterized in that: In S2, the LLM model is used for the audio information data in the original data set to convert the audio information therein into audio text information composed of text and emotions. Specifically, the LLM model is first pre-trained to prepare the corresponding audio information data and emotional information. The emotional information is first input into the LLM model in the form of text, and then the audio information data is input into the LLM model. After repeated pre-training, each audio information data in the original data set is input into the LLM model as an input. Finally, the LLM model outputs the emotional information and text information of the audio information data.

4. The method for predicting a multimodal data time series target based on BLT-BiLSTM according to claim 1, characterized in that: In S2, the Chinese-BERT-wwm model is used to extract word vector features from the audio text information and the text information data in the original data set. Specifically, CLS and SEP are filled on both sides of the text information, where CLS corresponds to the word vector of the first word of the input text information, the first neuron in the output layer is used to predict the category of the text information, and SEP corresponds to the word vector of the last word in the input information, which is used to segment different sentences; then the filled text information sequence is input into the RoBERTa encoder, and the resulting hidden layer is S: S=enc(CLS,w1,w2,w3,...,w i ,SEP) Where enc(·) represents the RoBERTa encoder, w represents a piece of text information, i represents the number of text information, and S∈R i×1024 , only extract the CLS part as the feature word vector of the text and record it as e: e=S [CLS] The features of the audio text information and the features of the text information data are taken as a text feature data set, and the date is used as a timestamp. A matching method is used to combine the extracted text feature data set and the date into a text feature set sample.

5. The multimodal data time series target prediction method based on BLT-BiLSTM according to claim 1, characterized in that: S3 is specifically: S31, embedding the features of structured information data, audio text information, and text information data to obtain input X embed ; S32, X embed Input into the multi-head attention mechanism in the Transformer model. At the beginning of the weight matrix W calculation, the vector matrix W corresponding to the query matrix Q, the key matrix K and the value matrix V is generated using uniform distribution initialization Q , W K and W V ; S33, matrix W Q , W K and W V With input X embed Linear transformation calculation is performed to obtain the query matrix Q, key matrix K and value matrix V; S34, calculating the dot product of each query matrix Q and the key matrix K to generate an original matching score matrix M; S35, using the original matching score matrix M divided by Scaling is performed to obtain the attention score, and the attention score is normalized using softmax to obtain the attention weight. The attention weight distribution result is multiplied by the numerical matrix V to obtain the attention output Attention(Q, K, V); S36, by calculating the single-layer attention output Attention (Q i , K i , V i ), i self-attention layers are calculated in parallel, and then all the self-attention layer output values ​​are connected in series. The attention mechanism of each single head is head i ; S37, the output head of the single-layer self-attention mechanism i Splice them together and generate the final output MultiHead (Q, K, V) through a linear change; S38, multi-head attention output MultiHead(Q, K, V) and sub-layer input X embed Get the residual connection Add; S39, perform layer normalization on the added Add to obtain output output attn ; S310, the forward feedback network realizes the output of the input through two linear changes and a nonlinear activation function ReLU attn The nonlinear transformation of the structured information data features, the audio text information features and the text information data features outputs a 512-dimensional feature vector Feature.

6. The multimodal data time series target prediction method based on BLT-BiLSTM according to claim 1, characterized in that: S4 is specifically: Set the size of a specific rolling window window_size as the time step t state and provide it to the BiLSTM model. Set the time step to 1, that is, in the BiLSTM model, only one date feature is moved down each cycle. Set the rolling window window_size to 50, use the 50-day fused extracted feature vector data to correspond to the target signal of the 51st date, and merge the 50-day fused extracted feature vector data as the input x at the time step t state. t : x t =[Feature1,...,Feature T ] Feature T represents the characteristic vector within the Tth date, where T is 50; The feature information saved by the forward LSTM obtained by the BiLSTM model at each time step t under each window_size is: The feature information saved by the backward LSTM at each time step t under each window_size is Finally, the feature information h at time step t is calculated t Where W h and W' t are the forward and reverse weight matrices in the BiLSTM network model, respectively, h is the bias parameter, the output dimension of the BiLSTM model is finally set to the dimension of the prediction target, and the final result is the prediction result.

7. A multimodal data time series target prediction system based on BLT-BiLSTM, characterized in that: The multimodal data time series target prediction method based on BLT-BiLSTM as described in any one of claims 1 to 6 includes: a data acquisition module for acquiring required structured information data, text information data, and audio information data; a data processing module for processing audio information data and text information data; a feature fusion module for performing feature soft fusion screening on structured information data in a text feature data set and an original data set; and a target prediction module for predicting a target value of a time series to obtain a final target prediction value.

Images