A feature fusion delivery prediction method and system based on an attention gate mechanism

Abstract

The application discloses a feature fusion childbirth prediction method and system based on an attention gate mechanism, and relates to the technical field of pregnancy risk assessment.The application effectively utilizes medical text data, accurately extracts corresponding features through a Bert pre-training language model, obtains deeper features and the relationship between the features by using a bidirectional long short-term memory network and a multi-head attention mechanism, and accurately obtains a prediction score through a GTRTU gate unit, thereby improving the accuracy of the prediction effect, solving the problems of manual feature production and manual feature input in traditional methods, greatly reducing the workload of doctors, providing a reference for doctors to determine a childbirth mode, and assisting doctors in making a selection.

Description

Technical Field

[0001] This invention relates to the field of pregnancy risk assessment technology, specifically to a feature fusion method and system for predicting childbirth based on attention gating mechanisms. Background Technology

[0002] Common delivery methods are vaginal delivery and cesarean section. Typically, the choice of delivery method is made by medical professionals. If complications arise during the selection process, doctors may choose a different method for the next patient, potentially neglecting the mother's individual circumstances. Therefore, predicting the final delivery method based on pre-labor examination data can assist doctors in making a decision and increase the likelihood of a correct outcome.

[0003] Currently, few studies consider the prediction of delivery method. Most existing research is based on traditional machine learning, which trains a traditional machine learning classifier by inputting manually created feature vectors. However, traditional machine learning for delivery prediction has drawbacks such as data sparsity and weak feature representation capabilities, resulting in incomplete feature extraction. Furthermore, features are independent of each other and their interactions are not considered, which may lead to poor prediction results and fail to provide accurate predictions to assist doctors in making judgments. Moreover, manually collecting and inputting features requires a significant amount of manpower and resources. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a feature fusion-based labor prediction method and system based on an attention gating mechanism. This solves the problems of traditional machine learning for labor prediction, which fails to consider the relationships between features and suffers from data sparsity and weak feature representation capabilities, resulting in incomplete feature information extraction and poor prediction performance.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] A feature fusion-based childbirth prediction method based on an attention gating mechanism is provided, which includes the following steps:

[0007] S1. Obtain the original text dataset of the mothers and filter it to obtain the filtered text dataset;

[0008] S2. Construct a childbirth prediction model and input the filtered text dataset into the childbirth prediction model to obtain a prediction score.

[0009] Furthermore, the original text dataset of the mother in step S1 includes an ultrasound examination report; the ultrasound examination report includes fetal position, head circumference, abdominal circumference, amniotic fluid status, and fetal heart rate data.

[0010] Furthermore, the specific process of filtering in step S1 is as follows: remove text data with a length of less than 200 from the original text dataset and retain text data with a length of not less than 200.

[0011] Furthermore, the labor prediction model in step S2 includes a feature extraction module, a representation extraction module, a multi-head attention layer, a GTRTU gating unit, and a data stitching layer connected in series. The feature extraction module uses a BERT pre-trained language model. The representation extraction module uses a bidirectional long short-term memory network. The bidirectional long short-term memory network includes t forward memory units. and t reverse memory units The data splicing layer includes a series fully connected layer and a softmax activation function layer; the GTRTU gate unit includes a tanh gate layer, a linear mapping layer, and an activation function layer; the tanh gate layer and the linear mapping layer are connected in parallel, and the outputs of the tanh gate layer and the linear mapping layer are multiplied by a dot product before being input to the activation function layer.

[0012] Furthermore, step S2 further includes:

[0013] S2-1. Input the filtered text dataset into the BERT pre-trained language model to obtain text word vector representations and global semantic information;

[0014] S2-2. Input the text word vector representation into a bidirectional long short-term memory network to obtain the semantic representation data of the upper and lower parts;

[0015] S2-3. Input the upper and lower semantic representation data and global semantic information into the multi-head attention layer to obtain the interactive feature data;

[0016] S2-4. Input the interactive feature data into the GTRTU gate unit to obtain the fused feature data;

[0017] S2-5. Input the upper and lower semantic representation data, global semantic information and fused feature data into the data splicing layer to obtain the prediction score.

[0018] Furthermore, the formula for step S2-1 is:

[0019] b0,B l =BERT(X)

[0020]

[0021] Among them, B l Let represent the output of the l-th layer of the BERT pre-trained language model, X represent the input text data (i.e., the filtered text dataset), BERT(·) represent the function of the BERT pre-trained language model, B represent the text word vector representation, and α represent the output of the l-th layer of the BERT pre-trained language model. lV represents the weights of the l-th layer of the BERT pre-trained language model. g b0 represents global semantic information, Σ(·) represents the summation function, and L represents the total number of layers in the BERT pre-trained language model.

[0022] Furthermore, the formula for step S2-2 is:

[0023]

[0024]

[0025] H = (h1, h2, ..., h t )

[0026]

[0027] Among them, h R This represents inverse feature data, where t represents the number of feature data with contextual semantic information, and h represents the number of feature data. L H represents the positive feature data, and H represents the output data of the bidirectional long short-term memory network, i.e., the upper and lower semantic representation data, (h1, h2, ..., h t ) represents the output data of each memory unit in a bidirectional long short-term memory network, h i This represents the output data of the i-th unit in a bidirectional long short-term memory network. This represents the output data of the i-th reverse memory unit in a bidirectional long short-term memory network. This represents the output data of each inverse memory unit in a bidirectional long short-term memory network. This represents the output data of the i-th positive unit in a bidirectional long short-term memory network. This represents the output data of each forward memory unit in the bidirectional long short-term memory network.

[0028] Furthermore, the formula for step S2-3 is:

[0029] Q = W Q I1,K=W K I1,V=W V I2

[0030]

[0031] head i =Attention(Q) i W i Q ,K i W i K V i W i V )

[0032] MultiheadAttention(Q,K,V)=Concat(head1,head2,…,head m W o

[0033] l i =MultiheadAttention(v g ,h i ,h i )

[0034] L=(l0,l1,…,l t )∈R n*d

[0035] f i =MultiheadAttention(h i ,v g ,v g )

[0036] F = (f0, f1, ..., f t )∈R n*d

[0037] Where Q, K, and V represent the query vector, key vector, and value vector of the multi-head attention layer, respectively, and W... Q W K W V Let I1 and I2 represent the weight matrices of the query vector Q, key vector K, and value vector V, respectively. Let softmax(·) represent the normalization function, and K represent the input matrices. T d represents the transpose of the key vector K. k The length of the input matrix is ​​represented by Q, and Attention(Q,K,V) represents the attention weights for the query vector Q, key vector K, and value vector V. i K i V i W represents the query vector, key vector, and value vector of the i-th attention layer, respectively. i Q W i K W i V Let Q represent the query vector Q of the i-th attention layer. i Key vector K i Value vector V i The transformation matrix, Attention(Q) i W i Q ,K i W iK V i W i V ) represents the query vector Q of the i-th attention layer. i Key vector K i Value vector V i Attention weights, head i W represents the attention score of the i-th attention layer. o This represents the output transformation matrix, where m represents the number of attention layers, and Concat(head1, head2, ..., head...) m The expression ) represents the concatenated attention scores of each attention layer. MultiheadAttention(Q,K,V) represents the multihead attention score of the query vector Q, key vector K, and value vector V. i (l0,l1,...,l) t ) represent the i-th text semantic vector after interaction with the preceding and following semantic representations, and each text semantic vector, respectively. g h i Representing global semantic information and upper and lower semantic representation data respectively, MultiheadAttention(v g ,h i ,h i MultiheadAttention(h) i ,v g ,v g ) represents the corresponding multi-head attention score, L represents the contextual semantic features learned to convey contextual meaning, R represents a constant, n and d represent the number of features and the vector dimension, respectively, F represents the contextual semantic features learned to convey global meaning, and f i (f0,f1,...,f t ) represent the i-th text semantic vector and each text semantic vector after interaction with global semantic information, respectively; the feature data after interaction includes the context semantic feature L that has learned the context meaning and the context semantic feature F that has learned the global meaning.

[0038] Furthermore, the formula for step S2-4 is:

[0039] g i =relu(W g f i +U g l i +b g )

[0040] v i =tanh(W v f i +U vl i +b v )

[0041]

[0042] Z = {z1,...,z} n}

[0043] Where tanh(·) represents the activation function, relu(·) represents the linear mapping function, and v i G represents the text flow intensity obtained through tanh gating. i The l represents the text flow intensity obtained through linear gating. i Let represent the i-th contextual semantic feature that has learned the contextual meaning, fi represent the i-th contextual semantic feature that has learned the global meaning, and W represent the i-th contextual semantic feature that has learned the global meaning. g W v U g U v These represent the weights of the GTRTU gated units, b g b v The value z represents the deviation. i This represents the ith fused feature data. Indicates v i and g i Perform dot product operation, z1,…,z n Z represents all the fused feature data, i.e., the fused feature data.

[0044] A feature fusion-based childbirth prediction system based on attention gating mechanism is provided, which includes a feature extraction module, a representation extraction module, a feature interaction module, a feature fusion module, and a prediction score module.

[0045] The feature extraction module is used to extract features from the filtered text data using a BERT pre-trained language model, and to obtain text word vector representations and global semantic information.

[0046] The representation extraction module is used to extract the word vector representation of the text using a bidirectional long short-term memory network to obtain the semantic representation data above and below.

[0047] The feature interaction module is used to interact with the upper and lower semantic representation data and global semantic information using a multi-head attention mechanism to obtain the interactive feature data.

[0048] The feature fusion module is used to fuse the interactive feature data using the GTRTU gating unit to obtain the fused feature data.

[0049] The prediction score module is used to concatenate and calculate the upper and lower semantic representation data, global semantic information, and fused feature data to obtain the prediction score.

[0050] The beneficial effects of this invention are as follows: This invention effectively utilizes medical text data, accurately extracts corresponding features through a BERT pre-trained language model, and then uses a bidirectional long short-term memory network and a multi-head attention mechanism to obtain deeper features and the relationships between features. The prediction score is accurately obtained through a GTRTU gating unit, which improves the accuracy of the prediction effect. It solves the problem of manually creating and inputting features in traditional methods, greatly reduces the workload of doctors, and provides a reference for doctors to decide on the delivery method, assisting doctors in making choices. Attached Figure Description

[0051] Figure 1 This is a detailed flowchart of the present invention;

[0052] Figure 2 This is a diagram illustrating the prediction steps of the present invention;

[0053] Figure 3 This is a detailed structural diagram of the present invention;

[0054] Figure 4 This is a detailed structural diagram of the bidirectional long short-term memory network of the present invention;

[0055] Figure 5 This is a detailed structural diagram of the GTRTU gate control unit of the present invention. Detailed Implementation

[0056] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0057] like Figure 1 As shown, a feature fusion-based childbirth prediction method based on attention gating mechanism includes the following steps:

[0058] S1. Obtain the original text dataset of the mothers and filter it to obtain the filtered text dataset;

[0059] The original text dataset of the mother in step S1 includes an ultrasound examination report; the ultrasound examination report includes fetal position, head circumference, abdominal circumference, amniotic fluid status, and fetal heart rate data.

[0060] The specific process of filtering in step S1 is as follows: remove text data with a length of less than 200 from the original text dataset and keep text data with a length of not less than 200.

[0061] S2. Construct a childbirth prediction model and input the filtered text dataset into the childbirth prediction model to obtain a prediction score.

[0062] like Figure 2 , Figure 3 , Figure 4 As shown, the labor prediction model in step S2 includes a feature extraction module, a representation extraction module, a multi-head attention layer, a GTRTU gating unit, and a data stitching layer connected in series. The feature extraction module uses a BERT pre-trained language model. The representation extraction module uses a bidirectional long short-term memory network. The bidirectional long short-term memory network includes t forward memory units. and t reverse memory units The data splicing layer includes a series fully connected layer and a softmax activation function layer; the GTRTU gate unit includes a tanh gate layer, a linear mapping layer, and an activation function layer; the tanh gate layer and the linear mapping layer are connected in parallel, and the outputs of the tanh gate layer and the linear mapping layer are multiplied by a dot product before being input to the activation function layer.

[0063] like Figure 5 As shown, step S2 further includes:

[0064] S2-1. The filtered text dataset is input into the BERT pre-trained language model to obtain text word vector representations and global semantic information. The input vector of the BERT pre-trained language model consists of the sum of three vectors, with [CLS] and [SEP] added at the beginning and end of the input text data, respectively, to indicate the start and end of the text data. The three vectors are the word embedding vector, the segment embedding vector, and the position embedding vector. The word embedding vector transforms individual words in the text into vectors with fixed dimensions, and [CLS] and [SEP] are inserted at the beginning and end. The segment embedding vector processes sentence pairs in the input text, representing the semantic similarity of multiple sentences, with different sentences having different feature representations. The position embedding vector encodes the position information of words in the text into vectors. The BERT pre-trained language model is encoded by a Transformer encoder to obtain word vectors corresponding to each character. Semantic relationships are fully considered when generating word vectors, resulting in word vectors with sufficiently large dimensions, enabling the machine to obtain higher-level feature representation and understanding capabilities.

[0065] The formula for step S2-1 is:

[0066] b0,B l =BERT(X)

[0067]

[0068] Among them, B l Let represent the output of the l-th layer of the BERT pre-trained language model, X represent the input text data (i.e., the filtered text dataset), BERT(·) represent the function of the BERT pre-trained language model, B represent the text word vector representation, and α represent the output of the l-th layer of the BERT pre-trained language model. l V represents the weights of the l-th layer of the BERT pre-trained language model. g b0 represents global semantic information, ∑(·) represents the summation function, and L represents the total number of layers in the BERT pre-trained language model.

[0069] S2-2. Input the text word vector representation into a bidirectional long short-term memory (LSTM) network to obtain contextual semantic representation data. The LSTM network captures long-distance dependencies, obtaining the contextual semantic information of the medical data and solving the problem of incomplete feature extraction. The basic idea of ​​the LSTM network is to map the original input sequence into a continuous vector space through the input layer, and use forward LSTM layers and backward LSTM layers to capture the left and right contextual information of the medical text data, respectively. Finally, these are concatenated to obtain the contextual semantic representation data.

[0070] The formula for step S2-2 is:

[0071]

[0072]

[0073] H = (h1, h2, ..., h t )

[0074]

[0075] Among them, h R This represents inverse feature data, where t represents the number of feature data with contextual semantic information, and h represents the number of feature data. L H represents the positive feature data, and H represents the output data of the bidirectional long short-term memory network, i.e., the upper and lower semantic representation data, (h1, h2, ..., h t ) represents the output data of each memory unit in a bidirectional long short-term memory network, h i This represents the output data of the i-th unit in a bidirectional long short-term memory network. This represents the output data of the i-th reverse memory unit in a bidirectional long short-term memory network. This represents the output data of each inverse memory unit in a bidirectional long short-term memory network. This represents the output data of the i-th positive unit in a bidirectional long short-term memory network. This represents the output data of each forward memory unit in the bidirectional long short-term memory network.

[0076] S2-3. Input the contextual semantic representation data and global semantic information into the multi-head attention layer to obtain the interactive feature data. The basic idea of ​​the multi-head attention mechanism is to learn the importance of each word in the input information within the sentence or paragraph. Through attention convergence, the query and key are combined, and different weights are applied to different keys to achieve a preference for the value. Scaled dot product attention uses Q to perform a dot product operation on each keyword matrix K, scales the result to adjust the inner product of Q and K, and uses the softmax function to normalize and obtain the attention weights. Then, the multi-head attention mechanism performs multiple parallel attention calculations, each time obtaining a different head, and concatenates the k heads into a feature matrix that integrates attention information. Since the contextual semantic representation data and feature data are correlated, the multi-head attention mechanism can be used for semantic interaction to obtain the interactive feature data.

[0077] The formula for step S2-3 is:

[0078] Q = W Q I1,K=W K I1,V=W V I2

[0079]

[0080] head i =Attention(Q) i W i Q ,K i W i K V i W i V )

[0081] MultiheadAttention(Q,K,V)=Concat(head1,head2,…,head m W o

[0082] l i =MultiheadAttention(v g ,h i ,h i )

[0083] L=(l0,l1,…,l t )∈R n*d

[0084] f i =MultiheadAttention(h i ,v g ,v g )

[0085] F = (f0, f1, ..., f t )∈R n*d

[0086] Where Q, K, and V represent the query vector, key vector, and value vector of the multi-head attention layer, respectively, and W... Q W K W V Let Q represent the weight matrices of the query vector Q, the key vector K, and the value vector V, respectively. softmax(·) represents the normalization function, and K... T d represents the transpose of the key vector K. k The length of the input matrix is ​​represented by Q, and Attention(Q,K,V) represents the attention weights for the query vector Q, key vector K, and value vector V. i K i V i W represents the query vector, key vector, and value vector of the i-th attention layer, respectively. i Q W i K W i V Let Q represent the query vector Q of the i-th attention layer. i Key vector K i Value vector V i The transformation matrix, Attention(Q) i W i Q ,K i W i K V i W i V ) represents the query vector Q of the i-th attention layer. i Key vector K i Value vector V i Attention weights, head i W represents the attention score of the i-th attention layer. o This represents the output transformation matrix, where m represents the number of attention layers, and Concat(head1, head2, ..., head...) mThe expression ) represents the concatenated attention scores of each attention layer. MultiheadAttention(Q,K,V) represents the multihead attention score of the query vector Q, key vector K, and value vector V. i (l0,l1,…,l) t ) represent the i-th text semantic vector after interaction with the preceding and following semantic representations, and each text semantic vector, respectively. g h i Representing global semantic information and upper and lower semantic representation data respectively, MultiheadAttention(v g ,h i ,h i MultiheadAttention(h) i ,v g ,v g ) represents the corresponding multi-head attention score, L represents the contextual semantic features learned to convey contextual meaning, R represents a constant, n and d represent the number of features and the vector dimension, respectively, F represents the contextual semantic features learned to convey global meaning, and f i (f0,f1,…,f t ) represent the i-th text semantic vector and each text semantic vector after interaction with global semantic information, respectively. I1 and I2 represent the input matrices, namely the upper and lower semantic representation data and global semantic information, respectively. The upper and lower semantic representation data are used as the query vector Q and the key vector K, and the global semantic information is used as the value vector V. Attention is then calculated. The feature data after interaction includes the contextual semantic feature L that has learned the contextual meaning and the contextual semantic feature F that has learned the global meaning.

[0087] S2-4. The interactive feature data is input into the GTRTU gated unit to obtain the fused feature data. To fuse the main information of the other feature vector, GTRTU (Gate Tanh Rule Tanh Unit) is used for feature fusion. Since the Sigmoid function value is more sensitive to changes than the tanh function value, it loses sensitivity and saturates once it approaches or exceeds the range, affecting the prediction accuracy. Therefore, the tanh function is used for calculation. The GTRTU gated unit allows the gradient to easily pass through the activated unit, achieving higher accuracy. Thus, it can obtain the correlation between the two features in the text, reduce the interference of noisy data, and improve text classification performance.

[0088] The formula for step S2-4 is:

[0089] g i =relu(W g f i +U g li +b g )

[0090] v i =tanh(W v f i +U v l i +b v )

[0091]

[0092] Z = {z1,...,z} n}

[0093] Where tanh(·) represents the activation function, relu(·) represents the linear mapping function, and v i G represents the text flow intensity obtained through tanh gating. i The l represents the text flow intensity obtained through linear gating. i f represents the i-th contextual semantic feature that has learned the meaning of the context. i W represents the i-th contextual semantic feature that has learned global meaning. g W v U g U v These represent the weights of the GTRTU gated units, b g b v The value z represents the deviation. i This represents the ith fused feature data. Indicates v i and g i Perform dot product operation, z1,...,z n Z represents all the fused feature data, i.e., the fused feature data.

[0094] S2-5. Input the upper and lower semantic representation data, text word vector representation, global semantic information and fused feature data into the data splicing layer to obtain the prediction score.

[0095] A feature fusion-based childbirth prediction system based on attention gating mechanism includes a feature extraction module, a representation extraction module, a feature interaction module, a feature fusion module, and a prediction score module.

[0096] The feature extraction module is used to extract features from the filtered text data using a BERT pre-trained language model, and to obtain text word vector representations and global semantic information.

[0097] The representation extraction module is used to extract the word vector representation of the text using a bidirectional long short-term memory network to obtain the semantic representation data above and below.

[0098] The feature interaction module is used to interact with the upper and lower semantic representation data and global semantic information using a multi-head attention mechanism to obtain the interactive feature data.

[0099] The feature fusion module is used to fuse the interactive feature data using the GTRTU gating unit to obtain the fused feature data.

[0100] The prediction score module is used to concatenate and calculate the upper and lower semantic representation data, global semantic information, and fused feature data to obtain the prediction score.

[0101] In one embodiment of the present invention, the present invention selects 4710 cesarean section data points and 4710 natural childbirth data points to form an original text dataset, and then filters it. Seven prediction methods—SVM, BiLSTM, TextCNN-BiLSTM-Attention, Bert-BiLSTM-Attention, TextCNN, TextRCNN, and BERT—are compared with the present invention, and accuracy (Acc), recall, and F1 score are used as evaluation metrics.

[0102] Table 1 Evaluation index data for each method

[0103] SVM 60.30 76.01 65.69 BiLSTM 63.35 82.81 47.89 TextCNN-BiLSTM-Attention 68.86 97.85 57.82 TextCNN 71.50 88.01 63.41 TextRCNN 71.40 87.97 61.96 BERT 70.87 89.24 61.96 Bert-BiLSTM-Attention 73.20 92.61 65.29 This method 75.11 91.29 69.04

[0104] As shown in Table 1, the accuracy (Acc) of this invention is 75.11%, which is higher than the accuracy (Acc) of the other seven methods. The recall of this invention is 91.29%, which is lower than the recall of the TextCNN-BiLSTM-Attention method and the Bert-BiLSTM-Attention method, but higher than the recall of the other five methods. The F1 score of this invention is 69.04%, which is higher than the F1 score of the other seven methods. It can be seen that this invention can improve the accuracy of prediction scores, provide doctors with reference, and assist doctors in choosing the delivery method.

[0105] In summary, this invention effectively utilizes medical text data, accurately extracts corresponding features through a BERT pre-trained language model, and then uses a bidirectional long short-term memory network and multi-head attention mechanism to obtain deeper features and the relationships between features. The GTRTU gating unit accurately obtains the prediction score, improving the accuracy of the prediction. This solves the problem of manually creating and inputting features in traditional methods, greatly reduces the workload of doctors, and provides a reference for doctors to decide on the delivery method, assisting them in making choices.

Claims

1. A feature fusion-based labor prediction method based on attention gating mechanism, characterized in that: Includes the following steps: S1. Obtain the original text dataset of the mothers and filter it to obtain the filtered text dataset; S2. Construct a childbirth prediction model and input the filtered text dataset into the childbirth prediction model to obtain a prediction score; The labor prediction model in step S2 includes a feature extraction module, a representation extraction module, a multi-head attention layer, a GTRTU gating unit, and a data stitching layer connected in series. The feature extraction module uses a BERT pre-trained language model. The representation extraction module uses a bidirectional long short-term memory network. The bidirectional long short-term memory network includes... One positive memory unit and A reverse memory unit The data splicing layer includes a series fully connected layer and a softmax activation function layer; the GTRTU gate unit includes a tanh gate layer, a linear mapping layer, and an activation function layer; the tanh gate layer and the linear mapping layer are connected in parallel, and the outputs of the tanh gate layer and the linear mapping layer are multiplied by a dot product and then input to the activation function layer; Step S2 further includes: S2-1. Input the filtered text dataset into the BERT pre-trained language model to obtain text word vector representations and global semantic information; S2-2. Input the text word vector representation into a bidirectional long short-term memory network to obtain the semantic representation data of the upper and lower parts; S2-3. Input the upper and lower semantic representation data and global semantic information into the multi-head attention layer to obtain the interactive feature data; S2-4. Input the interactive feature data into the GTRTU gate unit to obtain the fused feature data; S2-5. Input the upper and lower semantic representation data, global semantic information and fused feature data into the data splicing layer to obtain the prediction score.

2. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 1, characterized in that: The original text dataset of the parturient in step S1 includes an ultrasound examination report; the ultrasound examination report includes fetal position, head circumference, abdominal circumference, amniotic fluid status, and fetal heart rate data.

3. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 1, characterized in that: The specific process of filtering in step S1 is as follows: remove text data with a length of less than 200 from the original text dataset and retain text data with a length of not less than 200.

4. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 1, characterized in that: The formula for step S2-1 is: in, This indicates the first pre-trained language model of BERT. The output of the layer, This represents the input text data, i.e., the filtered text dataset. A function representing a BERT pre-trained language model. This represents the word vector representation of the text. This represents the first... Layer weights, , Represents global semantic information. This represents the summation function. This represents the total number of layers in the BERT pre-trained language model.

5. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 1, characterized in that: The formula for step S2-2 is: in, Represents inverse feature data, This represents the number of feature data with contextual semantic information. This represents positive feature data. This represents the output data of a bidirectional long short-term memory network, i.e., the semantic representation data between the upper and lower bounds. This represents the output data of each memory unit in a bidirectional long short-term memory network. The first bidirectional long short-term memory network represents the... Output data of each unit The first bidirectional long short-term memory network represents the... The output data of each reverse memory unit This represents the output data of each inverse memory unit in a bidirectional long short-term memory network. The first bidirectional long short-term memory network represents the... The output data of each positive unit This represents the output data of each forward memory unit in the bidirectional long short-term memory network.

6. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 1, characterized in that: The formula for step S2-3 is: in, , , These represent the query vector, key vector, and value vector of the multi-head attention layer, respectively. , , These represent the query vectors respectively. Key vector and value vector The weight matrix, , Represents the input matrix, Represents the normalization function. Represents the key vector The transpose of the matrix, Indicates the length of the input matrix. Represents the query vector Key vector and value vector Attention weights , , They represent the first The query vector, key vector, and value vector of the attention layer. , , They represent the first Query vectors of the attention layer Key vector Value vector The transformation matrix, Indicates the first Query vectors of the attention layer Key vector Value vector Attention weights Indicates the first Attention scores of the attention layers This represents the output transformation matrix. Indicates the number of attention layers. This represents the concatenated result of the attention scores from each attention layer. Represents the query vector Key vector and value vector Multi-head attention score , These represent the first and second semantic representations after interaction with the preceding and following semantic representations. Individual text semantic vectors, each text semantic vector , These respectively represent global semantic information and contextual semantic representation data. , This represents the corresponding multi-head attention score. This represents the contextual semantic features learned from the context. Represents a constant. , These represent the number of features and the vector dimension, respectively. This represents the contextual semantic features learned to convey a global meaning. , These represent the first and second generations after interacting with global semantic information. The text semantic vectors and the individual text semantic vectors; the interactive feature data includes contextual semantic features that have learned the contextual meaning. and contextual semantic features that learn global meaning .

7. The feature fusion-based labor prediction method based on attention gating mechanism according to claim 6, characterized in that: The formula for step S2-4 is: in, This represents the activation function. Represents a linear mapping function. This represents the text flow intensity obtained through tanh gating. This represents the intensity of the text flow obtained through linear gating. Indicates the first A contextual semantic feature that learns the meaning of the context. Indicates the first A contextual semantic feature that learns global meaning. , , , These represent the weights of the GTRTU gated units, , Indicates the deviation value. Indicates the first The fused feature data express and Perform dot product operation. This represents all the fused feature data, i.e., the fused feature data. .

8. A prediction system based on the feature fusion labor prediction method based on the attention gating mechanism as described in any one of claims 1 to 7, characterized in that: It includes a feature extraction module, a representation extraction module, a feature interaction module, a feature fusion module, and a prediction score module; The feature extraction module is used to extract features from the filtered text data using a BERT pre-trained language model to obtain text word vector representations and global semantic information. The representation extraction module is used to extract the text word vector representation using a bidirectional long short-term memory network to obtain the semantic representation data above and below. The feature interaction module is used to interact with the upper and lower semantic representation data and global semantic information using a multi-head attention mechanism to obtain the interactive feature data. The feature fusion module is used to fuse the interactive feature data using the GTRTU gate unit to obtain the fused feature data. The prediction score module is used to concatenate and calculate the upper and lower semantic representation data, global semantic information, and fused feature data to obtain the prediction score.

Images