A similarity retrieval method for electronic medical records integrating cross-knowledge attention
By integrating the cross-knowledge attention method, combining the multi-head attention mechanism and the Transformer layer, the problem of lack of semantic information and knowledge transmission in the similarity measurement between electronic medical records is solved, the similarity search accuracy is improved, and more accurate inter-text knowledge is achieved Information delivery.
Patent Information
- Application Number
- CN202011431643.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-07
AI Technical Summary
When measuring the similarity between electronic medical record texts, the prior art lacks the consideration of word position information and semantic information, and fails to effectively integrate the interaction of knowledge transmission, resulting in insufficient similarity retrieval accuracy.
The method of fusion of cross-knowledge attention is adopted, and the construction is carried out through entity marking and knowledge model, combining the multi-head attention mechanism and the Transformer layer, the cross-attention representation between texts is calculated, and the cross-entropy loss function is used for training to enhance the transfer of knowledge information.
The accuracy of similarity search between electronic medical record texts is improved, and by effectively characterizing the mutual relationship between entities, a channel for knowledge information transmission between texts is provided, and a more accurate multi-text similarity measurement evaluation is provided.
Smart Images

Figure FDA0004910323260000011 
Figure FDA0004910323260000021 
Figure FDA0004910323260000022
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical information technology, and in particular to an electronic medical record similarity retrieval method integrating cross-knowledge attention. Background Art
[0002] In the real world, compared with images and voice formats, natural language text is becoming the main form of information transmission due to its readability and convenience. In the information age, a huge amount of text data has been accumulated, such as blog articles, online message conversations, hospital electronic medical records, online reading comprehension texts, etc. How to accurately and efficiently measure the similarity between electronic medical record texts is a major problem currently faced. The current text similarity measurement methods are as follows: Traditional machine learning methods. The TF-IDF method based on statistical information constructs a word frequency vector matrix by counting the word frequency information of individual words in the text, and then uses cosine similarity to make judgments; the LCS method based on the longest common subsequence makes similarity judgments based on the number of character matches; Latent Semantic Analysis (LSA) uses word frequency or TF-IDF as weights to represent the text as a vector in the word (topic) vector space, and makes similarity judgments through inner product operations; Probabilistic Latent Semantic Analysis (PLSA) is similar to LSA, except that topics in PLSA are unobservable; Latent Dirichlet Allocation (LDA) extends PLSA, except that LDA assumes that the prior distribution obeys the Dirichlet distribution, and uses the Gibbs sampling method to learn the model and then generate the topic distribution of the text. The above traditional machine learning methods only measure similarity in a statistical sense, without considering the position information of words in the text, and lack the semantic information measurement of words.
[0003] Neural network model method. Since 2016, neural networks have been widely used in the field of natural language text measurement. In the field of auxiliary diagnosis and treatment, in 2018, scholars from Tsinghua University extracted multi-dimensional text features through multi-layer convolutional neural networks, established the CAMSE model, and tried to predict the current diagnosis of disease codes in terms of semantics. In the field of dialogue, in 2019, Microsoft researchers pre-generated a multi-level network structure of paragraphs, sentences, and entities, and established a multi-hop dialogue model through a graph neural network to measure the similarity between the current dialogue and the linked text. However, the above model does not take into account the interaction of knowledge transfer between texts, and incorporating prior knowledge information into text similarity assessment is still a problem that needs to be solved.
[0004] The cross-attention mechanism can be applied in many fields such as natural language processing and image recognition. There are also some research results, such as the application of cross-attention in semantic segmentation, natural scene text recognition, knowledge graph question answering and sentiment analysis. However, there are currently few research results on integrating the cross-mode of knowledge attention and enhancing the transfer of knowledge information between texts. Summary of the invention
[0005] In order to overcome the deficiencies of the above technologies, the present invention provides an electronic medical record similarity retrieval method integrating cross-knowledge attention with high similarity retrieval accuracy between electronic medical record texts.
[0006] The technical solution adopted by the present invention to overcome the technical problems is:
[0007] A similarity retrieval method for electronic medical records integrating cross-knowledge attention includes the following steps:
[0008] a) Data processing: Input electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n}, set the maximum character length to L, cut off the part of the text in the input electronic medical record that exceeds L, and fill in the insufficient part with specific characters, where L and n are both positive integers;
[0009] b) Entity tag acquisition: Entity tagging is performed by using an existing entity library or a neural network model to extract entity information;
[0010] c) The electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n The electronic medical record text set {S0,S1,S2,...,S n}After entity labeling in step b), text entity label information is obtained;
[0011] d) Knowledge model construction: Entity tags are combined with the electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n} are concatenated and input into the knowledge model together to output the text set {S1,S2,...,S n The similarity probability value of each text in} compared to S0;
[0012] e) Set the maximum number of returned items M, and determine the text {S1, S2, ..., S n} outputs M similar text sequences of S0, with the similarity decreasing in sequence, where M≤n.
[0013] Furthermore, in step a), the insufficient part is supplemented by a padding method.
[0014] The value of L in a) is 250.
[0015] Furthermore, the existing entity libraries in b) are cMeKG entity library and Chinese symptom library.
[0016] Further, c) comprises the following steps:
[0017] d-1) The electronic medical record text S0 and an electronic medical record judgment text S i , (i=1,...,n) are input into the pre-trained language model to generate text embedding representations V0∈R L×E and V i ∈R L×E , where E is the output dimension of the pre-trained language model and R is the real number space;
[0018] d-2) The entity set of the text entity label information obtained in step c) is expressed as E through the formula n ={ent0,ent1,...,ent K}, K is the text set {S0,S1,S2,...,S n}, by the formula Calculate the text S i The jth word w in i,j Entity k The distance between i,j ,ent k ), get the text S i The entity position of each word in , where j≤L, such as w i,j ∈ent k , w i,j With ent k The distance between them is 0, and the word embedding weight matrix is initialized according to the maximum number of entity positions entity_size, and the initialization weight matrix dimension is R entity_size×E The entity position embedding layer Entity_Embedding, according to formula B i =Entity_Embedding(S i )Get the text S i The entity position embedding B i According to the formula B0 = Entity_Embedding (S0), we get the entity position embedding B0 of text S0. i ∈R L×E , B0∈R L×E ;
[0019] d-3) Entity j Contains n j A single character in text S i The position in (l,l+n j -1), l is the entity ent j In the text S i The starting position in V i Intercept After the Meanpooling layer, the formula ENT i,j = Meanpooling(V i [l:l+n j -1])∈R E Calculate the text entity embedding ENT i,j , through the formula P i ={ENT i,1 ENT i,2 ;...;ENT i,K}∈R K×E Concatenate the K entity embeddings to get the entity embedding P of the entire text i , through the formula ENT 0,j =Meanpooling(V0[l:l+n j -1])∈R E Calculate the entity embedding ENT 0,j , through the formula P0 = {ENT 0,1 ENT 0,2 ;...;ENT 0,K}∈R K×E Concatenate the K entity embeddings to get the overall entity embedding P0;
[0020] d-4) Through the formula Embed text into V0, embed text into V i , entity position embedding B0, entity position embedding B i , entity embedding P0 and entity embedding P i Concatenate and output dimension V0∈R (2L+K)×E , V i ∈R (2L+K)×E ;
[0021] d-5) V0 obtained in step d-4) and V i After a layer of MLP network, the output vector dimension is V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E ;
[0022] d-6) Construct a cross-knowledge attention network and transform V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E After processing by the multi-head attention mechanism, we get the cross-attention representation of V0: CrossAttention(V0)∈R (2L+K)×E and V i CrossAttention (V i )∈R (2L+K)×E,CrossAttention(V0)∈R (2L+K)×E Represented as A0, CrossAttention(V i )∈R (2L+K)×E Indicated as A i ;
[0023] d-7) Represent the cross attention A0 and A i Import the Transformer layer;
[0024] d-8) Perform Maxpooling and Meanpooling operations on the output of step d-7) in the sentence length dimension to obtain Will Four tensors are concatenated, and the output dimension is output i ∈R (2L+K)×4E ,
[0025] d-9) The cross-knowledge attention network in step d-6) is trained step by step using the stochastic gradient descent method and the cross entropy loss function.
[0026] Furthermore, during training in step d-9), two MLP network layers are used, the middle layer activation function uses the ReLU activation function, and the last layer of the network uses the softmax activation function, through the formula Cross entropy function L,y i is the true label of the text, is the label value predicted by the model, . n is the total number of samples.
[0027] The beneficial effect of the present invention is that it provides an effective mode for transferring knowledge information between electronic medical record texts by using the cross-knowledge attention method. The knowledge information comes from two parts: one is the entity information in the text, and the other is the entity location information. The entities in the electronic medical record function as a whole. The traditional method only considers the role of single words in the entity and lacks the overall interaction of the entity. The knowledge attention mechanism can well characterize the relationship between entities. The interactive mode provides a channel for the transfer of knowledge information between texts and can give an effective evaluation in terms of multi-text similarity measurement. Given the original electronic medical record text, several judgment texts and similar text labels, the similarity retrieval accuracy between electronic medical record texts is improved. DETAILED DESCRIPTION
[0028] The present invention is further described below.
[0029] A similarity retrieval method for electronic medical records integrating cross-knowledge attention includes the following steps:
[0030] a) Data processing: Input electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n}, set the maximum character length to L, cut off the part of the text in the input electronic medical record that exceeds L, and fill in the insufficient part with specific characters, where L and n are both positive integers;
[0031] b) Entity tag acquisition: Entity tagging is performed by using an existing entity library or a neural network model to extract entity information;
[0032] c) The electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n The electronic medical record text set {S0,S1,S2,...,S n}After entity labeling in step b), text entity label information is obtained;
[0033] d) Knowledge model construction: Entity tags are combined with the electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n} are concatenated and input into the knowledge model together to output the text set {S1,S2,...,S n The similarity probability value of each text in} compared to S0;
[0034] e) Set the maximum number of returned items M, and determine the text {S1, S2, ..., S n} outputs M similar text sequences of S0, with the similarity decreasing in sequence, where M≤n.
[0035] By using the cross-knowledge attention method, an effective mode of knowledge information transfer between electronic medical record texts is provided. The knowledge information comes from two sources: entity information in the text and entity location information. The entities in the electronic medical record function as a whole. Traditional methods only consider the role of single words in the entity and lack the overall interaction of the entity. The knowledge attention mechanism can well characterize the relationship between entities. The interactive mode provides a channel for knowledge information transfer between texts and can give an effective evaluation in terms of multi-text similarity measurement. Given the original electronic medical record text, several judgment texts and similar text labels, the similarity retrieval accuracy between electronic medical record texts is improved.
[0036] Further, d) comprises the following steps:
[0037] d-1) The electronic medical record text S0 and an electronic medical record judgment text S i , (i=1,...,n) are input into the pre-trained language model to generate text embedding representations V0∈RL×E and V i ∈R L×E , where E is the output dimension of the pre-trained language model and R is the real number space;
[0038] d-2) The entity set of the text entity label information obtained in step c) is expressed as E through the formula n ={ent0,ent1,...,ent K}, K is the text set {S0,S1,S2,...,S n}, by the formula Calculate the text S i The jth word w in i,j Entity k The distance between i,j ,ent k ), get the text S i The entity position of each word in , where j≤L, such as w i,j ∈ent k , w i,j With ent k The distance between them is 0, and the word embedding weight matrix is initialized according to the maximum number of entity positions entity_size, and the initialization weight matrix dimension is R entity_size×E The entity position embedding layer Entity_Embedding, according to formula B i =Entity_Embedding(S i )Get the text S i The entity position embedding B i According to the formula B0 = Entity_Embedding (S0), we get the entity position embedding B0 of text S0. i ∈R L×E , B0∈R L×E ;
[0039] d-3) Entity j Contains n j A single character in text S i The position in (l,l+n j -1), l is the entity ent j In the text S i The starting position in V i Intercept After the Meanpooling layer, the formula ENT i,j = Meanpooling(V i [l:l+n j -1])∈R ECalculate the text entity embedding ENT i,j , through the formula P i ={ENT i,1 ENT i,2 ;...;ENT i,K}∈R K×E Concatenate the K entity embeddings to get the entity embedding P of the entire text i , through the formula ENT 0,j =Meanpooling(V0[l:l+n j -1])∈R E Calculate the entity embedding ENT 0,j , through the formula P0 = {ENT 0,1 ENT 0,2 ;...;ENT 0,K}∈R K×E Concatenate the K entity embeddings to get the overall entity embedding P0;
[0040] d-4) Through the formula Embed text into V0, embed text into V i , entity position embedding B0, entity position embedding B i , entity embedding P0 and entity embedding P i Concatenate and output dimension V0∈R (2L+K)×E , V i ∈R (2L+K)×E ;
[0041] d-5) V0 obtained in step d-4) and V i After a layer of MLP network, the output vector dimension is V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E ;
[0042] d-6) Construct a cross-knowledge attention network and transform V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E After processing by the multi-head attention mechanism, we get the cross-attention representation of V0: CrossAttention(V0)∈R (2L+K)×E and V i CrossAttention (V i )∈R (2L+K)×E ,CrossAttention(V0)∈R (2L+K)×E Represented as A0, CrossAttention(V i )∈R (2L+K)×E Indicated as A i ;
[0043] d-7) Represent the cross attention A0 and A i Import the Transformer layer;
[0044] d-8) Perform Maxpooling and Meanpooling operations on the output of step d-7) in the sentence length dimension to obtain Will Four tensors are concatenated, and the output dimension is output i ∈R (2L+K)×4E ,
[0045] d-9) The cross-knowledge attention network in step d-6) is trained step by step using the stochastic gradient descent method and the cross entropy loss function.
[0046] During training in step d-9), two MLP network layers are used, the middle layer activation function uses the ReLU activation function, and the last layer of the network uses the softmax activation function. Cross entropy function L,y i is the true label of the text, is the label value predicted by the model, . n is the total number of samples.
[0047] The present invention is further described below by a specific embodiment: There are 790,899 electronic medical records in the existing medical records. The maximum number of judgment samples corresponding to each medical record text is set to 15, the maximum number of entities contained in the text set is 30, the maximum number of entity positions is 250, and the number of similar texts returned is 5. In step a), the electronic medical record text S0 and 15 electronic medical record judgment texts {S1, S2, ..., S 15}, after statistics, the maximum character length is set to 250, the part of the text that exceeds 250 is cut off, and the insufficient part is filled by "Padding method". In step b), the existing entity knowledge base such as cMeKG and Chinese symptom library is used for entity tagging. In step c), the text set {S1, S2, ..., S 15}The text entity tag is obtained in step b), that is, for one end of the text, the entity in the text is marked. In step d), the entity tag is concatenated with the original character and input into the knowledge model together, and the output text {S1, S2, ..., S 15}Compared with S0 similarity probability value.
[0048] The specific steps of step d) are:
[0049] d-1) The electronic medical record text S0 and an electronic medical record judgment text S i , (i=1,...,n) are input into the pre-trained language model to generate text embedding representations V0∈R L×E and V i ∈R L×E , where E is the output dimension of the pre-trained language model and R is the real number space. Select the text embedding dimension E as 384, the pre-trained language model is the Bert model, and input the text into the Bert model to obtain the embedding representation of each text. The output tensor dimension is R 250×384 .
[0050] d-2) The entity set of the text entity label information obtained in step c) is expressed as E through the formula n ={ent0,ent1,...,ent K}, K is the text set {S0,S1,S2,...,S n}, by the formula Calculate the text S i The jth word w in i,j Entity k The distance between i,j ,ent), get the text S i The entity position of each word in , where j≤L, such as w i,j ∈ent k , w i,j With ent k The distance between them is 0, and the word embedding weight matrix is initialized according to the maximum number of entity positions entity_size, and the initialization weight matrix dimension is R entity_size×E The entity position embedding layer Entity_Embedding, according to formula B i =Entity_Embedding(S i )Get the text S i The entity position embedding B i According to the formula B0 = Entity_Embedding (S0), we get the entity position embedding B0 of text S0. i ∈R L×E , B0∈R L×E , at this time each text output dimension is R 250×384 ;
[0051] d-3) Entity j Contains n j A single character in text S i The position in (l,l+n j -1), l is the entity entj In the text S i The starting position in V i Intercept After the Meanpooling layer, the formula ENT i,j = Meanpooling(V i [l:l+n j -1])∈R E Calculate the text entity embedding ENT i,j , through the formula P i ={ENT i,1 ENT i,2 ;...;ENT i,K}∈R K×E Concatenate the K entity embeddings to get the entity embedding P of the entire text i , through the formula ENT 0,j =Meanpooling(V0[l:l+n j -1])∈R E Calculate the entity embedding ENT 0,j , through the formula P0 = {ENT 0,1 ENT 0,2 ;...;ENT 0,K}∈R K×E Concatenate the K entity embeddings to get the overall entity embedding P0, with an output dimension of R 30×384 ;
[0052] d-4) Through the formula Embed text into V0, embed text into V i , entity position embedding B0, entity position embedding B i , entity embedding P0 and entity embedding P i Concatenate and output dimension V0∈R (2L+K)×E , V i ∈R (2L+K)×E ;, the output dimension is R 530×384 .
[0053] d-5) V0 obtained in step d-4) and V i After a layer of MLP network, the output vector dimension is V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E , the output dimension is R 530×1152 ;
[0054] d-6) Construct a cross-knowledge attention network and transform V0∈R (2L+K)×3E , V i ∈R (2L+K)×3EAfter processing by the multi-head attention mechanism, we get the cross-attention representation of V0: CrossAttention(V0)∈R (2L+K)×E and V i CrossAttention (V i )∈R (2L+K)×E ,CrossAttention(V0)∈R (2L+K)×E Represented as A0, CrossAttention(V i )∈R (2L+K)×E It is represented by A1;
[0055] d-7) Import the cross attention representations A0 and A1 into the Transformer layer;
[0056] d-8) Perform Maxpooling and Meanpooling operations on the output of step d-7) in the sentence length dimension to obtain Will Four tensors are concatenated, and the output dimension is output i ∈R (2L+K)×4E , The output dimension is R 1536 ;
[0057] d-9) The cross-knowledge attention network in step d-6) is trained step by step using the stochastic gradient descent method and the cross entropy loss function.
[0058] During training in step d-9), two MLP network layers are used, and the middle layer activation function uses the ReLU activation function, and the output dimension is R 15×512 , the last layer of the network uses the softmax activation function, through the formula Cross entropy function L,y i is the true label of the text, is the label value predicted by the model, . n is the number of all samples, and the output dimension of the final output layer is R 15 .
[0059] e) In the actual prediction process, the maximum number of similar texts returned is set to 5, and 5 similar text sequences are output from the judgment text, with the similarity decreasing in descending order.
[0060] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A similarity retrieval method for electronic medical records integrating cross-knowledge attention, characterized in that: The steps include: a) Data processing: Input electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n }, set the maximum character length to L, cut off the part of the text in the input electronic medical record that exceeds L, and fill in the insufficient part with specific characters, where L and n are both positive integers; b) Entity tag acquisition: Entity tagging is performed by using an existing entity library or a neural network model to extract entity information; c) The electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n The electronic medical record text set {S0,S1,S2,...,S n }After entity labeling in step b), text entity label information is obtained; d) Knowledge model construction: Entity tags are combined with the electronic medical record text S0 and n electronic medical record judgment texts {S1, S2, ..., S n } are concatenated and input into the knowledge model together to output the text set {S1,S2,...,S n The similarity probability value of each text in} compared to S0; e) Set the maximum number of returned items M, and determine the text {S1, S2, ..., S n Output M similar text sequences of S0 in}, with the similarity decreasing in sequence, where M≤n; Step d) comprises the following steps: d-1) The electronic medical record text S0 and an electronic medical record judgment text S i , (i=1,...,n) are input into the pre-trained language model to generate text embedding representations V0∈R L×E and V i ∈R L×E , where E is the output dimension of the pre-trained language model and R is the real number space; d-2) The entity set of the text entity label information obtained in step c) is expressed as E through the formula n ={ent0,ent1,...,ent K }, K is the text set {S0,S1,S2,...,S n }, by the formula Calculate the text S i The jth word w in i,j Entity k The distance between i,j ,ent k ), get the text S i The entity position of each word in , where j≤L, such as w i,j ∈ent k , w i,j With ent k The distance between them is 0, and the word embedding weight matrix is initialized according to the maximum number of entity positions entity_size, and the initialization weight matrix dimension is R entity_size×E The entity position embedding layer Entity_Embedding, according to formula B i =Entity_Embedding(S i )Get the text S i The entity position embedding B i According to the formula B0 = Entity_Embedding (S0), we get the entity position embedding B0 of text S0. i ∈R L×E , B0∈R L×E ; d-3) Entity j Contains n j A single character in text S i The position in (l,l+n j -1), l is the entity ent j In the text S i The starting position in V i Intercept After the Meanpooling layer, the formula ENT i,j = Meanpooling(V i [l:l+n j -1])∈R E Calculate the text entity embedding ENT i,j , through the formula P i ={ENT i,1 ENT i,2 ;...;ENT i,K }∈R K×E Concatenate the K entity embeddings to get the entity embedding P of the entire text i , through the formula ENT 0,j =Meanpooling(V0[l:l+n j -1])∈R E Calculate the entity embedding ENT 0,j , through the formula P0 = {ENT 0,1 ENT 0,2 ;...;ENT 0,K }∈R K×E Concatenate the K entity embeddings to get the overall entity embedding P0; d-4) Through the formula Embed text into V0, embed text into V i , entity position embedding B0, entity position embedding B i , entity embedding P0 and entity embedding P i Concatenate and output dimension V0∈R (2L+K)×E , V i ∈R (2L+K)×E ; d-5) V0 obtained in step d-4) and V i After a layer of MLP network, the output vector dimension is V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E ; d-6) Construct a cross-knowledge attention network and transform V0∈R (2L+K)×3E , V i ∈R (2L+K)×3E After processing by the multi-head attention mechanism, we get the cross-attention representation of V0: CrossAttention(V0)∈R (2L+K)×E and V i CrossAttention (V i )∈R (2L+K)×E ,CrossAttention(V0)∈R (2L+K)×E Represented as A0, CrossAttention(V i )∈R (2L+K)×E Indicated as A i ; d-7) Represent the cross attention A0 and A i Import the Transformer layer; d-8) Perform Maxpooling and Meanpooling operations on the output of step d-7) in the sentence length dimension to obtain Will Four tensors are concatenated, and the output dimension is output i ∈R (2L+K)×4E , d-9) The cross-knowledge attention network in step d-6) is trained step by step using the stochastic gradient descent method and the cross entropy loss function.
2. The electronic medical record similarity retrieval method integrating cross-knowledge attention according to claim 1 is characterized by: In step a), the insufficient part is supplemented by the padding method.
3. The electronic medical record similarity retrieval method integrating cross-knowledge attention according to claim 1 is characterized by: The value of L in step a) is 250.
4. The electronic medical record similarity retrieval method integrating cross-knowledge attention according to claim 1 is characterized by: The existing entity libraries in step b) are cMeKG entity library and Chinese symptom library.
5. The electronic medical record similarity retrieval method integrating cross-knowledge attention according to claim 1 is characterized by: During training in step d-9), two MLP network layers are used, the middle layer activation function uses the ReLU activation function, and the last layer of the network uses the softmax activation function. Cross entropy function L,y i is the true label of the text, is the label value predicted by the model, n is the total number of samples.
