A cross-Transformer Chinese medical named entity recognition method based on multi-source dictionary

Abstract

The present invention relates to a cross-Transformer Chinese medical named entity recognition method based on a multi-source dictionary. The present invention incorporates multi-source dictionary information and, through a feature extraction module, extracts text features enhanced by general vocabulary and text features enhanced by medical vocabulary. Then, through a cross-Transformer module, the two features obtained in the previous step are interactively enhanced and dynamically fused. Finally, the fused text features are used to predict entities in Chinese medical texts. This method can simultaneously use general dictionary information and medical field dictionary information to improve the matching rate between dictionaries and entities.

Description

Technical Field

[0001] The present invention proposes a cross-Transformer Chinese medical named entity recognition method based on a multi-source dictionary, and the technical field involved is named entity recognition. Background Art

[0002] With the widespread adoption of electronic medical records, the research and application of natural language processing technology in the medical field has attracted widespread attention. Currently, most clinical medical information is stored in unstructured (or semi-structured) text. Medical named entity recognition (NER) can extract key patient information from this text to support medical research and treatment decisions. Unlike English NER, Chinese NER lacks clear word boundaries, and different word segments may have different meanings, making the task more difficult.

[0003] There has been a lot of recent work on Chinese medical NER. NER can be viewed as a sequence labeling task. An intuitive approach is to first perform word segmentation and then tackle word-level sequence labeling. However, this approach introduces the problem of error propagation: errors in the first step of word segmentation will also lead to errors in the second step of entity recognition. Another approach is to directly tackle the character-level sequence labeling task, which generally outperforms word-based approaches. The disadvantage of this approach is that it fails to leverage word information, while word boundaries are often crucial for Chinese entity recognition. Some new research has leveraged the advantages of character-based models to incorporate word information from dictionaries into character-based models, significantly improving the effectiveness of Chinese medical NER. However, these studies only utilize information from a single dictionary for Chinese medical NER and struggle to integrate information from multiple source dictionaries to enhance Chinese medical NER. For one thing, Chinese medical texts contain a large number of entities derived from both medical terms and non-medical vocabulary. A single general-purpose dictionary can only cover a subset of these entities, limiting its potential for improving Chinese medical NER performance. On the other hand, due to the different distributions of pre-trained word vectors on different corpora, multi-source dictionaries are difficult to use directly, and it is necessary to effectively integrate the multi-source dictionary information with different vector distribution spaces.

[0004] Therefore, in order to allow the dictionary to cover more entities in Chinese medical texts and effectively integrate dictionary information with different vector distribution spaces, this patent proposes a cross-Transformer Chinese medical named entity recognition method based on multi-source dictionaries. This method can simultaneously use general dictionary information and medical field dictionary information to improve the matching rate between dictionaries and entities. In addition, the cross-attention mechanism can be used to effectively integrate multi-source dictionary information with different distribution spaces to improve the performance of Chinese medical NER. In turn, it helps doctors obtain better-quality structured information of medical texts. Summary of the Invention

[0005] To address the shortcomings of existing research, this patent proposes a cross-Transformer Chinese medical named entity recognition method based on a multi-source dictionary, which can simultaneously use general dictionary information and medical field dictionary information. To implement this method, the present invention adopts the following technical solutions:

[0006] By incorporating multi-source dictionary information, the feature extraction module extracts text features enhanced with both general vocabulary and medical vocabulary. The cross-Transformer module then interactively enhances these two features and dynamically fuses them. Finally, the fused text features are used to predict entities in Chinese medical text.

[0007] The specific implementation and training methods of the above scheme include:

[0008] S1: Input the original text into the multi-source dictionary matching module MLM to obtain the segment vector sequence H containing the universal dictionary matching vocabulary g and a sequence of segment vectors H containing matching words from the medical dictionary m , and the relative position vector R between the fragments.

[0009] S2: H g 、H m , R are input into the feature extraction module FE to obtain the text features F enhanced by common vocabulary g and medical vocabulary enhanced text features F m .

[0010] S3: F g 、F m Input to the cross Transformer module CT to obtain interactively enhanced text features F gc 、F mc .

[0011] S4: F gc 、F mc Input into the feature dynamic fusion module FGF to obtain the text feature F containing multi-source dictionary information f .

[0012] S5::F f Input into the label prediction module LP to obtain the predicted text sequence labeling results.

[0013] The present invention proposes a cross-Transformer Chinese medical entity recognition method based on a multi-source dictionary, which can use general dictionary information and medical field dictionary information at the same time. By adding multi-source dictionary information, the matching rate between the dictionary and the entity is improved. At the same time, by designing a cross-Transformer structure, the word vectors with different distributions in the multi-source dictionary can be dynamically fused. Finally, the fused multi-source dictionary features are used to predict entities in Chinese medical texts. Through the practice and application of this method, multi-source dictionary information can be effectively utilized to improve the performance of Chinese medical entity recognition. At the same time, multi-source dictionary information can also be regarded as a kind of prior knowledge. By introducing rich external knowledge information, it can also alleviate the problems of difficult annotation of Chinese medical entity recognition corpus and small number of samples in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present invention is further described with reference to the accompanying drawings and embodiments.

[0015] Figure 1 This is a flow chart of the main embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of dictionary matching in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the main framework of the embodiment provided by the present invention. DETAILED DESCRIPTION

[0018] Now combine with Figure 1 、 Figure 2 and Figure 3 , the present invention is further described in detail, comprising the following steps:

[0019] S1: Multi-source dictionary matching module MLM:

[0020] 1. Copy the original text twice and match it with the general dictionary and medical dictionary respectively. Add the matching words to the end of the text to get a new text sequence S containing the general dictionary words. g and a new text sequence S containing medical dictionary words m . Subsequent and text sequence S g Related variables will be added with superscript g, and text sequence S m Related variables are given a superscript m.

[0021] 2. The characters and words in the text are collectively referred to as segments. By retrieving the pre-trained dictionary vector table, the text sequence S can be obtained. g The vector sequence is H g , text sequence S m The vector sequence is H m .

[0022] 3. Assign two position indexes to each fragment: the position index of the first character of the fragment is head, and the position index of the last character of the fragment is tail. The relative position size h between fragment i and fragment j is obtained by calculating the difference between the head and tail of the two fragments ij ,t ij , the formula is as follows:

[0023] h ij =head[i]-head[j] (1)

[0024] t ij =tai[i]-tail[j] (2)

[0025] Where head[i] represents the head index of segment i, head[j] represents the head index of segment j, tail[i] represents the tail index of segment i, and tail[j] represents the tail index of segment j. Then, the vector representation P of the relative position size is calculated. d , the formula is as follows:

[0026]

[0027] Where d represents the relative position size, k represents the position index, and d model Represents the dimensions of the model. Represents vector P d The value when the dimension index is odd, Represents vector P d The value when the dimension index is an even number. Through formulas (3) and (4), we can get the vector representation P when the relative position size is h , the relative position size is t ij The vector representation P t After that, calculate the relative position vector R of fragment i and fragment j ij The formula is as follows:

[0028]

[0029] Where W r are randomly initialized parameters. ReLU represents the ReLU activation function, Represents vector concatenation. Parameter W r It will be automatically updated as the model is trained over multiple rounds. The number of model training rounds is usually between 50 and 100. The parameter with the highest F1 value on the validation set among the multiple rounds of training results is saved. The method for determining subsequent parameters is the same. The F1 value is calculated as follows:

[0030]

[0031] Among them, gold num Indicates the gold standard quantity, predict num Indicates the predicted quantity, correct num It indicates the number of correct predictions, precision indicates the accuracy, and recall indicates the recall rate.

[0032] By using formula (5), we can get the vector sequence H g The relative position vector of fragment i and fragment j in Vector sequence H m The relative position vector of fragment i and fragment j in S2: Feature extraction module FE

[0033] The feature extraction module FE uses the Flat Lattice Transformer model FLAT. FLAT receives the text vector sequence H obtained in the previous step g 、H m And the corresponding relative position vector Then, the segment vector sequence H is extracted by embedding the relative position of the multi-head attention mechanism. g The semantic feature F g , extract the fragment vector sequence H m The semantic feature F m . F g 、F m It includes the contextual semantic features of the text and the lexical semantic features of the text matching words. The calculation formula is as follows:

[0034] F g =softmax(A g )V g (9)

[0035] F m =softmax(A m )V m (10)

[0036]

[0037] V g =W g H g (12)

[0038] V m =W m H m (13) Among them, V g 、V m Respectively represent H g 、Hm After the linear transformation, the vector H i The vector representing the segment i, H j A represents the vector of segment j, ij A represents the attention weight of segment i and segment j. m According to formula (11), the vector sequence H m The attention weight matrix between all the segments in A. g According to formula (11), the vector sequence H g The attention weight matrix between all segments in W. g 、W m 、W q 、W k,R 、W k,H , u, and v are all randomly initialized parameters and will be automatically updated as the model is trained over multiple rounds. The number of model training rounds usually ranges from 50 to 100.

[0039] S3: Cross Transformer Module CT

[0040] The cross Transformer module CT receives the text feature F after vocabulary enhancement obtained in the previous step g 、F m In the CT module, the cross attention mechanism is first used to interact F g 、F m The information between them is used to obtain the interactively enhanced text feature F inter-gc 、F inter-mc The calculation formula is as follows:

[0041] F inter-gc =CrossAtt g→m (F g ,F m )F m (14)

[0042] F inter-mc =CrossAtt m→g (F m ,F g )F g (15)

[0043] Where CrossAtt represents the calculation of cross attention weight. g→m represents the calculation of cross attention weight. g As the query condition of CrossAtt, F m As the query field of CrossAtt. m→g means F m As the query condition of CrossAtt, F gAs the query field of CrossAtt. Then, the information after interaction enhancement is concatenated with the information before input, and the dimension is kept unchanged by linear transformation Linear to obtain the final interaction enhancement feature F gc 、F mc The formula is as follows:

[0044] F gc =Linear[F g ; F inter-gc ] (16)

[0045] F mc =Linear[F m ; F inter-mc ] (17) S4: Feature dynamic fusion module FGF

[0046] Different features have different degrees of influence in different sentences, so the feature dynamic fusion module FGF is used to dynamically select the feature F gc 、F mc The proportion retained after fusion. FGF adopts a gating mechanism and uses the gating vector V gate The text feature F after vocabulary enhancement obtained in the previous step gc 、F mc Perform weighted summation to obtain the fused feature F f The formula is as follows:

[0047] h gc =tanh(F gc W gc +b gc ) (18)

[0048] h mc =tanh(F mc W mc +b mc ) (19)

[0049] V gate =σ(([h gc ;h mc ])W gate ) (20)

[0050] F f =V gate F gc +(1-V gate )F mc (twenty one)

[0051] Among them, h gc 、h mc Respectively represent F gc 、Fmc After nonlinear transformation, the vector W gc ,W mc ,W gate ,b gc ,b mc It is a randomly initialized parameter that will be automatically updated as the model is trained over multiple rounds. The number of model training rounds usually ranges from 50 to 100. σ represents the sigmoid activation function, and tanh represents the tanh activation function.

[0052] S5: Label prediction module LP

[0053] The label prediction module uses a fully connected network and CRF layer to predict the feature F obtained in the previous step. f Perform label prediction. The fully connected network maps text features to the label distribution space and outputs label prediction probabilities. The CRF layer receives the label prediction probabilities output by the fully connected network and outputs the optimal sequence labeling result.

Claims

1. A cross-Transformer Chinese medical named entity recognition method based on multi-source dictionary, characterized by The following steps are involved: S1: Multi-source dictionary matching module MLM: 1) Copy the original text twice and match it with the general dictionary and medical dictionary respectively; concatenate the matched words at the end of the text to obtain new text sequences S containing general dictionary words. g and a new text sequence S containing medical dictionary words m ; Subsequent and text sequence S g Related variables will be added with superscript g, and text sequence S m Related variables will have a superscript m added; 2) The characters and words in the text are collectively referred to as segments, and the text sequence S is obtained by retrieving the pre-trained dictionary vector table g The vector sequence is H g , text sequence S m The vector sequence is H m ; 3) Assign two position indexes to each fragment: the position index of the first character of the fragment is head, and the position index of the last character of the fragment is tail; the relative position size h between fragment i and fragment j is obtained by calculating the difference between the head and tail of the two fragments ij ,t ij , the formula is as follows: h ij =head[i]-head[j] (1)t ij =tai[i]-tail[j] (2) Where head[i] represents the head index of fragment i, head[j] represents the head index of fragment j, tail[i] represents the tail index of fragment i, and tail[j] represents the tail index of fragment j; then, the vector representation P of the relative position size is calculated. d , the formula is as follows: Where d represents the relative position size, k represents the position index, and d model Represents the dimensions of the model; Represents vector P d The value when the dimension index is odd, Represents vector P d The value when the dimension index is an even number; through formulas (3) and (4), the vector representation P when the relative position size is obtained h , the relative position size is t ij The vector representation P t ; Then, calculate the relative position vector R of fragment i and fragment j ij ; The formula is as follows: Where W r are randomly initialized parameters; ReLU represents the ReLU activation function, Represents vector concatenation; parameter W r It will be automatically updated as the model undergoes multiple rounds of training. The number of model training rounds ranges from 50 to 100, and the parameter with the highest F1 value on the validation set among the multiple rounds of training results is saved. The method for determining subsequent parameters is the same. By using formula (5), we can get the vector sequence H g The relative position vector of fragment i and fragment j in Vector sequence H m The relative position vector of fragment i and fragment j in S2: Feature extraction module FE The feature extraction module FE uses the Flat Lattice Transformer model FLAT; FLAT receives the text vector sequence H obtained in the previous step g 、H m And the corresponding relative position vector Then, the segment vector sequence H is extracted by embedding the relative position of the multi-head attention mechanism. g The semantic feature F g , extract the fragment vector sequence H m The semantic feature F m ; F g 、F m It includes the contextual semantic features of the text and the lexical semantic features of the text-matching vocabulary; The calculation formula is as follows: F g =softmax(A g )V g (9) F m =softmax(A m )V m (10) V g =W g H g (12) V m =W m H m (13) Among them, V g 、V m Respectively represent H g 、H m After the linear transformation, the vector H i The vector representing the segment i, H j A represents the vector of segment j, ij represents the attention weight of segment i and segment j; A m According to formula (11), the vector sequence H m The attention weight matrix between all the segments in A g According to formula (11), the vector sequence H g The attention weight matrix between all segments in ; W g 、W m 、W q 、W k,R 、W k,H , u, and v are all randomly initialized parameters and will be automatically updated as the model is trained over multiple rounds. The number of model training rounds ranges from 50 to 100; S3: Cross Transformer Module CT The cross Transformer module CT receives the text feature F after vocabulary enhancement obtained in the previous step g 、F m ; In the CT module, first, the cross attention mechanism is used to interact F g 、F m The information between them is used to obtain the interactively enhanced text feature F inter-gc 、F inter-mc The calculation formula is as follows: F inter-gc =CrossAtt g→m (F g ,F m )F m (14) F inter-mc =CrossAtt m→g (F m ,F g )F g (15) Where CrossAtt represents the calculation of cross attention weight; g→m represents the calculation of cross attention weight; g As the query condition of CrossAtt, F m As the query field of CrossAtt; m→g means F m As the query condition of CrossAtt, F g As the query field of CrossAtt; then, the information after interaction enhancement is concatenated with the information before input, and the dimension is kept unchanged by linear transformation Linear to obtain the final interaction enhancement feature F gc 、F mc ; The formula is as follows: F gc =Linear[F g ;F inter-gc ] (16) F mc =Linear[F m ;F inter-mc ] (17) S4: Feature dynamic fusion module FGF Different features have different degrees of influence in different sentences, so the feature dynamic fusion module FGF is used to dynamically select the feature F gc 、F mc The proportion retained after fusion; FGF adopts a gating mechanism and uses the gating vector V gate The text feature F after vocabulary enhancement obtained in the previous step gc 、F mc Perform weighted summation to obtain the fused feature F f ; The formula is as follows: h gc =tanh(F gc W gc +b gc ) (18) h mc =tanh(F mc W mc +b mc ) (19) V gate =σ(([h gc ;h mc ])W gate ) (20) F f =V gate F gc +(1-V gate )F mc (21) Among them, h gc 、h mc Respectively represent F gc 、F mc After nonlinear transformation, the vector W gc ,W mc ,W gate ,b gc ,b mc It is a randomly initialized parameter that will be automatically updated as the model is trained over multiple rounds. The number of model training rounds ranges from 50 to 100. σ represents the sigmoid activation function, and tanh represents the tanh activation function. S5: Label prediction module LP The label prediction module uses a fully connected network and CRF layer to predict the feature F obtained in the previous step. f Perform label prediction; the fully connected network maps text features to the label distribution space and outputs label prediction probability; the CRF layer receives the label prediction probability output by the fully connected network and outputs the optimal sequence labeling result.

Images