A medical corpus expansion method based on deep language model and template customization

Abstract

The present invention discloses a method for expanding medical corpus based on deep language model and template customization. Aiming at the problem that there are more professional terms for diseases, medicines and equipment in medical field corpus than in other scenarios, the present invention adopts a method of template customization to increase the probability of medical professional terms appearing in the generated corpus. For different types of diseases and different departments, the noun terms are very different, and it is unrealistic to manually make large-scale templates for different diseases and different departments. The present invention adopts a deep language model to automatically generate a large number of diverse medical template sentences from similar or related foreign text data sets, thereby effectively expanding the medical corpus and reducing the time and economic cost of manual data collection.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic speech recognition, and in particular relates to a method for enhancing Chinese language model data by expanding a medical corpus based on a deep language model and template customization. Background Art

[0002] In medical scenarios such as consultation, doctor-patient communication, expert consultation, and medical training, there will be a large number of professional terms and terminology in the medical field during the communication process, and the parties involved in the communication cannot fully understand these professional terms and terminology. For example, in the process of doctor-patient communication, when the doctor explains the pathology, treatment plan, drugs used and their side effects to the patient, there will be a large number of professional terms and terminology in the medical field, and the doctor cannot explain them in detail one by one, making it impossible for the patient to fully understand his or her own condition. At the same time, it is also possible to misunderstand the doctor and think that the doctor has not fulfilled his or her responsibilities, leading to tension in the doctor-patient relationship.

[0003] For the above problems, real-time recording can be used. After the consultation, the patient can check the relevant information according to the recording content to help understand the condition. At the same time, the recording content can also be used as a backup for the hospital so that it can be reviewed when needed in the future. Although this recording backup method is simple and easy, the disadvantages are also obvious: 1) The recording backup takes up more storage space; 2) When checking the content, the viewer cannot quickly locate it; 3) For professional terms, the patient only knows its pronunciation but not its specific content, which makes it difficult to check the relevant information.

[0004] Thanks to the emergence of deep learning technology, automatic speech recognition systems have developed rapidly, and the accuracy and speed of automatically converting speech signals into text have been rapidly improved. Therefore, by building an intelligent medical automatic speech recognition system, the conversation between doctors and patients can be recorded in real time and translated into text. Doctors can quickly write medical records based on these text records, reducing the time spent on writing medical records and investing precious time in the diagnosis and treatment of patients; at the same time, based on the recorded text content, it is convenient for patients to quickly access relevant information.

[0005] However, the main challenges in building an intelligent medical automatic speech recognition system are: 1) The currently popular automatic speech recognition models are trained on general data sets and lack differentiated customization for the medical field, resulting in poor recognition of medical terminology; 2) There is a lack of corpus data sets customized for the medical field. There are more professional terms about diseases, drugs, and equipment in medical scenarios than in other scenarios, and academia and industry lack large-scale medical speech data; 3) There are great differences in terms between different types of diseases and different departments, and it is unrealistic to collect large-scale corpus data for different diseases and different departments. Summary of the invention

[0006] In view of the problem that the intelligent medical automatic speech recognition system has poor recognition of medical proper nouns due to the lack of large-scale medical corpora in different specific medical scenarios, the present invention proposes a medical corpus expansion method based on deep language model and template customization.

[0007] The present invention adopts a template customization method to increase the probability of medical professional terms appearing in the generated corpus. A deep language model is used to automatically generate a large number of diverse medical template sentences from similar or related foreign text data sets, thereby effectively expanding the medical corpus and reducing the time and economic cost of manual data collection.

[0008] A medical corpus expansion method based on deep language model and template customization, the steps are as follows:

[0009] Step 1: Build an initial medical text dataset and an out-of-domain text dataset to create a medical template dataset.

[0010] Step 2: Pre-training deep language model:

[0011] Step 3: Label the medical template dataset and the processed out-of-domain text dataset differently to form a medical template label dataset and an out-of-domain text label dataset.

[0012] Step 4: Load the pre-trained deep language model weights obtained in step 2 into the deep language model to obtain a pre-trained deep language model, then add the label information to the input matrix, fine-tune the deep language model, and obtain a fine-tuned deep language model.

[0013] Step 5: Input the labels of the medical template dataset into the fine-tuned pre-trained language model, then perform word prediction on each word in the sentences in the processed out-of-domain text dataset and save each set of predicted words.

[0014] Step 6: Generate a vocabulary confusion network based on the groups of predicted words saved by the model prediction. Arrange and combine the words in the vocabulary confusion network in order to obtain all the template sentences.

[0015] Furthermore, the specific steps of step 1 are as follows:

[0016] (1) Initial small-scale medical text dataset collection: According to the department classification, the diagnosis and treatment process dialogue data in the target department scene is collected separately as the initial small-scale medical text dataset. At the same time, foreign text datasets related to or similar to the field are selected.

[0017] (2) Corpus cleaning: First, the medical text dataset and the foreign text dataset were divided into lines according to commas and periods, and other punctuation marks in the corpus were removed; then sentences containing English and numbers were deleted to obtain the cleaned corpus.

[0018] (3) Create a medical template dataset: Extract sentences with medical entity nouns from the cleaned medical text dataset according to the medical dictionary, and remove sentences without medical entity nouns. Then randomly extract several sentences from the corpus and replace the medical entity nouns with general terms. Delete sentences with missing words, extra words, or wrong words. Finally, record the length of the longest and shortest sentences in the medical template dataset. Then remove sentences in the cleaned out-of-domain text dataset that are shorter than the shortest sentence in the medical template dataset and longer than the longest sentence in the medical template dataset.

[0019] Furthermore, the specific steps of step 2 are as follows:

[0020] First, we build a deep language model consisting of three parts: (1) input embedding layer, (2) multi-head attention layer, and (3) linear layer. Then, we use the medical template dataset and the processed out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows:

[0021] (1) The input embedding layer is used to perform word segmentation, vocabulary building, initial encoding, one-hot encoding, word embedding compression, position encoding, and input matrix building on the medical template dataset and the processed out-of-domain text dataset:

[0022] 1) Word segmentation: Segment each sentence based on the shortest path and dynamic programming word segmentation algorithm, and save the word segmentation results to form a word segmentation set.

[0023] 2) Vocabulary creation: The words that appear in the word set are saved to form a vocabulary. There is one word per line in the vocabulary, and the line number of the word is the code of the word.

[0024] 3) Initial coding: Set a unified coding length and use a vocabulary to perform initial coding on the segmented sentences. Sentences shorter than the unified coding length are padded with 0s; sentences longer than the unified coding length do not encode the excess part.

[0025] 4) One-hot encoding: Each row in the one-hot encoding matrix represents the one-hot code of a word. In this row, only the value of the word encoding column of the word is 1, and the values ​​of other positions are 0.

[0026] 5) Word embedding compression: construct a random matrix w s , Where d is the dimension of the word vector, and then w SMultiply the matrix with the one-hot encoding matrix, perform word embedding compression, and obtain the word vector matrix. The process of word embedding compression can be expressed as:

[0027]

[0028] where w c is the word vector matrix, is the one-hot encoding matrix.

[0029] 6) Position encoding: Construct the position matrix pe for the word vector matrix and construct the position matrix:

[0030]

[0031]

[0032] Where pos is the position of the word in the sentence, d is the dimension of the word vector, and i is an integer value between 0 and d-1, including 0 and d-1. 2i refers to the even columns in the word vector, 2i+1 refers to the odd columns in the word vector, and pe (pos,2i+1) Indicates the value of the posth row and odd column in the pe matrix, pe (pos,2i) Represents the value of the posth row and even column in the pe matrix. For example, suppose the dimension of the word vector matrix is ​​5×4, that is, there are five words in the sentence, and each word is represented by a four-dimensional vector. Then the calculation formula of the position matrix is:

[0033]

[0034] When pos=1, we get the position vector of the first word:

[0035] Substitute pos=2 into the formula to get the position vector of the second word: The position vectors of all words are obtained in this way. They are combined into a position matrix, the dimension of which is 5×4, the same as the dimension of the word vector matrix.

[0036] 7) Input matrix establishment: directly add the obtained position matrix and word vector matrix to obtain the input matrix.

[0037] (2) The multi-head attention layer constructs a multi-head attention mechanism and calculates the attention score matrix to perform representation learning. The multi-head attention layer consists of multiple layers of multi-head attention mechanisms. The attention score matrix calculated by the previous layer of attention mechanism is used as the input of the next layer of attention mechanism. Its steps include:

[0038] 1) Construct three random matrices d is the word vector dimension. The input matrix is ​​multiplied by these three matrices to obtain the query matrix Q, key matrix K, and value matrix V.

[0039] 2) Calculate the multi-head attention score matrix. The specific calculation formula of the multi-head attention score matrix is:

[0040] MultiHead(Q,K,V)=Concat(head1,…,head h ) o

[0041]

[0042]

[0043] Where d is the dimension of the word vector, head h represents the hth attention head, Concat represents the matrix head1,…,head h To splice, in h represents the number of attention heads, hd v Represents the number of attention heads h and d v The product of .

[0044] The multi-head attention score matrix is ​​calculated through the above steps, and then the attention score matrix is ​​input into the next layer of multi-head attention mechanism to continue calculating the attention score matrix.

[0045] (3) Linear layer: The attention score matrix output by the multi-head attention layer is input into the linear layer for dimension transformation. The formula is:

[0046] F=MultiHead(Q,K,V)*w f

[0047] where w f is the linear transformation matrix,

[0048] Then the matrix F after dimension transformation is normalized using the softmax function, and the elements of each row are scaled to between (0, 1) to obtain the output matrix F N ;

[0049] (4) Use the medical template dataset and the processed out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows:

[0050] 1) Divide the data set into several batches, each of which consists of x sentences. Then perform a mask operation on each sentence in each batch.

[0051] 2) Send the BATCH after the MASK operation to the input embedding layer to obtain the input matrix.

[0052] 3) Send the input matrix to the multi-head self-attention layer and calculate the multi-head attention score matrix.

[0053] 4) Input the multi-head attention score matrix into the linear layer for dimension transformation to obtain the output matrix p c

[0054] 5) Construct the target matrix y according to the words masked in BATCH c ,y c The dimension and p c Same, y c Each row vector corresponds to the one-hot encoding vector of the word replaced by MASK in each sentence in BATCH.

[0055] 6) Then according to the target matrix y c With the output matrix p c , calculate the cross entropy function to get the Loss of the entire BATCH. The cross entropy calculation formula for a BATCH is:

[0056]

[0057] in Represents y c The i-th row vector in the matrix, Table p c The i-th row vector of the matrix, Represents a vector Perform logarithm operation on each element in . Representation vector With vector Do the vector product. x equals y c The number of columns in the matrix.

[0058] 7) According to Loss, the Adam algorithm is used to update the model parameters until convergence to obtain the weights of the pre-trained deep language model.

[0059] Furthermore, the specific method of step 3 is as follows:

[0060] A number '0' is added to the end of each sentence in the medical template dataset that has been word-classified, and a number '1' is added to the end of each sentence in the processed out-of-domain text dataset that has been word-classified.

[0061] Furthermore, the specific steps of step 4 are as follows:

[0062] (1) Extract sentence label information and assign the same label to each word to construct a conditional label vector.

[0063] (2) Construct a one-hot encoding matrix based on the conditional label vector The elements in the conditional label vector are composed of only '0' and '1'. The columns with a value of 1 correspond to the one-hot encoding vector [0,1], and the columns with a value of 0 correspond to the one-hot encoding vector [1,0]. The number of rows in the one-hot encoding matrix is ​​equal to the uniform encoding length.

[0064] (3) Construct matrix w l , and with the one-hot encoding matrix Perform matrix multiplication to obtain the conditional label matrix w c , the formula is:

[0065]

[0066] (4) Add the conditional label matrix to the input matrix to obtain the input matrix with label information

[0067] (5) The deep language model is trained using an input matrix with label information. The input matrix passes through a multi-head attention layer and a linear layer to obtain an output matrix. The cross entropy is calculated based on the output matrix and the target matrix to obtain the Loss. The model parameters are optimized based on the Loss until convergence, and a fine-tuned pre-trained deep language model is obtained.

[0068] Furthermore, the specific steps of step 5 are as follows:

[0069] (1) For sentences in the foreign text dataset, replace the words in the sentence with '[MASK]' from left to right in word order, replacing one word at a time.

[0070] (2) The replaced sentences and labels of the medical template dataset are input into the fine-tuned pre-trained deep language model. The sentences and labels are input into the embedding layer to obtain the input matrix, which is then sent to the multi-head attention layer to calculate the attention score matrix.

[0071] (3) The obtained attention score matrix is ​​input into the linear layer for dimension transformation, and the final model output dimension is l vocab The vector p l , save the column number of the first N elements with the largest value in the current vector. The calculation formula is:

[0072]

[0073] Where U is the number of columns saved, p l The output vector

[0074] (4) Then use the vocabulary to map the columns saved in U into corresponding words to form a set of predicted words.

[0075] (5) Repeat (1) to (4) until every word in the sentence has been replaced with [MASK] and each group of predicted words has been generated.

[0076] Furthermore, the specific steps of step 6 are as follows:

[0077] (1) Save each group of predicted words output by the model, and then arrange the groups of predicted words corresponding to each word according to the word order of the sentence in the foreign text dataset to obtain a lexical confusion network.

[0078] (2) Check whether there are special general words in each group of prediction words of the confusion network.

[0079] (3) If there is no general word in the prediction words of each group, the word confusion network will not be combined. If there is a general word in the prediction words of a group, the prediction words of other groups will participate in the permutation and combination, and only the general words of the prediction words of the group with general words will participate in the permutation and combination, and all template sentences will be listed.

[0080] The beneficial effects of the present invention are as follows:

[0081] 1. Aiming at the problem that the current automatic speech recognition model in the medical field has poor recognition effect on medical proper nouns, the present invention proposes a medical corpus generation method based on template customization and deep language model, which improves the accuracy of the automatic speech recognition model in transcribing medical proper nouns from speech to text;

[0082] 2. In view of the problem that there are more professional terms for diseases, drugs, and equipment in the medical field corpus than in other scenarios, the present invention adopts a template customization method to increase the probability of medical professional terms appearing in the generated corpus.

[0083] 3. There are great differences in noun terms between different types of diseases and different departments, and it is unrealistic to manually produce large-scale templates for different diseases and different departments. The present invention adopts a deep language model to generate medical template sentences from texts in similar fields, thereby improving the diversity of medical template sentences. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a model prediction flow chart of an embodiment of the present invention;

[0085] Figure 2 is a structural diagram of a language model according to an embodiment of the present invention;

[0086] Figure 3 is a flow chart of template generation in an embodiment of the present invention;

[0087] Figure 4 It is a schematic diagram of the overall process of an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings. The technical methods provided by each embodiment of the present invention are described in detail below in conjunction with the accompanying drawings.

[0089] A medical corpus expansion method based on deep language model and template customization, the steps are as follows:

[0090] Step 1: Build the initial medical text dataset and the out-of-domain text dataset to create a medical template dataset. This includes the following steps:

[0091] (1) Initial small-scale medical text dataset collection: According to the department classification, the diagnosis and treatment process dialogue data in the target department scene is collected separately as the initial small-scale medical text dataset. At the same time, foreign text datasets related to or similar to the field are selected.

[0092] (2) Corpus cleaning: First, the medical text dataset and the foreign text dataset were divided into lines according to commas and periods, and other punctuation marks in the corpus were removed; then sentences containing English and numbers were deleted to obtain the cleaned corpus.

[0093] (3) Create a medical template dataset: Extract sentences with medical entity nouns from the cleaned medical text dataset according to the medical dictionary, and remove sentences without medical entity nouns. Then randomly extract several sentences from the corpus and replace the medical entity nouns with general terms. For example, the specific name of cardiovascular disease is replaced by [cardiovascular disease]. Delete sentences with missing words, extra words, and wrong words. For example, "I am suffering from [cardiovascular disease] code you", in this sentence, the word "code" is a wrong word problem, the correct one should be "may", and the last word "you" is redundant in the sentence, which is a problem of multiple words. Finally, record the length of the longest and shortest sentences in the medical template dataset. Then remove sentences in the cleaned out-of-domain text dataset that are shorter than the shortest sentence in the medical template dataset and longer than the longest sentence in the medical template dataset.

[0094] Step 2: Pre-training deep language model:

[0095] First, we build a deep language model consisting of three parts: (1) input embedding layer, (2) multi-head attention layer, and (3) linear layer. Then, we use the medical template dataset and the processed out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows: Figure 2 As shown:

[0096] (1) The input embedding layer is used to perform word segmentation, vocabulary building, initial encoding, one-hot encoding, word embedding compression, position encoding, and input matrix building on the medical template dataset and the processed out-of-domain text dataset:

[0097] 1) Word segmentation: Segment each sentence based on the shortest path and dynamic programming word segmentation algorithm, and save the word segmentation results to form a word segmentation set. For example, the sentence "I have a fever caused by a cold" is segmented into "I", "is", "cold", "caused", "of", "fever".

[0098] 2) Vocabulary creation: The words that appear in the word set are saved to form a vocabulary. There is one word per line in the vocabulary, and the line number of the word is the code of the word.

[0099] 3) Initial coding: Set the unified coding length, use the vocabulary to perform initial coding on the segmented sentences, and fill in 0s for sentences shorter than the unified coding length; for sentences longer than the unified coding length, the excess part is not coded. For example, set the unified coding length to 10 words, and there are 6 words in the sentence "I have a fever caused by a cold". Assuming that the positions of these six words in the vocabulary are 17, 19, 18, 15, 31, and 30, respectively, the initial coding of the sentence is [17, 19, 18, 15, 31, 30, 0, 0, 0, 0].

[0100] 4) One-hot encoding: If the initial encoding of the sentence is [17, 19, 18, 15, 31, 30, 0, 0, 0, 0], then the one-hot encoding of the sentence is a 10×l vocab The matrix (l vocab Represents the number of words in the vocabulary). Each row in the one-hot encoding matrix represents the one-hot code of a word. In this row, only the value of the word encoding column of the word is 1, and the values ​​of other positions are 0. For example, the code of the first word in the initial encoding of the sentence is 17. In the first row, only the value of the 17th column is 1, and the values ​​of the other columns are 0, that is,

[0101] 5) Word embedding compression: construct a random matrix w s , Where d is the dimension of the word vector, and then w S Multiply the matrix with the one-hot encoding matrix, perform word embedding compression, and obtain the word vector matrix. The process of word embedding compression can be expressed as:

[0102]

[0103] where w c is the word vector matrix, is the one-hot encoding matrix.

[0104] 6) Position encoding: Construct the position matrix pe for the word vector matrix and construct the position matrix:

[0105]

[0106]

[0107] Where pos is the position of the word in the sentence, d is the dimension of the word vector, and i is an integer value between 0 and d-1, including 0 and d-1. 2i refers to the even columns in the word vector, 2i+1 refers to the odd columns in the word vector, and pe (pos,2i+1) Indicates the value of the posth row and odd column in the pe matrix, pe (pos,2i) Represents the value of the posth row and even column in the pe matrix. For example, suppose the dimension of the word vector matrix is ​​5×4, that is, there are five words in the sentence, and each word is represented by a four-dimensional vector. Then the calculation formula of the position matrix is:

[0108]

[0109] When pos=1, we get the position vector of the first word:

[0110] Substitute pos=2 into the formula to get the position vector of the second word: The position vectors of all words are obtained in this way. They are combined into a position matrix, the dimension of which is 5×4, the same as the dimension of the word vector matrix.

[0111] 7) Input matrix establishment: directly add the obtained position matrix and word vector matrix to obtain the input matrix.

[0112] (2) The multi-head attention layer constructs a multi-head attention mechanism and calculates the attention score matrix to perform representation learning. The multi-head attention layer consists of multiple layers of multi-head attention mechanisms. The attention score matrix calculated by the previous layer of attention mechanism is used as the input of the next layer of attention mechanism. Its steps include:

[0113] 1) Construct three random matrices d is the word vector dimension. The input matrix is ​​multiplied by these three matrices to obtain the query matrix Q, key matrix K, and value matrix V.

[0114] 2) Calculate the multi-head attention score matrix. The specific calculation formula of the multi-head attention score matrix is:

[0115] MultiHead(Q,K,V)=Concat(head1,…,head h ) o

[0116]

[0117]

[0118] Where d is the dimension of the word vector, head h represents the hth attention head, Concat represents the matrix head1,…,head h To splice, in h represents the number of attention heads, hd v Represents the number of attention heads h and d v The product of .

[0119] The multi-head attention score matrix is ​​calculated through the above steps, and then the attention score matrix is ​​input into the next layer of multi-head attention mechanism to continue calculating the attention score matrix.

[0120] (3) Linear layer: The attention score matrix output by the multi-head attention layer is input into the linear layer for dimension transformation. The formula is:

[0121] F=MultiHead(Q,K,V)*w f

[0122] where w f is the linear transformation matrix,

[0123] Then the matrix F after dimension transformation is normalized using the softmax function, and the elements of each row are scaled to between (0, 1) to obtain the output matrix F N ;

[0124] (4) Use the medical template dataset and the processed out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows:

[0125] 1) Divide the data set into several batches, each of which consists of x sentences. Then perform a MASK operation on each sentence in each batch, that is, replace part of the words in a sentence with the special symbol '[MASK]', replace part of the words with random words, and keep the rest of the words unchanged. For example, after processing "I have a fever caused by a cold", the sentence that may be obtained may be "I have a fever caused by [MASK]". Among them, 'cold' is replaced by '[MASK]', 'cause' is replaced by the random word 'occurrence', and the rest of the words remain unchanged.

[0126] 2) Send the BATCH after the MASK operation to the input embedding layer to obtain the input matrix.

[0127] 3) Send the input matrix to the multi-head self-attention layer and calculate the multi-head attention score matrix.

[0128] 4) Input the multi-head attention score matrix into the linear layer for dimension transformation to obtain the output matrix pc

[0129] 5) Construct the target matrix y according to the words masked in BATCH c ,y c The dimension and p c Same, y c Each row vector corresponds to the one-hot encoding vector of the word replaced by MASK in each sentence in BATCH. For example, suppose "I have a fever caused by MASK" is the yth sentence in BATCH, and the word 'cold' in the sentence is replaced by '[MASK]', then the target matrix y c The yth row of is the one-hot encoded vector of the word 'cold'.

[0130] 6) Then according to the target matrix y c With the output matrix p c , calculate the cross entropy function to get the Loss of the entire BATCH. The cross entropy calculation formula for a BATCH is:

[0131]

[0132] in Represents y c The i-th row vector in the matrix, Table p c The i-th row vector of the matrix, Represents a vector Perform logarithm operation on each element in . Representation vector With vector Do the vector product. x equals y c The number of columns in the matrix.

[0133] 7) According to Loss, the Adam algorithm is used to update the model parameters until convergence to obtain the weights of the pre-trained deep language model.

[0134] Step 3: Label the medical template dataset and the processed out-of-domain text dataset differently to form a medical template label dataset and an out-of-domain text label dataset.

[0135] Specifically, a number '0' is added to the end of each sentence in the medical template dataset that has been word-classified, and a number '1' is added to the end of each sentence in the out-of-domain text dataset that has been word-classified.

[0136] Step 4: Load the pre-trained deep language model weights obtained in step 2 into the deep language model to obtain a pre-trained deep language model, then add the label information to the input matrix, fine-tune the deep language model, and obtain a fine-tuned deep language model. The specific steps are as follows:

[0137] (1) Extract sentence label information and assign the same label to each word to construct a conditional label vector. For example, if the uniform encoding length is 10 words and the label of “I have a fever caused by a cold” is 1, then the conditional label vector is [1,1,1,1,1,1,0,0,0,0], where the last four digits are 0 for padding.

[0138] (2) Construct a one-hot encoding matrix based on the conditional label vector The elements in the conditional label vector are composed of only '0' and '1'. The columns with a value of 1 correspond to the one-hot encoding vector [0,1], and the columns with a value of 0 correspond to the one-hot encoding vector [1,0]. The number of rows in the one-hot encoding matrix is ​​equal to the uniform encoding length. For example: if the uniform encoding length is 5 and the length of a sentence is 4, and the label information of the sentence is '1', then the conditional label vector of this sentence is [1,1,1,1,0], and the one-hot encoding matrix of this vector is a 5×2 matrix specifically expressed as:

[0139]

[0140] (3) Construct matrix w l , and with the one-hot encoding matrix Perform matrix multiplication to obtain the conditional label matrix w c , the formula is:

[0141]

[0142] (4) Add the conditional label matrix to the input matrix to obtain the input matrix with label information

[0143] (5) The deep language model is trained using an input matrix with label information. The input matrix passes through a multi-head attention layer and a linear layer to obtain an output matrix. The cross entropy is calculated based on the output matrix and the target matrix to obtain the Loss. The model parameters are optimized based on the Loss until convergence, and a fine-tuned pre-trained deep language model is obtained.

[0144] Step 5: Input the labels of the medical template dataset into the fine-tuned pre-trained language model, then perform word prediction on each word in the sentence in the processed out-of-domain text dataset and save each set of predicted words. The specific steps are:

[0145] (1) For sentences in the foreign text dataset, replace the words in the sentence with '[MASK]' from left to right in word order, replacing one word at a time.

[0146] (2) The replaced sentences and labels of the medical template dataset are input into the fine-tuned pre-trained deep language model. The sentences and labels are input into the embedding layer to obtain the input matrix, which is then sent to the multi-head attention layer to calculate the attention score matrix.

[0147] (3) The obtained attention score matrix is ​​input into the linear layer for dimension transformation, and the final model output dimension is l vocab The vector p l , save the column number of the first N elements with the largest value in the current vector. The calculation formula is:

[0148]

[0149] Where U is the number of columns saved, p l The output vector

[0150] (4) Then use the vocabulary to map the columns saved in U into corresponding words to form a set of predicted words.

[0151] (5) Repeat (1) to (4) until every word in the sentence has been replaced with [MASK] and each group of predicted words has been generated.

[0152] For example: Figure 1 In the sentence "I have a stuffy nose due to a cold" in the out-of-domain text dataset, replace the third word "cold" with '[MASK]', and then the model predicts the three words "[cardiovascular disease] fever and fall". This is called a set of predicted words.

[0153] Step 6: Generate a vocabulary confusion network based on the groups of predicted words saved by the model prediction. Arrange and combine the words in the vocabulary confusion network in order to obtain all the template sentences. Specifically:

[0154] (1) Save each group of predicted words output by the model, and then arrange the groups of predicted words corresponding to each word according to the word order of the sentence in the foreign text dataset to obtain a lexical confusion network.

[0155] (2) Check whether there are special general terms (such as [cardiovascular disease]) in each group of prediction words of the confusion network.

[0156] (3) If there is no general word in the prediction words of each group, the word confusion network will not be combined. If there is a general word in the prediction words of a group, the prediction words of other groups will participate in the permutation and combination, and only the general words of the prediction words of the group with general words will participate in the permutation and combination, and all template sentences will be listed.

[0157] Reference Figure 4 , which expresses a flowchart of the steps of an embodiment of the present invention, specifically includes the following steps:

[0158] Step 1: Build the initial medical text dataset and the out-of-domain text dataset to create a medical template dataset. This includes the following steps:

[0159] (1) According to the department classification, the initial medical text dataset in the target department scenario is collected separately. The dataset of each department only contains the dialogues during the diagnosis and treatment process of that department. Here, the cardiovascular department is selected to construct the initial medical text dataset. There are 120,000 sentences of dialogues during the diagnosis and treatment process in the dataset. The Chinese dialogue corpus KdConv is selected as the out-of-domain text dataset.

[0160] (2) Corpus cleaning: First, the medical text dataset and the foreign text dataset were divided into lines according to commas and periods, and other punctuation marks in the corpus were removed; then sentences containing English and numbers were deleted to obtain the cleaned corpus.

[0161] (3) Create a medical template dataset: Extract sentences containing cardiovascular disease names and cardiovascular drug names from the corpus based on the medical dictionary, and remove sentences without disease names and drug names. Then randomly extract 500 sentences from the corpus, replace the cardiovascular disease names in the sentences with "[cardiovascular disease]', and replace the cardiovascular drug names with "[cardiovascular drug]'. Delete the noisy sentences, and finally get 320 template sentences to form the medical template dataset. Count and record the longest sentence in the medical template dataset, which is 41 words, and the shortest sentence is 11 words. Then remove the sentences with a length of more than 41 words and a length of less than 11 words in the cleaned out-of-domain text dataset, and finally there are 48,030 sentences left in the out-of-domain text dataset.

[0162] Step 2: Pre-training the deep language model: Build a deep language model, which mainly consists of three parts: 1) input embedding layer, 2) multi-head attention layer, and 3) linear layer. Then use the medical template dataset and the cleaned out-of-domain text dataset to pre-train the deep language model. The detailed steps are as follows: Figure 2 shown.

[0163] (1) The input embedding layer is used to perform word segmentation, vocabulary building, sentence one-hot encoding, word embedding compression, and position encoding on the medical template dataset and the processed out-of-domain text dataset:

[0164] 1) Segment each sentence. Use the Jieba word segmentation tool to segment each sentence in the medical template dataset and the processed out-of-domain text dataset.

[0165] 2) Save the words that appear in the word set to form a word list. There is one word in one line in the word list, and the number of the line where the word is located is the code of the word. The word list has a total of 38421 lines, that is, the word list stores 38421 words.

[0166] 3) Set the unified coding length, use the word list to perform initial coding on the segmented sentences, and set the unified coding length to 41. For sentences shorter than 41, the insufficient part is filled with 0; for sentences longer than 41, the excess part is not coded.

[0167] 4) Construct a one-hot encoding matrix. Each row in the one-hot encoding matrix represents the one-hot code of a word. In this row, only the value of the word encoding column of the word is 1, and the values ​​of other positions are 0. For example, the code of the first word in the initial encoding of the sentence is 17. In the first row, only the value of the 17th column is 1, and the values ​​of the other columns are 0, that is, [0,0,0…1…0,0,0] 1×38421 Because the unified encoding length is set to 41, the dimension of the one-hot encoding matrix of each sentence is 41×38421.

[0168] 5) Construct a random matrix w s , Among them, 768 is the dimension of the word vector, and then w S Multiply the word vector matrix by the unique hot encoding matrix of the sentence. The process of obtaining the word vector matrix can be expressed as follows:

[0169]

[0170] where w c is the word vector matrix, is the one-hot encoding matrix.

[0171] 6) Construct a position matrix for the word vector matrix. The formula for constructing the position matrix is:

[0172]

[0173]

[0174] Where pos is the position of the word in the sentence, d is the dimension of the word vector, which is 768, and i is an integer value between 0 and 768-1. 2i refers to the even columns in the word vector, 2i+1 refers to the odd columns in the word vector, and pe (pos,2i+1) Indicates the value of the posth row and odd column in the pe matrix, pe (pos,2i) Represents the value of the posth row and even column in the pe matrix.

[0175] 7) Directly add the obtained position matrix and word vector matrix to obtain the input matrix.

[0176] (2) The multi-head attention layer constructs a multi-head attention mechanism and calculates the attention score matrix to perform representation learning. The multi-head attention layer consists of multiple layers of multi-head attention mechanisms. The attention score matrix calculated by the previous layer of attention mechanism is used as the input of the next layer of attention mechanism. Here, a 12-layer attention mechanism is used to construct the multi-head attention layer. Its main steps include:

[0177] 1) Construct three random matrices w q ,w k ,w v , d is the word vector dimension of 768. The input matrix is ​​multiplied by these three matrices to obtain Q (query matrix), K (key matrix), and V (value matrix).

[0178] 2) Calculate the multi-head attention score matrix. The specific calculation formula of the multi-head attention score matrix is:

[0179] MultiHead(Q,K,V)=Concat(head1,…,head h ) o

[0180]

[0181]

[0182] Where d is the dimension of the word vector, which is 768, and h represents the number of attention heads, which is 8. h represents the hth attention head, Concat represents the matrix head1,…,head h To splice, in hd v Represents the number of attention heads h and d v The product is 768.

[0183] The attention score matrix is ​​calculated by the above formula, and then the attention score matrix is ​​input into the next layer of multi-head attention mechanism to continue calculating the attention score matrix until all 12 layers are calculated, and the final multi-head attention score matrix is ​​output.

[0184] (3) Linear layer: The multi-head attention score matrix output by the multi-head attention layer is input into the linear layer for dimension transformation. The formula is:

[0185] F=MultiHead(Q,K,V)*w f

[0186] where w f is the linear transformation matrix,

[0187] Then the dimension-transformed matrix F is normalized using the softmax function, scaling the elements of each row to between (0, 1) to obtain the output matrix F N

[0188] (4) Use the medical template dataset and the cleaned out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows:

[0189] 1) Divide the dataset into 251 batches, each batch consists of 192 sentences, and train for 50 rounds. Perform a mask operation on each sentence in each batch, replace 10% of the words in a sentence with the special symbol '[MASK]', replace 10% of the words with random words, and keep the remaining 80% unchanged.

[0190] 2) Send the BATCH after the MASK operation to the input embedding layer to obtain the input matrix.

[0191] 3) Send the input matrix to the multi-head self-attention layer and calculate the multi-head attention score matrix.

[0192] 4) Input the multi-head attention score matrix into the linear layer for dimension transformation to obtain the output matrix p c

[0193] 5) Construct the target matrix y based on the words replaced by 'MASK' in BATCH c ,y c The dimension and p c Same, y c Each row vector corresponds to the one-hot encoding vector of the word replaced by MASK in each sentence in BATCH.

[0194] 6) Then according to the target matrix y c With the output matrix p c , calculate the cross entropy function to get the Loss of the entire BATCH. The cross entropy calculation formula for a BATCH is:

[0195]

[0196] in Represents y c The i-th row vector in the matrix, Table p c The i-th row vector of the matrix, Represents a vector Perform logarithm operation on each element in . Representation vector With vector Do the cross product. X is equal to 192.

[0197] 7) Update the model parameters using the Adam algorithm according to the Loss until convergence to obtain the pre-trained deep language model weights.

[0198] Step 3: Label the medical template dataset and the cleaned out-of-domain text dataset differently to form a medical template label dataset and an out-of-domain text label dataset.

[0199] Specifically, add the number '0' to the end of each sentence in the medical template dataset that has been segmented, and add the number '1' to the end of each sentence in the out-of-domain text dataset that has been segmented. For example, the sentence 'I have a fever caused by a cold' will be labeled as 'I have a fever caused by a cold 1'.

[0200] Step 4: Load the pre-trained deep language model weights obtained in step 2 into the language model to obtain a pre-trained deep language model, then add the label information to the input matrix, fine-tune the deep language model, and obtain a fine-tuned pre-trained deep language model. The specific steps are as follows:

[0201] (1) Extract sentence label information from the medical template label dataset and the out-of-domain text label dataset, assign the same label to each word, and construct a conditional label vector. For example, the unified encoding length is 10 words. The last character of the sentence "I have a fever caused by a cold 1" is the number '1', then the label of this sentence is '1', and the conditional label vector is [1,1,1,1,1,1,0,0,0,0], where the last four numbers 0 are filled in.

[0202] (2) Construct a one-hot encoding matrix based on the conditional label vector The elements in the conditional label vector are composed of only '0' and '1'. The columns with a value of 1 correspond to the one-hot encoding vector [0,1], and the columns with a value of 0 correspond to the one-hot encoding vector [1,0]. The number of rows in the one-hot encoding matrix is ​​equal to the uniform encoding length of 41, and the number of columns is fixed to 2.

[0203] (3) Construct matrix w l , and with the one-hot encoding matrix Perform matrix multiplication to obtain the conditional label matrix w c , the formula is:

[0204]

[0205] (3) Add the conditional label matrix to the input matrix to obtain the input matrix with label information

[0206] (4) Retrain the deep language model using the input matrix with label information. The input matrix passes through the multi-head attention layer and the linear layer to obtain the output matrix. The cross entropy between the output matrix and the target matrix is ​​calculated to obtain the Loss. The model parameters are optimized according to the Loss until convergence, and the fine-tuned pre-trained deep language model is obtained.

[0207] Step 5: Input the label '0' of the medical template label dataset into the fine-tuned pre-trained deep language model, and then perform word prediction on each word in the sentence in the cleaned out-of-domain text dataset, and save each group of predicted words. The specific steps are as follows:

[0208] (1) For sentences in the foreign text dataset, replace the words in the sentence with '[MASK]' from left to right in word order, replacing one word at a time.

[0209] (2) The replaced sentence and the label '0' of the medical template dataset are input into the fine-tuned pre-trained deep language model. The sentence and the label are input into the embedding layer to obtain the input matrix, and then the input matrix is ​​sent to the multi-head attention layer to calculate the multi-head attention score matrix.

[0210] (3) The obtained multi-head attention score matrix is ​​input into the linear layer for dimension transformation, and the final model output dimension is 1×l vocab The vector p l , save the column number of the first three elements with the largest values ​​in the current vector. The calculation formula is:

[0211]

[0212] Where U is the number of columns saved, p l The output vector

[0213] (4) Then use the vocabulary to map the three numbers stored in U into corresponding words to form a set of predicted words.

[0214] (5) Repeat (1) to (4) until every word in the sentence has been replaced with [MASK] and each group of predicted words has been generated.

[0215] For example: Figure 1 In the sentence "I have a stuffy nose due to a cold" in the out-of-domain text dataset, replace the third word "cold" with '[MASK]', and then the model predicts the three words "[cardiovascular disease] fever and fall". This is called a set of predicted words.

[0216] Step 6: Generate a vocabulary confusion network based on the groups of predicted words saved by the model prediction. Arrange and combine the words in the vocabulary confusion network in order to obtain all the template sentences. Specifically:

[0217] (1) Save each group of predicted words output by the model, and then arrange the groups of predicted words corresponding to each word according to the word order of the sentence in the foreign text dataset to obtain a lexical confusion network.

[0218] (2) Check whether "[cardiovascular disease]" or "cardiovascular drugs" exists in each group of prediction words of the confusion network.

[0219] (3) If the prediction words of each group do not contain "[cardiovascular disease]" or "[cardiovascular drugs]", the word confusion network will not be used for word combination. If the prediction words of a group contain "[cardiovascular disease]" or "[cardiovascular drugs]", the prediction words of other groups will participate in the permutation and combination, and only "[cardiovascular disease]" or "[cardiovascular drugs]" will participate in the permutation and combination of the prediction words of the group containing "[cardiovascular disease]" or "[cardiovascular drugs]", and all template sentences will be listed.

[0220] For example: Figure 3 The third group of predicted words in the word confusion network in the word confusion network has generalized words. The words in this word confusion network are combined, where the first group of predicted words are combined with the second group of predicted words to generate 3 2 The word combination is then combined with the third group's '[cardiovascular disease]' to get 3 2 ×1 word combination. And so on, the remaining three groups of prediction words are also combined, and finally 3 5 Template sentences are generated such as 'Old people with cardiovascular disease can cause headaches', 'I have cardiovascular disease which causes dizziness', etc.

[0221] It should be noted that the above embodiments can be freely combined as needed. The above description is only a detailed description of the preferred embodiments and principles of the present invention, but is not intended to limit the present invention. For those of ordinary skill in the art, according to the ideas provided by the present invention, there will be changes in the specific implementation methods, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. A medical corpus expansion method based on deep language model and template customization, characterized in that: Here are the steps: Step 1: Build the initial medical text dataset and the out-of-domain text dataset to create a medical template dataset; Step 2: Pre-train the deep language model; Step 3: Label the medical template dataset and the processed out-of-domain text dataset differently to form a medical template label dataset and an out-of-domain text label dataset; Step 4: Load the pre-trained deep language model weights obtained in step 2 into the deep language model to obtain a pre-trained deep language model, then add the label information to the input matrix, fine-tune the pre-trained deep language model, and obtain a fine-tuned deep language model; Step 5: Input the labels of the medical template dataset into the fine-tuned deep language model, then perform word prediction on each word in the sentence in the processed out-of-domain text dataset and save each set of predicted words; Step 6: Generate a vocabulary confusion network based on each group of predicted words saved by the model prediction; arrange and combine the words in the vocabulary confusion network in order to obtain all template sentences; Step 2 The specific steps are as follows: First, a deep language model consisting of three parts is constructed: (1) input embedding layer, (2) multi-head attention layer, and (3) linear layer. Then, the deep language model is pre-trained using the medical template dataset and the processed out-of-domain text dataset. (1) The input embedding layer is used to perform word segmentation, vocabulary building, initial encoding, one-hot encoding, word embedding compression, position encoding, and input matrix building on the medical template dataset and the processed out-of-domain text dataset: 1) Word segmentation: Segment each sentence based on the shortest path and dynamic programming word segmentation algorithm, and save the word segmentation results to form a word segmentation set; 2) Vocabulary establishment: save the words that appear in the word set to form a vocabulary; there is one word per line in the vocabulary, and the line number of the word is the code of the word; 3) Initial coding: Set the unified coding length, use the vocabulary to perform initial coding on the segmented sentences, and fill in 0 for sentences shorter than the unified coding length; for sentences longer than the unified coding length, the excess part will not be coded; 4) One-hot encoding: Each row in the one-hot encoding matrix represents the one-hot code of a word. In this row, only the value of the word encoding column of the word is 1, and the values ​​of other positions are 0. 5) Word embedding compression: construct a random matrix w s , Where d is the dimension of the word vector, and then w S Multiply the matrix with the one-hot encoding matrix to perform word embedding compression to obtain the word vector matrix. The process of word embedding compression is expressed as follows: where w c is the word vector matrix, is the one-hot encoding matrix; 6) Position encoding: Construct the position matrix pe for the word vector matrix and construct the position matrix: Where pos is the position of the word in the sentence, d is the dimension of the word vector, i is an integer value between 0 and d-1, including 0 and d-1; 2i refers to the even columns in the word vector, 2i+1 refers to the odd columns in the word vector, and pe (pos,2i+1) Indicates the value of the posth row and odd column in the pe matrix, pe (pos,2i) Represents the value of the posth row and even column in the pe matrix; suppose the dimension of the word vector matrix is ​​5×4, that is, there are five words in the sentence, and each word is represented by a four-dimensional vector; then the calculation formula of the position matrix is: When pos=1, we get the position vector of the first word: Substitute pos=2 into the formula to get the position vector of the second word: The position vectors of all words are obtained in this way; they are combined into a position matrix, the dimension of which is 5×4, the same as the dimension of the word vector matrix; 7) Input matrix establishment: directly add the obtained position matrix and word vector matrix to obtain the input matrix; (2) The multi-head attention layer constructs a multi-head attention mechanism and calculates the attention score matrix to perform representation learning; The multi-head attention layer consists of multiple layers of multi-head attention mechanisms. The attention score matrix calculated by the previous layer of attention mechanism is used as the input of the next layer of attention mechanism. Its steps include: 1) Construct three random matrices d is the word vector dimension; and the input matrix is ​​multiplied by these three matrices to obtain the query matrix Q, key matrix K, and value matrix V; 2) Calculate the multi-head attention score matrix; the specific calculation formula of the multi-head attention score matrix is: MultiHead(Q,K,V)=Concat(head1,…,head h )w o Where d is the dimension of the word vector, head h represents the hth attention head, Concat represents the matrix head1,…,head h To splice, in h represents the number of attention heads, hd v Represents the number of attention heads h and d v The product of The multi-head attention score matrix is ​​calculated through the above steps, and then the attention score matrix is ​​input into the next layer of multi-head attention mechanism to continue calculating the attention score matrix; (3) Linear layer: The attention score matrix output by the multi-head attention layer is input into the linear layer for dimension transformation. The formula is: F=MultiHead(Q,K,V)*w f where w f is the linear transformation matrix, Then the matrix F after dimension transformation is normalized using the softmax function, and the elements of each row are scaled to between (0, 1) to obtain the output matrix F N ; (4) Use the medical template dataset and the processed out-of-domain text dataset to pre-train the deep language model. The specific steps are as follows: 1) Divide the data set into several BATCHs, each BATCH consists of x sentences; then perform a MASK operation on each sentence in each BATCH; 2) Send the BATCH after the MASK operation to the input embedding layer to obtain the input matrix; 3) Send the input matrix to the multi-head self-attention layer and calculate the multi-head attention score matrix; 4) Input the multi-head attention score matrix into the linear layer for dimension transformation to obtain the output matrix p c ; 5) Construct the target matrix y according to the words masked in BATCH c ,y c The dimension and p c Same, y c Each row vector corresponds to the one-hot encoding vector of the word replaced by MASK in each sentence in BATCH; 6) Then according to the target matrix y c With the output matrix p c , calculate the cross entropy function to get the Loss of the entire BATCH. The cross entropy calculation formula for a BATCH is: in Represents y c The i-th row vector in the matrix, Table p c The i-th row vector of the matrix, Represents a vector Perform logarithmic operation on each element in ; Representation vector With vector Do a vector product; x equals y c The number of columns of the matrix; 7) According to Loss, the Adam algorithm is used to update the model parameters until convergence to obtain the weights of the pre-trained deep language model.

2. According to claim 1, a medical corpus expansion method based on deep language model and template customization is characterized in that: Step 1 The specific steps are as follows: (1) Initial small-scale medical text dataset collection: According to the department classification, the diagnosis and treatment process dialogue data in the target department scene is collected separately as the initial small-scale medical text dataset; at the same time, foreign text datasets related or similar to the field are selected; (2) Corpus cleaning: First, the medical text dataset and the foreign text dataset were divided into lines according to commas and periods, and other punctuation marks in the corpus were removed; then sentences containing English and numbers were deleted to obtain the cleaned corpus; (3) Create a medical template dataset: Extract sentences with medical entity nouns from the cleaned medical text dataset according to the medical dictionary, and remove sentences without medical entity nouns; then randomly extract several sentences from the corpus and replace the medical entity nouns with general terms; delete sentences with missing words, extra words, or typos; finally, record the lengths of the longest and shortest sentences in the medical template dataset; then remove sentences in the cleaned out-of-domain text dataset that are shorter than the shortest sentence in the medical template dataset and longer than the longest sentence in the medical template dataset.

3. A medical corpus expansion method based on deep language model and template customization according to claim 2, characterized in that: Step 3: A number '0' is added to the end of each sentence in the medical template dataset that has been word-classified, and a number '1' is added to the end of each sentence in the processed out-of-domain text dataset that has been word-classified.

4. The method for expanding medical corpus based on deep language model and template customization according to claim 3, characterized in that: Step 4 The specific steps are as follows: (1) Extract sentence label information and assign the same label to each word to construct a conditional label vector; (2) Construct a one-hot encoding matrix based on the conditional label vector The elements in the conditional label vector are composed of only '0' and '1'; the columns with a value of 1 correspond to the one-hot encoding vector [0,1], and the columns with a value of 0 correspond to the one-hot encoding vector [1,0]; The number of rows in the one-hot encoding matrix is ​​equal to the uniform encoding length; (3) Construct matrix w l , and with the one-hot encoding matrix Perform matrix multiplication to obtain the conditional label matrix w c , the formula is: (4) Add the conditional label matrix to the input matrix to obtain the input matrix with label information; (5) The deep language model is trained using an input matrix with label information. The input matrix passes through a multi-head attention layer and a linear layer to obtain an output matrix. The cross entropy is calculated based on the output matrix and the target matrix to obtain the Loss. The model parameters are optimized based on the Loss until convergence to obtain a fine-tuned deep language model.

5. The method for expanding medical corpus based on deep language model and template customization according to claim 4, characterized in that: Step 5 The specific steps are as follows: (1) Replace the words in the sentences in the foreign text dataset with '[MASK]' from left to right in word order, replacing one word at a time; (2) The replaced sentences and labels of the medical template dataset are input into the fine-tuned deep language model. The sentences and labels are input into the embedding layer to obtain the input matrix, which is then sent to the multi-head attention layer to calculate the attention score matrix. (3) The obtained attention score matrix is ​​input into the linear layer for dimension transformation, and the final model output dimension is l vocab The vector p l , save the column number of the first N elements with the largest value in the current vector; The calculation formula is: Where U is the number of columns saved, p l is the output vector; (4) Then use the vocabulary to map the columns saved in U into corresponding words to form a set of predicted words; (5) Repeat (1) to (4) until every word in the sentence has been replaced with [MASK] and each group of predicted words has been generated.

6. The method for expanding medical corpus based on deep language model and template customization according to claim 5, characterized in that: Step 6 The specific steps are as follows: (1) Save each group of predicted words output by the model, and then arrange the groups of predicted words corresponding to each word according to the word order of the sentence in the foreign text dataset to obtain a lexical confusion network; (2) Check whether there are special generalized words in each group of prediction words of the confusion network; (3) If there are no general words in the prediction words of each group, the word confusion network will not be combined; if there are general words in the prediction words of a certain group, the prediction words of other groups will participate in the permutation and combination, and only the general words of the prediction words of the group with general words will participate in the permutation and combination, and all template sentences will be listed.

Images