A medical record search method based on language model

Abstract

The invention discloses a medial case searching method based on a language model. The method has the steps of 1, extracting a single structural medical scheme from a medical case book through OCR and text structured processing; 2, preprocessing all the medical cases by a Chinese word segmentation tool, wherein the preprocessing includes work segmentation and stop word elimination; 3, obtaining the unigram language model of each medical case by maximum likelihood estimation; 4, counting the number of works corresponding to each work frequency level for each medical case, and using a statistical data fitted curve; 5, smoothing the unigram language model of each medical case by a Good-Turing estimation method; 6, building a language model taking all the medical cases as a whole, wherein the language model can modify the unigram language model of a single medical case; 7, realizing medical case search by using the modified language model. The information search based on the language model is realized. The respective language model of each medical case is built by the N-gram, the probability of generating the text by the language model is used as the search result ranking basis.

Description

technical field [0001] The invention relates to the field of information retrieval, in particular to a method for searching medical cases based on a language model. Background technique [0002] A language model is a model that generates text based on probabilities. Given a sentence, that is, a sequence of words, the language model can obtain this sequence, that is, p(w 1 ,...,w n )The probability. It has many application scenarios, such as speech recognition, machine translation, POS tagging, handwritten font recognition, information retrieval and so on. [0003] The N-gram model is a language model with fast training and high probability of calculating and generating text, which is suitable for information retrieval. The typical N-gram is the Unigram model, a sentence, that is, a sequence of words, w 1 ,...,w n The probability p(w 1 ,...,w n ), according to the chain rule, should be equal to p(w 1 )×p(w 2 |w 1 )…p(w n |w 1 ,...,w n-1 ). If one makes the simp...

AI Technical Summary

Problems solved by technology

[0004] In actual operation, some words that do not exist in the model dictionary will appear in the training data or test data. This is very common, because the model dictionary is unlikely and does not need to include all words. In the n-gram model, the word It is represented by a vector, and the dimension of the vector is the size of the dictionary. If the dictionary is very large, the dimension of the word vector will be very high, and the model needs to do more calculations during the training process, which increases the training time.

Moreover, a larger dictionary does not improve the effect of the model. Some uncommon words may only appear once or twice in the entire training data set, and the estimation of its probability these two times is very inaccurate. overfitting

Images