Text extraction from handwritten case sheets using crnn

Abstract

The present invention provides a text extraction from handwritten case sheets using CRNN that can be employed in healthcare sector to efficiently enter data, access data, data analysis of a patient becomes easier and likelihood of medical errors are reduced which ensures patient safety, interoperability and integration is enhanced as this data can be compatible with Electronic Health Record (EHR) systems and seamlessly integrated into healthcare workflows, efficient sharing of information and collaboration among healthcare providers can be achieved with ease. The present invention combines the development of our custom CRNN model with leveraging a benchmark model like VGG-16. We plan to train some convolution layers using the VGG-16 model as a starting point, and then incorporate the saved weights from our custom pre-trained CRNN model to complete the training of a new, deep CRNN model tailored to our specific needs.

Description

[0001] PCT25-IN-006 PAGE 1

[0002] TEXT EXTRACTION FROM HANDWRITTEN CASE SHEETS USING CRNN

[0003] FIELD OF THE INVENTION

[0004]

[0001] The present invention relates to the field of Artificial Intelligence and Machine Learning, and more particularly, the present invention relates to the text extraction from handwritten case sheets using CRNN that has the potential to significantly improve healthcare data management by facilitating efficient digitization and analysis of patient records. Further, the invention relates to the process of development of automated text extraction system that can accurately extract text from handwritten case sheets.

[0005] BACKGROUND FOR THE INVENTION:

[0006]

[0002] The following discussion of the background to the invention is intended to facilitate an understanding of the present invention. However, it should be appreciated that the discussion is not an acknowledgment or admission that any of the material referred to was published, known, or part of the common general knowledge in any jurisdiction as of the priority date of the application. The details provided herein the background if belongs to any publication is taken only as a reference for describing the problems, in general terminologies or principles or both of science and technology in the associated prior art.

[0007]

[0003] Handwriting recognition (HWR) plays a crucial role in digitizing medical case sheets, enabling efficient data extraction and analysis. Prior works in this field have focused on various approaches, including traditional machine learning techniques and deep learning models. However, existing systems often lack the capability to handle the complexities of doctor’s handwritten notes, leading to errors and inefficiencies in data processing. Some difficulties like the presence of extreme variability in the writing style among writers. Even PCT25-IN-006 PAGE 2 within a group of writers coming from the same cultural and educational background, variability can still be encountered. Under some circumstances, a writer might have difficulties reading his own handwriting, the gap between the different words of a handwritten text do not have a fixed value, and mainly a large amount of training data is required for a recognition system to achieve good performance. The training set must cover all the variability that may occur in the documents.

[0008]

[0004] The absolute need for text extraction from handwritten text images lies in the significance of digitizing and processing handwritten information efficiently. This process is crucial for various industries, including healthcare, insurance, banking, and online libraries, where handwritten notes and documents are prevalent. In healthcare, handwriting recognition can help digitize patient prescriptions, improving the user experience and streamlining the process of handling medical records. In insurance, it can help process claims documents more efficiently, reducing delays and improving customer satisfaction. In banking, handwriting recognition can help automate the processing of cheques, reducing costs and improving accuracy.

[0009]

[0005] In the past and recent times, researchers have employed various methodologies involving Optical Character Recognition (OCR) algorithms to convert static handwritten text images to machine-readable text. The algorithms implemented earlier for handwriting recognition focused on printed text and used techniques such as pattern recognition, feature extraction, and classification. However, these methods often fell short when dealing with handwritten text due to the high variance in handwriting styles and poor quality of handwritten text compared to printed text. For example, researchers have used techniques like Hidden Markov Models (HMM), Support Vector Machines (SVM), and Neural Networks PCT25-IN-006 PAGE 3

[0010] (NN) for handwriting recognition. These techniques have shown some success in recognizing handwritten text, but they often require a large amount of labeled data and can be computationally expensive.

[0011]

[0006] The reliance on extensively large labeled datasets and the difficulty in processing heterogeneous documents with high variability in writing styles made us to explore different methods. Researchers began to investigate data generation, few / zero-shot learning, transfer learning to overcome these obstacles. So by considering all the advantages there is a need for an automated text extraction system.

[0012]

[0007] In light of the foregoing, there is a need for the Text extraction from handwritten case sheets using CRNN that overcomes problems prevalent in the prior art.

[0013] OBJECTS OF THE INVENTION:

[0014]

[0008] Some of the objects of the present disclosure, which at least one embodiment herein satisfies, are as follows.

[0015]

[0009] The principal object of the present invention is to overcome the disadvantages of the prior art by providing the Text extraction from handwritten case sheets using CRNN.

[0016]

[0010] A primary objective of the present invention is to design a model for automated system which can extract text from handwritten case sheet images.

[0017] [Oil] Another objective of the invention to gather few handwritten case sheet data and then extract the handwritten words present in it and then manually annotate each image to create a dataset for training. PCT25-IN-006 PAGE 4

[0018]

[0012] Furthermore, this invention relates to the combination of Convolutional Neural Networks and Long Short-Term Memory (RNNs) and creating a pre-trained CRNN model.

[0019]

[0013] Yet another objective of the present invention is to use transfer learning approach i.e. use a pre-trained model for few layers of CRNN model and then remaining layers are to be fine-tuned with the initially saved model weights obtained from pre-trained CRNN.

[0020]

[0014] Other objects and advantages of the present disclosure will be more apparent from the following description, which is not intended to limit the scope of the present disclosure.

[0021] SUMMARY OF THE INVENTION:

[0022]

[0015] Generally, Convolutional Recurrent Neural Network architecture has proven to be effective in tasks such as image text recognition and extraction. This combines the strengths of both convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The innovation resides in the creation of a pre-trained CRNN model using a custom handwritten dataset, with the subsequent preservation and application of the model’s weights for transfer learning. The novelty extends to the method of fine-tuning a new, more sophisticated CRNN model. This model integrates a few CNN layers trained with a pretrained network like VGG16, while the remaining CNN layers, along with the Bi- LSTM and CTC layers, are fine-tuned using the weights from the previously trained model. This dual strategy of employing a pretrained network for initial layers and custom- trained weights for the subsequent layers is particularly distinctive. It leverages the generalizability of features learned from large datasets and the specificity of features learned from the domain- specific dataset, thereby enhancing the model’s performance on the task of handwritten text recognition. PCT25-IN-006 PAGE 5

[0023] BRIEF DESCRIPTION OF DRAWINGS:

[0024]

[0016] Reference will be made to embodiments of the invention, examples of which may be illustrated in accompanying figures. These figures are intended to be illustrative, not limiting. Although the invention is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the invention to these particular embodiments.

[0025]

[0017] FIGURE 1: WORKFLOW OF THE SYSTEM;

[0026]

[0018] FIGURE 2: CUSTOM CRNN MODEL; and

[0027]

[0019] FIGURE 3: DETECTION OF HANDWRITTEN WORDS.

[0028] DETAILED DESCRIPTION OF DRAWINGS:

[0029]

[0020] While the present invention is described herein by way of example using embodiments and illustrative drawings, those skilled in the art will recognize that the invention is not limited to the embodiments of drawing or drawings described and are not intended to represent the scale of the various components. Further, some components that may form a part of the invention may not be illustrated in certain figures, for ease of illustration, and such omissions do not limit the embodiments outlined in any way. It should be understood that the drawings and the detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents, and alternatives falling within the scope of the present invention as defined by the appended claim. PCT25-IN-006 PAGE 6

[0030]

[0021] As used throughout this description, the word "may" is used in a permissive sense (i.e. meaning having the potential to), rather than the mandatory sense, (i.e. meaning must). Further, the words "a" or "an" mean "at least one” and the word “plurality” means “one or more” unless otherwise mentioned. Furthermore, the terminology and phraseology used herein are solely used for descriptive purposes and should not be construed as limiting in scope. Language such as "including," "comprising," "having," "containing," or "involving," and variations thereof, is intended to be broad and encompass the subject matter listed thereafter, equivalents, and additional subject matter not recited, and is not intended to exclude other additives, components, integers, or steps. Likewise, the term "comprising" is considered synonymous with the terms "including" or "containing" for applicable legal purposes. Any discussion of documents, acts, materials, devices, articles, and the like are included in the specification solely for the purpose of providing a context for the present invention. It is not suggested or represented that any or all these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention.

[0031]

[0022] In this disclosure, whenever a composition or an element or a group of elements is preceded with the transitional phrase “comprising”, it is understood that we also contemplate the same composition, element, or group of elements with transitional phrases “consisting of’, “consisting”, “selected from the group of consisting of, “including”, or “is” preceding the recitation of the composition, element or group of elements and vice versa.

[0032]

[0023] The present invention is described hereinafter by various embodiments with reference to the accompanying drawing, wherein reference numerals used in the accompanying drawing correspond to the like elements throughout the description. This invention may, however, be embodied in many different forms and should not be construed as limited to the PCT25-IN-006 PAGE 7 embodiment set forth herein. Rather, the embodiment is provided so that this disclosure will be thorough and complete and will fully convey the scope of the invention to those skilled in the art. In the following detailed description, numeric values and ranges are provided for various aspects of the implementations described. These values and ranges are to be treated as examples only and are not intended to limit the scope of the claims. In addition, several materials are identified as suitable for various facets of the implementations. These materials are to be treated as exemplary and are not intended to limit the scope of the invention.

[0033]

[0024] The present invention provides Text extraction from handwritten case sheets using CRNN that addresses the need for efficient and accurate data processing, automation of workflows, and digitization of legacy data, data analysis, cost effectiveness, data security, enhanced accessibility and integration with existing systems. This can be very useful in healthcare, finance and banking, education, historical preservation etc.

[0034]

[0025] Mainly, this technology has the potential to significantly improve healthcare data management by facilitating efficient digitization and analysis of patient records. So, by employing this text extraction system in healthcare sector we can efficiently enter data, access data, data analysis of a patient becomes easier and likelihood of medical errors are reduced which ensures patient safety, interoperability and integration is enhanced as this data can be compatible with Electronic Health Record(EHR) systems and seamlessly integrated into healthcare workflows, efficient sharing of information and collaboration among healthcare providers can be achieved with ease. And by employing this system, the information becomes standardized and easily understandable to everyone.

[0035]

[0026] So, by considering the uses of text extraction from images and in order to reduce time consumption and to reduce human errors which ensures patient safety, we are creating a deep PCT25-IN-006 PAGE 8 learning model which can extract the text from handwritten case sheets. We propose a novel approach that combines the development of our custom CRNN model with leveraging a benchmark model like VGG-16. We plan to train some convolution layers using the VGG-16 model as a starting point, and then incorporate the saved weights from our custom pre-trained CRNN model to complete the training of a new, deep CRNN model tailored to our specific needs.

[0036]

[0027] This text extraction system initially had two problems. One is detecting the words in the images and the second is to recognize the detected words. Firstly in order to address the first problem that is detecting all the handwritten words present in the case sheets which are collected from hospital. Space Scale technique did not work to detect the words. So, we use neural network approach for detecting these words present as the inter-word and intersentence spacing is uneven in our case. Detected images are shown in figure 3. The next stage after detecting the words is to extract the words and annotate each image to prepare a dataset that can be shown in Figure 1. Resizing and gray scaling is done as the part of preprocessing.

[0037]

[0028] The next stage after preprocessing is to design a CRNN model and train this model with the data set created. CRNN stands for Convolutional Recurrent Neural Network. It is a type of deep learning architecture that combines the strengths of both convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The initial layers of the CRNN are composed of CNNs, which are used to extract spatial (high-level) features from the input image.

[0038]

[0029] After the convolutional layers, the CRNN incorporates RNNs, which process the sequential information from the feature maps generated by the CNNs. The final part of the PCT25-IN-006 PAGE 9

[0039] CRNN is the transcription layer, which typically uses a Connectionist Temporal

[0040] Classification (CTC) loss function. This layer translates the RNN outputs into a sequence of characters or words.

[0041]

[0030] First, the input image is fed through a number of convolutional layers to extract the feature maps. These feature maps are then divided into a sequence of feature vectors. These feature vectors are then fed to a bi-directional LSTM. Because of the soft max activation function, this LSTM layer outputs the probability distribution at each time step over the character set.

[0042]

[0031] To obtain the per-time step output, we can take the max of the probability distribution at each time step obtained by soft max activation function. To align the final output (text transcriptions) with the per-time step predictions in order to train this network we consider Connectionist Temporal Classification (CTC).

[0043]

[0032] We began by training a CRNN model that consists of two convolutional layers in the CNN part, two Bi-LSTM units, and one CTC layer using our custom dataset. After training, we saved the weights of this model as we can use this saved model as pre-trained CRNN model. Next, we designed a new CRNN model with an expanded architecture, which includes four convolutional layers and double the number of Bi-LSTM units compared to the first model. For this new model, we incorporated a pre-trained VGG16 network for a few of the CNN layers. This is to leverage the powerful feature extraction capabilities of VGG16. We then unfroze a few convolutional layers in the new model and loaded the weights from your previously trained CRNN model into these layers. The purpose of this approach is to utilize the learned features from both the pre-trained VGG16 network and your initial CRNN model, aiming to enhance the performance of the new, more complex CRNN model on our custom PCT25-IN-006 PAGE 10 dataset. This is shown in Figure 2.

[0044]

[0033] The disclosure has been described with reference to the accompanying embodiments herein and the various features and advantageous details thereof are explained with reference to the non-limiting embodiments in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein.

[0045]

[0034] The foregoing description of the specific embodiments so fully revealed the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the scope of the embodiments as described herein.

Claims

PCT25-IN-006 PAGE 11CLAIMS:1) A text extraction system from handwritten case sheets using CRNN, the system comprises: an electronic device comprising: a processor, input output devices, and power source, communication module; the processor comprises:- a neural network for detecting words present in an image captured by image capturing device;- a dataset module that extracts the words and annotate each image to prepare a dataset;- a pre-processing module that designs a Convolutional Recurrent Neural Network (CRNN) model and trains said model with said dataset; the Convolutional Recurrent Neural Network (CRNN) comprises: a Convolutional Neural Networks (CNNs) that extracts spatial (high-level) features from the input image andRecurrent Neural Networks (RNNs) that processes the sequential information from the feature maps generated by the CNNs; wherein the feature maps comprises a sequence of feature vectors that are fed to a bidirectional LSTM.2) The system as claimed in claim 1, wherein the resizing and gray scaling is done as the part of preprocessing.3) The system as claimed in claim 1, wherein the LSTM layer outputs the probability distribution at each time step over the character set.4) The system as claimed in claim 1, wherein for obtaining the per-time step output, the processor takes the max of the probability distribution at each time step obtained by softmax activation function and for aligning the final output (text transcriptions) with the per-timePCT25-IN-006 PAGE 12 step predictions in order to train this network the system considers Connectionist Temporal Classification (CTC).5) The system as claimed in claim 1, wherein the CTC layer using our custom dataset.6) The system as claimed in claim 1, wherein a pre-trained VGG16 network is incorporated for a few of the CNN layers.

Images