Classifying pharmacovigilance documents using image analysis

Abstract

To provide system, apparatus, device, method, and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for classifying a document using image analysis.SOLUTION: In a system where a server, a client device, an intake device, a database and a user device are connected through a network, a server that trains and implements a deep learning model to classify documents using image representations of the documents includes a learning engine 101, a conversion engine 103, and a script 109. The learning engine has an API 102 to receive requests to classify documents from the conversion engine. The conversion engine converts documents into image files so that the learning engine uses the image files to classify and prioritize the corresponding documents. The script is an automated script or bot to detect the documents to be classified.SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 056,501, filed July 24, 2020, the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] Entities such as businesses, government agencies, and educational institutions often receive thousands of documents containing a combination of text, images, charts, tables, and other forms of data, information, and knowledge representation. These documents can be of various types, including Microsoft Word, Microsoft Excel documents, PNG, TIFF, JPG, RAW, GIF, PDF, emails, TXT files, handwritten notes, HTML, and XML scanned documents. Manually categorizing and prioritizing such documents based on their content is a laborious and error-prone task. Entities attempt to automate the process using certain types of machine learning algorithms, such as natural language processing (NLP). However, the machine learning models used require either manual extraction of information or highly intelligent third-party tools to extract the text content of each PDF with acceptable accuracy (e.g., optical character recognition (OCR)) and then successfully extract and integrate this data back into a machine-readable format. Furthermore, traditional methodologies for implementing such machine learning models can face many obstacles when attempting to extract text from documents, such as optical clarity, alphanumeric characters, and orientation. As such, these NLP machine learning models can take years to train and implement, proving costly, making traditional methods of classifying and prioritizing documents laborious, expensive, and prone to error. Summary of the Invention

[0003] Provided herein are embodiments of systems, apparatus, devices, methods, and / or computer program products, and / or combinations and sub-combinations thereof, for classifying documents using image analysis.

[0004] A given embodiment includes a computer-implemented method for automatically classifying documents. In a given embodiment, a document set and metadata for each document are obtained. Each document in the document set is converted into an image file, thereby creating a set of image files. A learning engine is trained to classify each document in the document set using the set of image files and the metadata corresponding to each document. A request to classify another document is received. The other document is converted into a new image file. Pages of the image files are combined into a single image stack. The learning engine assigns first, second, and third classifications to the other document based on the single image stack. The learning engine determines a priority value for the other document based on one or more of the first, second, or third classifications.

[0005] Another embodiment includes a system for automatically classifying documents. The system may include a memory and a processor coupled to the memory. In one embodiment, the processor is configured to obtain a document set and metadata for each document in the document set. The processor is further configured to convert each document in the document set into an image file to generate a set of image files. The processor is further configured to train a learning engine to classify each document in the document set using the set of image files. The processor is further configured to receive a request to classify another document, convert the another document into a new image file, and combine pages of the another image file into a single image stack. The processor then uses the trained learning engine to assign first, second, and third classifications to the another document based on the single image stack. The processor uses the trained learning engine to determine a priority value for the another document based on one or more of the first, second, or third classifications.

[0006] Further embodiments include a non-transitory computer-readable medium having instructions stored thereon, where execution of the instructions by one or more processors of a device causes the one or more processors to perform operations. In one embodiment, the operations include obtaining a document set and metadata for each document in the document set. The operations further include converting each document in the document set into an image file to generate a set of image files. The operations further include training a learning engine to classify each document in the document set using the set of image files. The operations further include receiving a request to classify another document, converting the another document into a new image file, and merging pages of the another image file into a single image stack. The operations further include using the trained learning engine to assign first, second, and third classifications to the another document based on the single image stack. The operations further include determining a priority value for the another document based on one or more of the first, second, or third classifications.

[0007] In a given embodiment, the first classification corresponds to the effectiveness of the other document, the second classification corresponds to an indication of the seriousness of the content of the other document, and the third classification corresponds to one or more factors corresponding to the seriousness of the content.

[0008] Another embodiment includes a device comprising a memory and a processor coupled to the memory. The processor is configured to obtain metadata for a document set and each document in the document set, convert each document in the document set into an image file, and generate a set of image files. The processor is further configured to train a learning engine to classify each document in the document set using the metadata and the set of image files corresponding to each document in the document set. The processor is further configured to receive a request to classify another document and to composite pages of the another document into a single image stack. The processor is further configured to use the trained learning engine to assign a classification to the another document based on the single image stack and display the classification. [Brief explanation of the drawings]

[0009] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the disclosure and to enable one skilled in the relevant art to make and use the disclosure.

[0010] [Figure 1] FIG. 1 is a block diagram of an example system for classifying documents using image analysis. [Figure 2A] FIG. 2A is a block diagram of data flow in the system for classifying documents using image analysis shown in FIG. [Figure 2B] FIG. 2B is a block diagram illustrating a user device for classifying documents according to an example embodiment. [Figure 3] FIG. 3 is a block diagram of an example of a learning model implementing a CNN algorithm, according to an example embodiment. [Figure 4] FIG. 4 is a block diagram of a model approval flow according to an example embodiment. [Figure 5]FIG. 5 is a block diagram illustrating a data flow for classifying documents using a trained learning engine, according to an example embodiment. [Figure 6] FIG. 6 is a block diagram illustrating a confusion matrix for case severity detection using a learning model, according to an example embodiment. [Figure 7] FIG. 7 is an exemplary single image stack, according to an exemplary embodiment. [Figure 8] FIG. 8 is a vector space distribution in accordance with an example embodiment. [Figure 9] FIG. 9 illustrates a receiver operating characteristic (ROC) graph generated based on classifications assigned to documents by the first, second, and third learning models, according to an example embodiment. [Figure 10] FIG. 10 is a diagram illustrating feature maps generated using convolutional layers of a CNN algorithm, according to an example embodiment. [Figure 11A] FIG. 11A is a graph illustrating a principal component analysis (PCA) of validity classifications, in accordance with an illustrative embodiment. [Figure 11B] FIG. 11B illustrates the distribution of valid documents compared to invalid documents according to an example embodiment. [Figure 12A] FIG. 12A is a graph illustrating PCA of severity classification, according to an exemplary embodiment. [Figure 12B] FIG. 12B illustrates the distribution of serious documents compared to non-serious documents according to an example embodiment. [Figure 13A] FIG. 13A is a graph illustrating PCA of severity, relevance, and expectancy (SRE) classification, according to an exemplary embodiment. [Figure 13B] FIG. 13B illustrates the distribution of documents with expected effects compared to documents with unexpected effects, according to an example embodiment. [Figure 14] FIG. 14 is a tensorboard graph illustrating a visualization of the training of a learning model, according to an example embodiment. [Figure 15]FIG. 15 is a flowchart illustrating a process for preparing image files for training a learning model, according to an example embodiment. [Figure 16] FIG. 16 is a flowchart illustrating a process for training a learning model, according to an example embodiment. [Figure 17] FIG. 17 is a flowchart illustrating a process for classifying documents using a trained learning model, according to an example embodiment. [Figure 18] FIG. 18 is a flowchart illustrating a process for classifying documents using a trained learning model implemented on a user device, according to an example embodiment. [Figure 19] FIG. 19 is a block diagram of exemplary components of a device according to one embodiment.

[0011] The drawing in which an element first appears is typically indicated by the leftmost digit(s) in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0012] Provided herein are embodiments of systems, apparatus, devices, methods, and / or computer program products, and / or combinations and sub-combinations thereof, for classifying documents using image analysis.

[0013] As mentioned above, traditional methods of classifying and prioritizing documents can be burdensome, expensive, and error-prone. For example, in fields such as pharmacovigilance (PV), e-discovery, and regulated industries, large volumes of documents need to be classified. This may require performing optical character recognition (OCR) and manually classifying documents, which can be error-prone and result in high operational costs.

[0014] As an example, in the field of pharmacovigilance (PV) operations, companies receive documents such as individual case safety reports (ICSRs) for various pharmaceutical products. PV documents may contain important medical information about users. It may be important to quickly determine the seriousness of the medical information presented in the PV documents. For example, an ICSR is a report of an adverse event experienced by a patient undergoing a particular treatment or taking a particular drug, which may be potentially related to that treatment or drug.

[0015] For an ICSR to be considered "valid", it must contain information relating to four elements: an identifiable patient, an identifiable reporter, the suspect drug, and the adverse event. If the ICSR is valid, it is determined whether the adverse event described is a "serious" adverse event. An adverse event is serious if it meets any of the following requirements: resulting in death or being life-threatening, requiring inpatient hospitalization or prolonging an existing hospitalization; or is medically significant because treatment and / or intervention is necessary to prevent any of the above. Additionally, when conducting clinical trials of a drug or other product, it may be determined whether the adverse effect indicated on the ICSR form is a serious unexpected adverse reaction (SUSAR).

[0016] An ICSR may correspond to a specific case. Different regulatory agencies may need to address cases that have corresponding ICSRs. Regulatory agencies may provide different timelines for different cases. For example, if a case involves a serious adverse effect listed in the ICSR, the case may be prioritized so that a company can take action on the case. Conversely, if a case involves a non-serious adverse effect listed in the ICSR, the case may be given a lower priority.

[0017] PV documents can be provided in a variety of formats, including Microsoft Word, Microsoft Excel documents, png, tiff, jpg, raw, gif, email, PDF, txt file, handwritten notes, HTML, and XML scanned documents. PV documents can also be a combination of formats. For example, a PV document can be in .doc format but contain embedded JPEG images. In another example, part of an ICSR document can be an email message and another part can be in Microsoft Word or Microsoft Excel format.

[0018] PV documents may be provided by various reporters, such as pharmacies, clinicians, and patients. Furthermore, each document may contain reported adverse effects of a drug along with other information about the drug. Companies may need to determine, for example, whether a document is a valid PV document, the severity of the content of the PV document (e.g., the severity of adverse effects listed in the ICSR document), and the severity, relevance, and expectancy (SRE) of the content of the PV document. Given the number of reports and the various formats of the documents, classifying documents in this manner can be a daunting task. Therefore, traditional methods may not be able to effectively and efficiently classify PV documents.

[0019] For example, traditional methods may involve subject matter experts (SMEs) manually reviewing PV documents and making a decision. Alternatively, individuals may manually extract relevant information from PV documents and enter the information into a database, which is later reviewed by medical experts to classify the PV documents. However, a company may receive thousands of PV documents in a short period of time. Given the large volume of PV documents that a company may receive, manually reviewing PV documents can be a time-consuming task. Furthermore, many of the PV documents may be irrelevant because they may not be valid documents, may not indicate a serious effect, or may not indicate a serious, relevant, or expected effect. This can result in a large backlog of documents and delays in processing relevant and important PV documents.

[0020] Traditional methods may also involve using machine learning algorithms that require converting documents to text (e.g., via optical character recognition (OCR)) before they can operate. These machine learning algorithms may include natural language processing (NLP) / named entity recognition (NER). Due to the complexity of OCR coupled with the creation of normalized templates, these machine learning algorithms require significant time, human resources, and financial resources to train, implement, and update the algorithms. For example, training and implementing machine learning algorithms such as NLP / NER for classifying PV documents and text conversion methods can take more than two years using 90 developers. As such, these machine learning algorithms can be operationally inefficient and costly to train and implement.

[0021] In a given embodiment, a server can automatically classify pharmacovigilance (PV) documents. Specifically, the server can obtain a PV document set and metadata for each PV document in the PV document set. The server can convert each PV document in the PV document set into an image file, resulting in an image file set. Furthermore, the server can train a learning engine to classify each PV document in the PV document set using the image file set. The learning engine can implement a convolutional neural network (CNN). The server can further receive a request to classify another PV document, convert the another PV document into a new image file, and composite pages of the another image file into a single image stack. Furthermore, the server can use the trained learning engine to assign first, second, and third classifications to another PV document based on the single image stack.

[0022] The PV document set and the separate PV document are reports of adverse events experienced by patients receiving treatment or medication. The first classification corresponds to the effectiveness of the separate PV document. The second classification corresponds to the severity of the adverse effect noted in the separate PV document. The third classification corresponds to whether the adverse effect noted in the separate PV document is serious, relevant, or expected.

[0023] The above configuration allows for processing and classification of multiple document formats and languages ​​without transcribing and extracting data entities from source documents. This allows for document classification at the time of ingestion, providing the opportunity for entities to scale data ingestion. Furthermore, compared to traditional machine learning methodologies, the above configuration requires one-quarter the time and approximately one-twenty-fifth the human and financial resources to train and implement a learning model. As noted above, training and implementing a machine learning algorithm, such as NLP / NER, to classify ICSR documents is known to take more than two years, using 90 people, including developers, data scientists, and subject matter experts, in one known example. By comparison, an embodiment of the present disclosure can be trained and implemented in two months using three people. In the field of PV, efficiently classifying and prioritizing documents reduces costs and improves the efficiency and consistency of data collection, enhancing signal detection and risk management. This enables more robust analysis and a more detailed characterization of product safety / risk benefits.

[0024] Furthermore, the above configuration avoids text processing, including but not limited to transcription and translation, by leveraging image processing. This methodology uses image representations of documents to improve the speed at which models can be trained to understand domain concepts within and outside of PV.

[0025] In some embodiments, a user device can implement a fully trained learning engine configured to classify a PV document. Specifically, the user device can locally capture an image of the PV document and assign a classification for the PV document. The fully trained learning engine implemented on the user device may not need to identify or extract PII from the PV document when assigning the classification. This configuration allows a user to determine the severity of a medical condition or condition described in the PV document without sharing or transmitting PII over a network. In doing so, this configuration eliminates security concerns associated with sharing PII, such as phishing, cyber attacks, and hacking.

[0026] 1 is a block diagram of a system for classifying documents using image analysis. The system may include a server 100, a client device 110, a capture device 115, a database 120, and a user device 130. The devices of the system may be connected via a network. For example, the devices of the system may be connected via wired connections, wireless connections, or a combination of wired and wireless connections. In an exemplary embodiment, one or more portions of the network may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular network, a wireless network, a WiFi network, a WiMax network, other types of networks, or a combination of two or more such networks. Alternatively, the server 100, the client device 110, and the database 120 may be located on a single physical or virtual machine.

[0027] In some embodiments, server 100 and database 120 may reside in a cloud computing environment. In other embodiments, server 100 may reside in a cloud computing environment and database 120 may reside outside of the cloud computing environment. Furthermore, in other embodiments, server 100 may reside outside of the cloud computing environment and database 120 may reside in the cloud computing environment.

[0028] The capture device 115 may be a device operated by an individual associated with the administrator of the server 100. The capture device 115 may be configured to receive documents to be classified. The capture device 115 may be an email server or other server capable of receiving large amounts of bulk data. The capture device 115 may include a decision engine 116. The decision engine 116 may be an executable application configured to determine whether a document needs to be classified. The operation of the decision engine is described in more detail with respect to FIG. 2.

[0029] Client device 110 may be a device operated by an individual (e.g., a programmer, a user, etc.) associated with the administrator of server 100. Client device 110 may include training application 112 and classification application 114. A cloud computing environment may also host training application 112 and classification application 114. Alternatively, one or both of training application 112 and classification application 114 may be installed on client device 110 or capture device 115.

[0030] The training application 112 and the classification application 114 may be executable applications configured to interface with the server 100. The training application 112 may send requests to the server 100 to train a learning model to classify documents using image analysis. The classification application 114 may be configured to send requests to the server 100 to classify documents using the learning model. The classification application 114 may also be installed and executed on a third-party user device. In this regard, an authorized third party may send requests to classify documents using the server 100.

[0031] The classification application 114 may include a mail or messaging service, which may further include an inbox configured to store or hold documents to be classified.

[0032] Database 120 may be one or more data storage devices configured to store documents of various types and formats. Database 120 may store structured or unstructured data.

[0033] The user device 130 may include a camera 132 and a user classification application 134. The user classification application 134 may be configured to implement a well-trained learning model to classify documents in images captured by the camera 132. Alternatively, a user may upload documents for classification using the user classification application 134 (e.g., from email, local memory, external memory coupled to the user device 130, etc.), and the user classification application 134 may classify the uploaded documents. The user device 130 may be operated by a user, such as a patient. The patient may use the user classification application 134 to receive a classification for a given PV document. The user classification application 134 may also send the classification for a given PV document to the server 100. The server 100 may store the classification in the database 120.

[0034] In some embodiments, the user classification application 134 can send a request to the server 100 to classify a document. The user classification application 134 can include a mail or messaging service. Further, the mail or messaging service can include an inbox configured to store or hold the document to be classified.

[0035] The server 100 may include a learning engine 101, a conversion engine 103, and a script 109. The script 109 may be an automated script or bot configured to poll an inbox of an email or messaging service included in the classification application 114 or the user classification application 134 to detect documents to be classified. The script 109 may be scheduled to periodically poll an inbox of an email or messaging service included in the classification application 114 or the user classification application 134. Once the script 109 retrieves a document from an inbox of an email or messaging service included in the classification application 114 or the user classification application 134, the script 109 may send the document to the conversion engine 103.

[0036] The conversion engine 103 may be configured to convert documents into image files so that the learning engine 101 can use the image files to classify and prioritize corresponding documents. In some embodiments, the documents are converted directly to image files. In some other embodiments, this is a two-step process, where the electronic document is first converted into, for example, a Portable Document Format (PDF) file, and then the PDF file is converted into an image file. The conversion engine 103 may use a word processing application, such as LibreOffice, developed by The Document Foundation of Berlin, Germany. The word processing application may be configured to save the document as a PDF file. The conversion engine 103 may use a file conversion application to convert the PDF file into an image file, such as Ghostscript. The file conversion application may be configured to raster convert the PDF file into a JPEG image file. Additionally, the conversion engine 103 may be configured to use an image manipulation application to composite all pages of the image file into a single image stack. The single image stack may be a composite of each page of the image file. Thus, the conversion engine 103 may generate a single image stack for each image file. If the image file contains a single page, the conversion engine 103 can convert the single page to a whitescale or grayscale image. The single image stack can include a whitescale or grayscale image representation of the single page of the corresponding document. In other embodiments, the conversion engine 103 can convert the single page to a different scale, and the single image stack of the single page can include a scaled image representation of the single page of the corresponding document. The image file may include a single image stack. The single image stack may include an image fingerprint that a CNN algorithm can use to classify the image using image analysis.Image files corresponding to documents may be provided as input to the learning engine 101 for training a learning model.

[0037] Image fingerprinting provides the large amount of data necessary to train and test learning models. For example, as an experiment, a learning model can be trained and tested using ICSR documents received over, say, a 10-year period. This represents a 12-fold increase in the number of ICSRs utilized when compared to traditional training corpus approaches for text-based learning. Using image fingerprinting, maintenance and updates of the learning model can be set up on a regular schedule, during which all ICSRs received during that period are provided to the learning model as inputs for tuning.

[0038] The learning engine 101 may include an API 102, a first learning model 104, a second learning model 106, and a third learning model 108. The API 102 may be configured to receive a request to classify a document from the conversion engine 103. The API 102 may forward the request to the first learning model 104.

[0039] The first learning model 104, the second learning model 106, and the third learning model 108 may be configured to recursively implement a deep machine learning algorithm, such as a convolutional neural network (CNN), to classify and prioritize documents using their image representations. The first learning model 104, the second learning model 106, and the third learning model 108 may be trained to generate first, second, and third classifications of documents, respectively. Each of these classifications is described in further detail below. In some embodiments, fewer or more learning modules may be used to classify a document set. For example, additional learning models may be used to assign additional classifications to a document set. Alternatively, a single learning model may be configured to assign multiple classifications to a document set.

[0040] 2A is a block diagram of data flow in a system for classifying documents using image analysis. As described above, the system for classifying documents using image analysis may include a server 100, a client device 110, and a database 120. The client device 110 may include a training application 112 configured to interface with the server 100.

[0041] The training application 112 may send a request 200-1 to use a document set to train the first learning model 104, the second learning model 106, and the third learning model 108 of the learning engine 101. The request 200-1 may include a query 200-2 to retrieve the document set and corresponding metadata from the database 120.

[0042] The server 100 can receive a request 200-1. The conversion engine 103 can send a query 200-2 to the database 120 to retrieve a document set and corresponding metadata. The metadata corresponding to the documents can include first, second, and third classifications of the documents identified by a subject matter expert (SME) or a third party. Alternatively, the first, second, and third classifications included in the metadata can be pre-assigned by the first, second, and third learning models 104-108. The database 120 can send a response 200-3 to the server 100. The response 200-3 can include a document set and corresponding metadata for each document. The document set can be provided in various formats, such as Microsoft Word, Microsoft Excel documents, PNG, TIFF, JPG, raw, GIF, PDF, TXT files, handwritten notes, HTML, XML, email, scanned documents, etc. A given document can be a combination of multiple formats. For example, a given document can be in .doc format but can also include embedded JPEG images. In another example, part of a given document may be an email message and another part may be in .doc format.

[0043] The conversion engine 103 can verify that each document in the document set is in a common format. While reference is made herein to PDF format, other formats may additionally or alternatively be used. If a given document is not in PDF format, the conversion engine 103 can convert the given document to PDF format. The conversion engine 103 can convert each document currently in PDF format into an image file. Thus, the conversion engine 103 can generate an image file for each document, resulting in a set of image files. In some embodiments, the conversion engine 103 can convert a given document from the document set directly to an image file, even if the given document is not in PDF format. The image file may be, for example, a JPEG image file. The JPEG image file may contain an image representation of the corresponding document.

[0044] Additionally, the conversion engine 103 can generate a single image stack for each image file. The single image stack may include a composite of all pages of a given image file. The single image stack for each document may be, for example, a JPEG image. If the image file includes a single page, the conversion engine 103 can convert the single page into a whitescale or grayscale image. The single image stack may include a whitescale or grayscale image representation of the single page of the corresponding document. In other embodiments, the conversion engine 103 can convert the single page to a different scale, and the single image stack for the single page may include a scaled image representation of the single page of the corresponding document. If the given document has multiple pages, the single image stack for the document may include all pages of the given document combined into a single JPEG image, as further described with respect to FIG. 6 . Each image file may include a corresponding single image stack for the image file. By combining the pages of the image files, the conversion engine 103 creates an image fingerprint. A CNN algorithm can use the image fingerprint to classify the image file using image analysis.

[0045] The conversion engine 103 can send the image files, parameters for characterizing the image files of the document set, and metadata corresponding to the document set to the learning engine 101 to train the first learning model 104, the second learning model 106, and the third learning model 108 to classify each document in the document set. The parameters for characterizing the image files can include the sender of the document (corresponding to the image file), the time the document was received, and an urgency value associated with the document. The parameters can include characteristics related to the image files, such as watermarks, markings, redactions, etc. The learning engine 101 can receive the image files of the document set, the parameters characterizing the image files, and metadata corresponding to the document set.

[0046] The learning engine 101 can train the first, second, and third learning models 104-108 to assign first, second, and third classifications to documents using a set of image files. The learning engine 101 can train the first, second, and third learning models 104-108 sequentially or in parallel. Specifically, the learning engine 101 can instruct the first, second, and third learning models 104-108 to recursively assign first, second, and third classifications to a set of documents corresponding to the set of image files and validate the classifications against document metadata. The first, second, and third learning models 104-108 can implement one or more CNN algorithms to assign and validate their respective classifications. Furthermore, the CNN algorithms can use image analysis to identify patterns in the set of image files and identify specific features in the images. In some embodiments, the CNN algorithms can identify patterns in the set of image files using parameters that describe the characteristics of the image files. For example, a CNN algorithm can take into account edits, watermarks, or other markings when performing image analysis. CNN algorithms can use learned features to classify a given document that corresponds to a given image file. CNN algorithms are described in more detail with reference to FIG. 3. While CNN algorithms are particularly useful for image processing and are therefore referred to herein, those skilled in the art will understand that other statistical learning models can be used in addition to or instead of CNNs.

[0047] The first, second, and third learning models 104-108 may be configured to use a CNN algorithm to identify first, second, and third patterns, respectively, in image representations of alphanumeric text across the set of image files. The first, second, and third patterns may include similarities in features extracted from other image files. Additionally, the first, second, and third patterns may include differences in features extracted from other image files. The first, second, and third learning models 104-108 may use the first, second, and third patterns to identify or recognize features within a given image file. Based on the identification or recognition of the features of a given image file, the first, second, and third learning models 104-108 may identify first, second, and third attributes, respectively, associated with the given image file. Thus, the first, second, and third learning models 104-108 may identify first, second, and third attributes, respectively, for each image file in the set of image files. Furthermore, the first, second, and third learning models 104-108 can use the first, second, and third attributes to assign first, second, and third classifications, respectively, to documents in the document set.

[0048] The first, second, and third learning models 104-108 can respectively verify the first, second, and third classifications of each subset of documents in the document set. Specifically, the first, second, and third learning models 104-108 can compare each document's metadata with the classification of each document in the document set. As noted above, the metadata for each document can include the document's first, second, and third classifications, as identified by an SME or a third party. If the first, second, and third classifications assigned to a given document by the first, second, and third learning models 104-108 match the given document's metadata, the respective learning model (first, second, or third) can determine that the assigned classification is correctly assigned. If the first, second, or third classification assigned to a given document by the first, second, or third learning model 104-108 does not match the metadata of the given document, the respective learning model (first, second, or third) can determine that the assigned classification is incorrectly assigned. The first, second, and third learning models 104-108 can generate validation results. The validation results may include correctly classified documents and incorrectly classified documents.

[0049] The learning engine 101 can instruct the first, second, and third learning models 104-108 to recursively run a CNN algorithm on each image file in the set of image files to classify each corresponding document in the set of documents for a predetermined number of iterations. The first, second, and third learning models 104-108 can generate validation results after each iteration. The first, second, and third learning models 104-108 can use the validation results to optimize the CNN algorithm to better identify features within a given image file.

[0050] The predetermined number of iterations or other parameters may be included in the request 200-1. Alternatively, the predetermined number of iterations may be pre-programmed by the learning engine 101. In response to completing the predetermined number of iterations and other parameters by each of the first, second, and third learning models 104-108, the first, second, and third learning models 104-108 may be considered fully trained. The server 100 may send a response 200-4 to the client device 110. The response 200-4 may indicate that the first, second, and third learning models 104-108 have been fully trained to assign the first, second, and third classifications to a given document.

[0051] The capture device 115 can receive another document for classification. The another document can be a new document or a document previously received by the capture device 115. As an example, the capture device 115 can receive the another document for classification via email. Alternatively, the capture device 115 can receive the another document as a bulk capture of documents received in batches of documents. The another document can be provided in a variety of formats, such as a Microsoft Word document, a Microsoft Excel document, PNG, JPG, Raw, GIF, TIFF, PDF, TXT file, handwritten notes, XML, email, HTML, or scanned document. The another document can also be a combination of formats. For example, the another document can be in .doc format but include an embedded JPEG image. In another example, part of the another document can be an email message and another part can be in .doc format.

[0052] The decision engine 116 can determine whether the other document needs to be classified. For example, the decision engine 116 can electronically scan the other document to determine the document's content. If the decision engine 116 can identify a majority of the other document's content, the decision engine 116 can determine that the document does not need to be classified. The decision engine 116 may also be able to independently assign a classification to the document. If the decision engine 116 cannot identify a threshold amount of the other document's content, the decision engine 116 can determine that the other document needs to be classified. The decision engine 116 can send a request 202-1 to the client device 110 to classify the other document. The decision engine 116 can store the other document in the database 120. The request 202-1 can also include parameters associated with the other document. The parameters can include the sender of the other document, the date and time the other document was received, the urgency of the other document, etc. The parameters can also include any characteristics of the other document, such as redactions, watermarks, or other markings.

[0053] The classification application 114 can receive the request 202-1 and can send a request 202-2 to the server 100 to classify another document using the currently trained first, second, and third learning models 104-108. In some embodiments, the classification application 114 can persist or store the request 202-2 in an inbox of a mail or messaging application. The request 202-2 can include the document to be classified.

[0054] The server 100 can receive a request 202-2. The request 202-2 can include a query 202-3 to retrieve another document from the database 120. The request 202-2 can also include parameters for the other document. The transformation engine 103 can send the query 202-3 to the database 120. The database 120 can send a response 202-4 to the server 100. The response 202-4 can include another document.

[0055] In some embodiments, the script 109 can periodically poll the inbox of a mail or messaging application included in the classification application 114. The script 109 can retrieve the request 202-2 and the other document included in the request 202-2. The script 109 can determine whether the other document is in a predetermined format (e.g., PDF format). If the other document is not in the predetermined format, the script 109 can convert the other document to the predetermined format. The script 109 can forward the other document to the conversion engine 103.

[0056] The conversion engine 103 can verify that the other document is in PDF format. If the other document is not in PDF format, the conversion engine 103 can convert the other document to PDF format. The conversion engine 103 can convert the other document to an image file. In some embodiments, the conversion engine 103 can convert the other document to an image file even if the other document is not in PDF format. Additionally, the conversion engine 103 can composite all pages of an image file into a single image stack. If the image file contains a single page, the conversion engine 103 can convert the single page into a whitescale or grayscale image. The single image stack can include a whitescale or grayscale image representation of the single page of the corresponding document. In other embodiments, the conversion engine 103 can convert the single page to different scales, and the single image stack of the single page can include a scaled image representation of the single page of the corresponding document. If the image file contains multiple pages, the single image stack of the document may include a composite of all pages of the image file as a single JPEG image. The image file may include a single image stack. The transformation engine 103 can send the image file to the learning engine with a request to use the image file to classify another document.

[0057] The API 102 can receive an image file and a request to classify a document using the image file. The API 102 can forward another document and a request to classify the document using the image file to the first, second, and third learning models 104-108. In response to receiving the image file and the request, the first, second, and third learning models 104-108 can use a trained CNN algorithm in each learning model to identify first, second, and third attributes associated with the image file, respectively. The trained CNN learning model for each of the first, second, and third learning models 104-108 can take into account parameters of the image file, including edits, watermarks, or other markings. The CNN algorithm can use image analysis to identify the first, second, and third attributes associated with the image file. The first, second, and third learning models 104-108 can assign first, second, and third classifications to another document based on the first, second, and third attributes, respectively. The first, second, and third learning models 104-108 may also assign a confidence score to each classification, which indicates the estimated accuracy of each classification. As mentioned above, the CNN algorithm is described in further detail with reference to FIG. 3.

[0058] The learning engine 101 can use the first, second, and third classifications for another document to generate a priority value. The priority value can indicate whether the document needs someone's attention. For example, a higher priority value can indicate a more immediate response is needed for the document. The learning engine 101 can also use factors such as productivity, capacity, and demand. For example, the learning engine 101 can identify already classified documents that are awaiting action. The learning engine 101 can compare the classification of a classified document awaiting action with the first, second, and third classifications of another document. The first classification corresponds to the effectiveness of the other document. The second classification corresponds to an indicator of the severity of the content of the other document. The third classification corresponds to one or more factors corresponding to the severity of the content.

[0059] Additionally, the learning engine 101 can identify parameters associated with another document. For example, the learning engine 101 can determine the sender of the other document, the time the other document was received, and an urgency value associated with the other document. The learning engine 101 can also determine productivity, capacity, and demand factors associated with the human or agent resources processing the already classified document. The learning engine 101 can determine a priority value for the other document using the first, second, and third classifications of the already classified document awaiting action, the first, second, and third classifications of the other document, parameters of the other document, and the productivity, capacity, and demand of the human resources processing the document. The priority value can indicate the position in the queue where the other document needs to be processed. The priority value may indicate that there is a large volume of documents that need to be processed, requiring more human or agent resources.

[0060] The learning engine 101 can send a response 202-5 to the client device 110. The response 202-5 can include the first, second, and third classifications of the other document and a priority value. The classification application 114 can display the first, second, and third classifications on the display of the client device 110. The output of the first, second, and third classifications can be expressed as labels, percentages, numeric ranges, etc. The labels can be binary or other classifications. The percentages or numeric ranges can indicate the values ​​associated with the document with respect to the respective classifications. The response 202-5 can include the first, second, and third classifications and the priority value. The learning engine 101 can also store the assigned classification as metadata for the other document in the database 120. The metadata can be used to update and optimize the first, second, and third learning models 104-108.

[0061] In some embodiments, a user of the client device 110 can use the classification application 114 to approve or reject the priority value, the first classification, the second classification, and the third classification included in the response 202-5 based on the user's understanding of the different documents. The approval or rejection of the priority value, the first classification, the second classification, and the third classification can be transmitted to the server 100. The learning engine 101 can update and optimize the first, second, and third learning models 104-108 based on the approval or rejection. The first, second, and third learning models 104-108 can optimize their respective CNN algorithms based on the approval or rejection. For example, the first, second, and third learning models 104-108 can better understand how to classify documents based on the approval or rejection. In this regard, the first, second, and third learning models 104-108 can be continuously updated as they classify their respective different documents. For example, the client device 110, the server 100, the learning engine 101, or the database 120 can store the user feedback for future optimization of the learning model.

[0062] As a non-limiting example, the system described above for classifying documents using image analysis can be used to classify PV documents. PV documents may include ICSR documents (such as literature articles or clinical reports). Additionally, PV documents may include information about the patient, geography, adverse effects, ICSR quality and compliance characteristics, benefit-risk characteristics, product details, study details, consumer complaints, legal concepts, or other medical concepts related to the use of FDA-regulated products. Companies in the pharmaceutical industry may need to process PV documents to determine whether any action is required for a particular product.

[0063] The PV workflow may include three processing blocks: case intake, case processing, and case reporting. At intake, the PV department receives PV documents from various sources around the world in various formats and languages. Reports come from various reporters, medical professionals and non-medical professionals, and various mediums such as email, fax, mail, and telephone. To meet predefined regulatory guidelines, several key assessments are made at case intake, which are crucial in routing cases according to their severity.

[0064] Compliance with regulatory authorities is determined based on the likelihood of reporting to country-specific regulatory authorities within each specified timeline. Therefore, upfront prioritization needs to be accurate to limit the spread of work efforts performed on less urgent reports. The prioritization assessment may include the following key characteristics: case validity (valid or not valid), case severity (serious or not serious), relevance (related or not related to the suspected product), and adverse effect SRE (labeled or not). Case validity can indicate whether the PV document is a valid document. Case severity can indicate whether the adverse effect described in the PV document is serious or not serious. SRE can indicate whether the adverse effect is serious, relevant, and expected (e.g., labeled on the product).

[0065] If the adverse effects described in a valid PV document are severe and unexpected, the company may need to take action regarding a particular product. As a result, a first learning model 104 may be trained to classify a given PV document as valid or invalid. A second learning model 106 may be trained to classify a given PV document based on the severity of the adverse effects described in the PV document. A third learning model 108 may be trained to classify a given PV document based on the severity, relevance, or expectancy (SRE) of the adverse effect.

[0066] The first learning model 104 can assign a binary value (effective or ineffective). The second and third learning models 106-108 can assign percentages or numerical ranges. For example, the second learning model 106 can assign a numerical range or percentage indicating the severity of the adverse effect. The third learning model 108 can assign a numerical range or percentage associated with the SRE of the adverse effect described in the PV document.

[0067] The learning engine 101 can generate a priority value using a combination of a binary classification of effective or ineffective, a numerical range or percentage indicating the severity of the adverse effect, and a numerical range or percentage associated with the SRE of the adverse effect described in the PV document. Additionally, the learning engine 101 can also use process information such as productivity, capacity, and demand to determine the priority value. The priority value can indicate that the case corresponding to the PV document needs to be addressed based on a predetermined timeline. For example, as described above, regulatory compliance is determined based on the reportability to country-specific regulatory authorities within each specified timeline. A PV document indicating a serious and effective adverse effect of a product may need to be reported or addressed within a predetermined timeline.

[0068] The training application 112 can send a request 204-1 to train the first, second, and third learning models 104-108. This can include a query 204-2 to retrieve a PV document set and corresponding metadata. The corresponding metadata can include the efficacy, severity, or classification of the adverse effect (SRE). The classification can be assigned by an SME or a third party. For example, a training set of PV documents can include approximately 500,000 ICSR documents, of which approximately 80% are used to train the models and approximately 20% are used to validate the models.

[0069] The conversion engine 103 can send a query 204-2 to the database 120 to retrieve the ICSR document set and corresponding metadata. The database 120 can return a response 204-3 to the server 100. The response 204-3 can include the ICSR document set and corresponding metadata. The conversion engine 103 can identify the format of each of the ICSR documents. If the ICSR is not in an image format, the conversion engine 103 can convert the format of the ICSR document to an image file. In some embodiments, the document undergoes an intermediate step of PDF conversion before the image conversion. In such embodiments, if the ICSR is not in PDF format, the conversion engine 103 can convert the format of the ICSR document to PDF format.

[0070] The conversion engine 103 can then convert each ICSR document into an image file to generate a set of image files. As described above, the conversion engine 103 can convert a given document into an image file even if the given document is not in PDF format. The conversion engine 103 can generate a single image stack for each image file. The single image stack for each image file can include a composite of the pages of the corresponding image file. Each image file can include a corresponding single image stack. The conversion engine 103 can provide the set of image files as input parameters to the learning engine 101. The conversion engine 103 can also provide parameters for each image file to the learning engine 101. The parameters can include the sender of the document (corresponding to the image file), the date and time the document was received, the urgency associated with the document, etc. The parameters can also include characteristics related to each image file, including redactions, watermarks, other markings, etc.

[0071] The learning engine 101 can use the set of image files to train first, second, and third learning models 104, 106, and 108 to assign first, second, and third classifications, respectively, to each of the ICSR documents corresponding to the set of image files. The first, second, and third learning models 104-108 can assign the first, second, and third classifications using a CNN algorithm. The CNN algorithm can perform image analysis on each image file to assign the first, second, and third classifications. Furthermore, the CNN algorithm can take into account parameters of each image file, such as edits, watermarks, and markings, when performing the image analysis.

[0072] The first learning model 104 can use a CNN algorithm to identify one or more attributes associated with a given image file corresponding to a given ICSR document that indicate validity. The first learning model 104 can use the identified attributes to classify the given ICSR document as valid or invalid. The first learning model 104 can assign a classification of valid or invalid for each ICSR document corresponding to the set of image files. The first learning model 104 can verify the classification of valid or invalid based on the ICSR document subset using metadata corresponding to the ICSR documents. The first learning model 104 can generate validation results. The validation results can indicate whether the first learning model 104 accurately assigned a classification to the ICSR document.

[0073] The second learning model 106 can use a CNN algorithm to identify one or more attributes associated with a given image file corresponding to a given ICSR document that indicate severity. As a non-limiting example, the CNN algorithm can identify pixel intensities associated with a given ICSR document by performing image analysis on the corresponding image file. For example, the second learning model 106 can determine that ICSR documents containing serious adverse effects are likely to correspond to image files with higher pixel intensities. The second learning model 106 can use the identified attributes to classify the given ICSR document as serious or non-serious. The second learning model 106 can assign a severity classification to each ICSR document corresponding to the set of image files. The classification may be a binary value, a numeric range, or a percentage indicating the severity of the adverse effect described in the given ICSR document. The second learning model 106 can validate the classification of serious or non-serious for a subset of ICSR documents using metadata corresponding to the ICSR documents. The second learning model 106 can generate validation results. The validation results may indicate whether the second learning model 106 correctly assigned classifications to the ICSR documents.

[0074] Table 1 contains example experimental results when training the second learning model 104 to assign severity classifications to ICSR documents and pixel densities of the respective image files.

[0075] [Table 1]

[0076] On average, "severe" cases tend to be more "black" (mean density = 195) compared to "non-severe" cases (mean density = 207). Furthermore, at the 25%, 50%, and 75% quantiles, there is a similar increase in "black" pixels for "severe" cases compared to "non-severe" cases.

[0077] The third learning model 108 can use a CNN algorithm to identify one or more attributes associated with a given image file corresponding to a given ICSR document that indicate an SRE of an adverse effect. The third learning model 108 can use the one or more attributes to assign a classification associated with the SRE of an adverse effect described in the given ICSR document. The third learning model 108 can assign a classification associated with the SRE of an adverse effect described in each ICSR document corresponding to the set of image files. The third learning model 108 can validate the classification associated with the SRE of an adverse effect described in a given ICSR document for the ICSR document subset using metadata corresponding to the ICSR documents. The third learning model 108 can generate validation results. The validation results can indicate whether the third learning model 108 accurately assigned the classification associated with the SRE of an adverse effect included in the ICSR document.

[0078] The first, second, and third learning models 104-108 can use the validation results to optimize the CNN algorithm of their respective learning models (first, second, and third) and re-perform the identification, classification, and validation steps. The first, second, and third learning models 104-108 can generate validation results after each iteration of the identification, classification, and validation steps. Furthermore, the first, second, and third learning models 104-108 can optimize the CNN algorithm after each iteration of the identification, classification, and validation steps. The first, second, and third learning models 104-108 can re-perform the identification, classification, and validation steps a predetermined number of iterations. Alternatively, the first, second, and third learning models 104-108 can re-perform the identification, classification, and validation steps until a desired level of accuracy in assigning classifications to documents is achieved.

[0079] The identification, classification, and verification steps are re-run, and once a predetermined number of iterations or a desired accuracy in classifying documents is achieved, the first, second, and third learning models 104-108 may be considered fully trained. Once the CNN algorithms of the first, second, and third learning models 104-108 are fully trained, the learning engine 101 may send a response 204-4 to the client device 110. The response 204-4 may indicate that the CNN algorithms of the first, second, and third learning models 104-108 are fully trained.

[0080] Once the CNN algorithm is trained, the capture device 115 can receive another PV document for classification. The another PV document may be another ICSR document. The decision engine 116 can determine whether the another ICSR document needs to be classified. For example, the decision engine 116 can electronically scan the another ICSR document to determine the content of the another ICSR document. If the decision engine 116 can identify a majority of the content of the other ICSR document, the decision engine 116 may determine that the ICSR document does not need to be classified. If the decision engine 116 cannot identify a threshold amount of the content of the other document, the decision engine 116 may determine that the other ICSR document needs to be classified. The decision engine 116 can send a request 206-1 to the client device 110 to classify the other ICSR document. The decision engine 116 can store the other document in the database 120. The request 206-1 may include parameters associated with the other ICSR document. The parameters may include the sender of the other ICSR document, the date and time the other ICSR document was received, the urgency of the other ICSR document, etc. The parameters may also include any characteristics about the other ICSR document, such as redactions, watermarks, or other markings.

[0081] The classification application 114 can receive the request 206-1 and send a request 206-2 to the server 100 to classify another ICSR document. The other ICSR document may be an ICSR document that is not included in the training set of ICSR documents. The request 206-2 can include a query 206-3 to retrieve another ICSR document from the database 120.

[0082] In some embodiments, the classification application 114 can retain or store the request 206-2 in an inbox of a mail or messaging application. The request 206-2 may include another ICSR document to be classified.

[0083] The server 100 can receive the request 206-2, and the transformation engine 103 can send a query 206-3 to the database 120 to search for the additional ICSR document. In some embodiments, the script 109 can periodically poll the inbox of an email or messaging application included in the classification application 114. The script 109 can retrieve the request 206-2 and the additional ICSR document included in the request 206-2. The script 109 can determine whether the additional ICSR document is in a predetermined format (e.g., PDF format). If the additional ICSR document is not in the predetermined format, the script 109 can convert the additional ICSR document to the predetermined format. The script 109 can forward the additional ICSR document to the transformation engine 103.

[0084] The conversion engine 103 can verify that the other ICSR document is in PDF format. If the other ICSR document is not in PDF format, the conversion engine 103 can convert the other ICSR document to PDF format. As described above, the conversion engine 103 can generate an image file of the other ICSR document even if the other ICSR document is not in PDF format. The conversion engine 103 can also generate a single image stack of the image file. The single image stack can include a composite of all pages of the image file. The image file can include a single image stack. The conversion engine 103 can send the image file along with the parameters of the image file to the learning engine 101.

[0085] The API 102 can receive an image file and a request to classify another ICSR document. The API 102 can forward the image file to first, second, and third learning models 104-108 for sequentially or in parallel assigning first, second, and third classifications to the other ICSR document, respectively. The first, second, and third learning models 104-108 can use a CNN algorithm to identify one or more attributes associated with the image file corresponding to the other ICSR document. The CNN algorithm can perform image analysis on the image file to identify one or more attributes associated with the image file. Additionally, the CNN algorithm can take into account parameters of the image file, including edits, watermarks, or markings, when performing the image analysis.

[0086] In certain embodiments, the CNN algorithm implemented by the first learning model 104 can identify one or more attributes indicative of the effectiveness of another ICSR document. The first, second, and third learning models 104-108 can assign first, second, and third classifications, respectively. The first classification corresponds to the effectiveness of the other document. The second classification corresponds to an indicator of the severity of the content of the other document. The third classification corresponds to one or more factors corresponding to the severity of the content.

[0087] More specifically, the first learning model 104 can assign a classification of valid or invalid to another ICSR document based on the identified attributes. The CNN algorithm implemented by the second learning model 106 can identify one or more attributes of the image file that indicate severity. The second learning model 106 can assign a classification that indicates the severity of an adverse effect described in another ICSR document based on the identified attributes. As a non-limiting example, severity can be correlated with health risk. The higher the health risk, the higher the severity. Conversely, the lower the health risk, the lower the severity. The CNN algorithm implemented by the third learning model 108 can identify one or more attributes that indicate an SRE of an adverse effect included in another ICSR document. The third learning model 108 can assign a classification associated with the SRE of an adverse effect included in the ICSR document based on the identified attributes.

[0088] The learning engine 101 can generate output including a classification assigned to the ICSR document. The learning engine 101 can also output a confidence score indicating the estimated accuracy of the first, second, and third classifications. The learning engine 101 can use the assigned classification to generate a priority value (e.g., high, medium, low) for the ICSR document. The priority value can indicate a timeline within which action should be taken on the ICSR document. The learning engine 101 can also use factors such as productivity, capacity, and demand.

[0089] For example, the learning engine 101 can identify an already classified ICSR document that is awaiting action. The learning engine 101 can compare the classification of an ICSR document that has been classified and is awaiting action with the first, second, and third classifications of another ICSR document. Additionally, the learning engine 101 can identify parameters associated with another document. For example, the learning engine 101 can determine the sender of another ICSR document, the time the other ICSR document was received, and the urgency value associated with the other document. This allows a particular regulatory authority to identify whether another ICSR document is associated with the ICSR document or whether the ICSR document is part of a clinical trial. The learning engine 101 can also determine productivity, capacity, and demand factors associated with the human resources processing the already classified ICSR document. The learning engine 101 can determine a priority value for another ICSR document using the first, second, and third classifications of already classified ICSR documents awaiting action, the first, second, and third classifications of another ICSR document, parameters of the other ICSR document, and the productivity, capacity, and demand of human resources to process the ICSR document. The priority value can indicate the position in the queue where another ICSR document needs to be processed. The priority value can also indicate the need for more human resources due to a high demand for ICSR documents that need to be processed.

[0090] The priority value may be included in the output. The learning engine 101 may include the output in a response 206-5 and send the response 206-5 to the client device 110. The training application 112 may render the output on a display of the client device 110. The learning engine 101 may also store the assigned classification as metadata for the ICSR document in the database 120.

[0091] In some embodiments, the learning engine 101 can sequentially classify an ICSR document using the first, second, and third classification learning models 104-108. That is, the learning engine 101 can first instruct the first learning model 104 to classify the ICSR document to determine whether the document is a valid document. In response to determining that the ICSR document is valid, the learning engine 101 can instruct the second learning model 106 to determine whether the adverse effects described in the ICSR document are serious or non-serious. Alternatively, in response to determining that the ICSR document is not valid, the learning engine 101 can generate an output indicating that the ICSR is not valid. Further, the output can include a confidence score indicating the accuracy of the classification that the ICSR document is not valid, and can be sent to the client device 110 without instructing the second or third learning models 106-108 to classify the ICSR document.

[0092] In response to the second learning model 106 classifying the adverse effect described in the ICSR document as serious, the learning engine 101 can instruct the third learning engine 108 to assign a classification associated with the SRE of the adverse effect included in the ICSR document. Alternatively, in response to determining that the adverse effect in the ICSR document is not serious, the learning engine 101 can generate an output indicating that the adverse effect in the ICSR document is not serious and send the output to the client device 110 without instructing the third learning model 108 to classify the ICSR document. The output can also include a confidence score indicating the estimated accuracy of the adverse effect in the ICSR document being not serious.

[0093] The system for classifying documents using image analysis may also be used in other fields such as law, business, finance, medicine, insurance, education, etc. For example, the system may be used to classify legal documents in the field of e-discovery. Additionally, the system may be used to classify business documents such as invoices, purchase orders, and order forms.

[0094] Systems for classifying documents using image analysis can be used in other domains by training respective learning models to identify specific features of documents in the respective domains. For example, a learning model can be trained to identify document features by converting a document set (for a particular domain, such as e-discovery, retail, business, insurance, energy, transportation and logistics, or healthcare) into a single image stack set and training the learning model using the single image stack set. The features can be used to identify document attributes so that the system can properly classify the documents.

[0095] For example, in the field of e-discovery, systems that classify documents using image analysis may be used to classify documents as privileged or non-privileged by training learning models to identify features of privileged documents based on the image files corresponding to such documents.

[0096] FIG. 2B is a block diagram illustrating a user device for classifying documents according to an example embodiment. In one embodiment, a user can launch a user classification application 134 to classify documents. The user classification application 134 can implement a fully trained learning engine to classify documents. The user classification application 134 can include a learning engine trained to assign classifications to documents, such as the first, second, and third learning models 104-108. More specifically, the user classification application 134 can include a learning engine for classifying documents using CNNs in a specific domain (e.g., medical, regulatory, legal, etc.).

[0097] Launching the user classification application 134 enables the camera 132. A user can use the camera 132 to capture an image of a document. In some embodiments, a user can use the user classification application 134 to upload a document from email, local memory, external memory coupled to the user device 130, etc. In some embodiments, the user classification application 134 can be connected to Internet of Things (IoT) devices such as smart watches, sensors, appliances, computing devices, etc. The user classification application 134 can receive documents from IoT devices.

[0098] The user classification application 134 can convert the document images into an image file containing a single stack of images, as described above with respect to the conversion engine 103. Alternatively, the user classification application 134 can determine whether the uploaded document is an image file or a PDF. If the uploaded document is not an image file or a PDF, the user classification application 134 can convert the uploaded document into an image file. Additionally, the user classification application 134 can convert the image file into an image file containing a single stack of images, as described above with respect to the conversion engine 103.

[0099] For example, the user classification application 134 may be configured to assign a classification to a document using an image file, as described above with respect to Figure 2. The user classification application 134 may generate output including first, second, and third classifications, a priority assigned to the document based on the first, second, and third classifications, and a confidence score, which indicates the estimated accuracy of the classification.

[0100] As a non-limiting example, a user can interact with the user classification application 134 to classify a medical document. The medical document may be, but is not limited to, a lab report, an electrocardiogram (EKG) report, a genomics report, a blood test report, etc. The user classification application 134 can use a CNN to assign a classification indicating the severity of the content of the medical document. The user classification application 134 can also determine whether the user should consult a medical professional based on the severity of the content of the medical document. The user classification application 134 can generate output including the classification, a recommendation whether the user should consult a medical professional, and a confidence score. The confidence score indicates the estimated accuracy of the classification. The user classification application 134 can display the classification and a determination whether the user should consult a medical professional on the user device 130.

[0101] The user classification application 134 can generate a classification and determine whether the user should consult a medical professional locally on the user device 130. Furthermore, the user classification application 134 can generate a classification and determine whether the user should consult a medical professional without identifying or extracting personally identifiable information (PII) from the document, thereby mitigating security risks associated with the disclosure of PII (e.g., through phishing, hacking, cyber-attacks, etc.).

[0102] For example, the document may be an EKG report, so that the user can identify the severity of the EKG report based on the classification assigned by the user classification application 134 without transmitting or disclosing any PII.

[0103] In some embodiments, the user classification application 134 can submit a document or image file containing a single image stack of a document to the server 100 for classification. The first, second, or third learning models 104-108 can use the image file to classify the document.

[0104] 3 is a block diagram of an example of a learning model implementing a CNN algorithm, according to an exemplary embodiment. FIG. 3 will be described with reference to FIGS. 1-2.

[0105] As described above, the first, second, and third learning models 104-108 may implement a supervised deep learning algorithm, such as a CNN, to classify documents based on their corresponding image files. The CNN algorithm 300 may be trained to classify the documents. Each of the first, second, and third learning models 104-108 may implement a different embodiment of the CNN algorithm 300.

[0106] The CNN algorithm 300 may be trained in two phases: a forward phase and a backward phase. The forward phase includes convolutional layers 304 and 308, pooling layers 306 and 310, and fully connected layers 312 and 314. The convolutional layers 304 and 308 may apply filters to the input image file 302 to generate feature maps. The pooling layers 306 and 310 may generate reduced feature maps. The fully connected layers 312 and 314 may then classify image features using weights and biases to generate an output 316. The output 316 may be a classification of the input image file 302. The values ​​of the filters, weights, and biases may be parameters received by the CNN algorithm 300 (e.g., from the client device 110).

[0107] In the backward phase, the CNN algorithm 300 can use backpropagation to determine whether the CNN algorithm was able to correctly classify the image. The CNN algorithm can then update the filter, weight, and bias values ​​using a gradient descent algorithm and re-run the forward phase on the input image file 302. Thus, the CNN algorithm can be trained to classify the input image file 302 using feature learning and classification.

[0108] As an example, the CNN algorithm 300 may be trained using a set of image files, including an input image file 302. The input image file 302 may correspond to a document containing alphanumeric text. The input image file 302 may include an image representation of the document. Additionally, the input image file 302 may be a composite of all pages of the document. As a non-limiting example, the input image file 302 may have a size of 160x160 pixels, which may be a 160x160 dimensional array. The 160x160 dimensional array may be flattened into a 25600 dimensional vector. Each component of the vector may include a value between 0 and 1. The value may represent the intensity of the pixel. The input image file 302 may include a unique JPEG "fingerprint" in grayscale format.

[0109] In the convolutional layer 304, the CNN algorithm 300 can perform feature extraction on the input image file 302. The features may include portions of the input image file 302. For example, the features may be different edges or shapes of the input image file 302. The CNN algorithm can extract different types of features to generate different types of feature maps. For example, the CNN algorithm 300 can apply an array of numbers (e.g., a kernel) across different portions of the input image file 302. A kernel is sometimes referred to as a filter. As described above, different types of filters may be applied to the input image file 302 to generate different feature maps. For example, a filter for identifying shapes in the input image file 302 may be different from a filter for edge detection. Thus, a different kernel may be applied to identify shapes in the input image file 302 than for edge detection. Each kernel may include a different array of numbers. The filter or kernel values ​​are randomly assigned and optimized over time (e.g., using a gradient descent algorithm). The kernel may be applied as a sliding window across different portions of the input image file 302. The kernels can be summed over a given portion of the input image file 302 to generate an output value. The output value can be included in a feature map. The feature map can include the output values ​​from the various kernels applied to each portion of the input image file 302. The generated feature map can be a two-dimensional array.

[0110] In the pooling layer 306, the CNN algorithm 300 can reduce the dimensionality of each feature map generated in the convolutional layer 304. Specifically, the CNN algorithm 300 can extract a portion of a given feature map and discard the remainder. Pooling an image preserves important features. For example, a feature map may contain activated and deactivated regions. Activated regions may contain detected features, while deactivated regions may indicate that portions of the image did not contain the feature. Pooling can remove deactivated regions. In this way, the size of the image is reduced. The CNN algorithm 300 can perform these operations using max or mean pooling in the pooling layer. Max pooling preserves higher values ​​in portions of the feature map while discarding the remaining values. Mean pooling preserves the average values ​​of various portions of the feature map. Thus, the CNN algorithm 300 can generate a reduced feature map for each of the feature maps generated in the convolutional layer 304.

[0111] In the convolutional layer 308, the CNN algorithm 300 can generate additional feature maps based on the reduced feature maps generated in the pooling layer 306. Additionally, in the pooling layer 310, the CNN algorithm 300 can generate additional reduced feature maps based on the feature maps generated in the convolutional layer 308. Multiple different convolutional and pooling layers may be added to the CNN algorithm 300.

[0112] The convolutional layers 304 and 308 may also apply a Rectified Linear Unit (ReLU) function to the input image file 302. The ReLU function is applied to the input image file 302 to remove linearity from the input image file 302. For example, the ReLU function may remove all black components from the input image 302, retaining only gray and white colors. This may result in more abrupt changes in color in the input image 302 and remove linearity from the input image 302.

[0113] The convolutional layers 304 and 308 and the pooling layers 306 and 310 may be used for feature learning, which enables the CNN algorithm 300 to identify desired features in the input image file 302 and accurately classify the input image file 302. Therefore, by optimizing the convolutional layers 304 and 308 and the pooling layers 306 and 310, the CNN algorithm 300 can apply the correct filters to the input image file 302 to extract the necessary features needed to classify the input image file 302.

[0114] In the fully connected layer 312, the CNN algorithm 300 can flatten the reduced feature map generated in the pooling layer 310 into a one-dimensional array (or vector). The fully connected layer is a neural network. The CNN algorithm 300 can perform a linear transformation on the one-dimensional array in the fully connected layer 312. The CNN algorithm 300 can perform the linear transformation by applying weights and biases to the one-dimensional array to generate the fully connected layer 314. The weights and biases can be initially initialized randomly and then optimized over time.

[0115] In the fully connected layer 314, the CNN algorithm 300 can perform a nonlinear transformation, such as an activation layer function (e.g., softmax or sigmoid), to classify the input image file 302. The CNN algorithm 300 can use the classification of the input image file 302 to classify the corresponding document.

[0116] The CNN algorithm 300 can use backpropagation to verify the classification of the input image file 302. As described in FIG. 2, in training mode, the CNN algorithm 300 can receive the image file and metadata of the corresponding document. The metadata can include a classification assigned to the document. The CNN algorithm 300 can compare the classification assigned to the document by the CNN algorithm 300 with the classification included in the document's metadata. If the classification assigned to the document by the CNN algorithm 300 matches the classification included in the document's metadata, the CNN algorithm 300 can determine that the CNN algorithm 300 correctly assigned a classification to the document. Alternatively, if the classification assigned to the document by the CNN algorithm 300 does not match the classification included in the document's metadata, the CNN algorithm 300 can determine that the CNN algorithm 300 incorrectly assigned a classification to the document.

[0117] Backpropagation can also include optimizing input parameters, which can enable the CNN algorithm 300 to more accurately classify documents. The input parameters can include values ​​for kernels, weights, biases, etc. Gradient descent can be used to optimize the parameters. Specifically, gradient descent can be used to optimize the CNN algorithm 300's classification of documents.

[0118] Gradient descent is an iterative process for optimizing the CNN algorithm 300. Gradient descent may update parameters of the CNN algorithm 300, causing the CNN algorithm 300 to classify the input image file 302 and other image files in the image file set based on the updated parameters, and to verify the classification of the image file subset. After each iteration of the CNN algorithm 300 classifying the image file set and verifying the subset of the image file set, the parameters may be further updated (or optimized). Each iteration of the CNN algorithm 300 classifying the image file set and verifying the subset of the image file set may be referred to as an epoch. The CNN algorithm 300 may perform a predetermined number of epochs before being considered fully trained. The predetermined number of epochs may be selected based on the accuracy of the CNN algorithm 300 in classifying image files (or documents).

[0119] Once the CNN algorithm 300 is sufficiently trained, the CNN algorithm 300 can classify additional image files that are not included in the training set of image files. For example, the CNN algorithm 300 can receive additional image files as input. The CNN algorithm 300 can then run the convolutional layers 304 and 308, the pooling layers 306 and 310, and the fully connected layers 312 and 314 on the additional image files to classify the additional image files.

[0120] As a non-limiting example, ICSR documents can be classified using a CNN algorithm 300. The CNN algorithm 300 can be implemented by first, second, and third learning models 104-108 to assign classifications related to efficacy, severity, and SRE to a given ICSR document.

[0121] As described above, the first, second, and third learning models 104-108 may implement different embodiments of the CNN algorithm 300. For example, the first learning model 104 may train the CNN algorithm 300 using a first set of parameters to classify a given ICSR document based on effectiveness. The parameters may be filters, weights, biases, etc. Additionally, the second learning model 106 may train the CNN algorithm 300 using a second set of parameters to classify a given ICSR document based on the severity of adverse effects described in the ICSR document.

[0122] The CNN algorithm 300 can use feature learning to identify specific features of valid documents compared to invalid documents. Similarly, the CNN algorithm 300 can use feature learning to identify specific features of documents describing serious adverse effects compared to documents describing non-serious adverse effects. Additionally, the CNN algorithm 300 can use feature learning to identify specific features of documents that include SREs of adverse effects.

[0123] As described above, the CNN algorithm 300 can run for a predetermined number of epochs before being considered sufficiently trained. A user (e.g., a developer) can determine whether the CNN algorithm is sufficiently trained based on the F-1 score and Area Under the Curve (AUC) score of the CNN algorithm 300. The F-1 score and the Area Under the Curve (AUC) curve can indicate the accuracy of the classification assigned by the CNN algorithm 300.

[0124] Below are some example experimental results when identifying the classification accuracy of a CNN algorithm after a certain number of epochs. After training the CNN algorithm 300 for 25 epochs to assign classifications relevant to effectiveness, the CNN algorithm 300 achieved an F-1 score of 96% and an AUC of 0.99 when compared to the actual classification of ICSR documents. After training the CNN algorithm 300 for 30 epochs to assign severity-related classifications, the CNN algorithm 300 achieved an F-1 score of 81% and an AUC of 0.90 when compared to the actual classification of ICSR documents. After training the CNN algorithm 300 for 25 epochs to assign classifications related to SREs of adverse events, the CNN algorithm 300 achieved an F-1 score of 78% and an AUC of 0.87 when compared to the actual classifications of ICSR documents.

[0125] FIG. 4 is a block diagram of a model approval flow according to an example embodiment. Model approval flow 400 can be used to determine whether a learning model classifies documents with an acceptable level of accuracy. At operation 402, a subject matter expert (SME) or developer submits a request to train a learning model using a document set from database 120. The learning model can implement a CNN algorithm. Thus, the learning model can be trained to classify documents using image representations of the document set, as described above. Training the learning model can include recursively classifying each document in the document set and verifying the classification of each document subset against the metadata of each document. The document subset can be 10% of the document set.

[0126] In operation 404, the learning model can determine whether the classification accuracy of the learning model meets a threshold based on the examination of the document subset. The threshold can be pre-programmed or provided in the request to train the learning model. If the accuracy of the learning model does not meet the threshold, a request can be sent to retrain the learning model using the same set of documents or a different set of documents.

[0127] If the classification accuracy of the learning model meets a threshold, model approval flow 400 can proceed to operation 406. At operation 406, an SME can manually review the classification assigned by the learning engine to a document subset of the document set. In some embodiments, the SME can verify the classification of the learning model and the same subset of documents. Alternatively, the SME may verify the classification of a different subset of documents. For example, the document subset may be 10% of the document set.

[0128] The SME can verify whether the learning engine correctly classified each document in the document subset based on the document's metadata. Additionally, the SME can verify whether the metadata is accurate. For example, as described above, the metadata may be assigned to the document by the SME or received from a third party. Thus, the SME can identify scenarios in which the learning model correctly classified a document and the metadata contains a misclassification of the document. In response to verifying the classification of the document subset, the SME can determine whether an Acceptable Quality Limit (AQL) was achieved. The AQL may be a value associated with a desired accuracy level for the learning model. In some embodiments, the AQL may be the same as a threshold value. In other embodiments, the AQL may be different from the threshold value. If the SME determines that the AQL is not met, a request can be sent to retrain the learning model.

[0129] In operation 408, in response to the SME determining that the AQLs have been met, the SME may approve the learning model. The learning model may be considered sufficiently trained and ready to classify documents.

[0130] 5 is a block diagram illustrating a data flow for classifying documents using a trained learning engine according to an example embodiment. FIG. 5 is described with respect to FIG.

[0131] In one embodiment, the ingestion engine 115 can receive a request to classify a document. For example, a user 500 can provide an initial rating for the document. For example, the user 500 can provide an initial priority value (e.g., high, medium, or low). The request can include the initial rating and the document to be classified. The document can be a PV document.

[0132] The ingestion engine 115 can send the request to the classification application 114. The classification application 114 can store the request in the inbox 502 of a mail or messaging application.

[0133] The script 109 can periodically poll the inbox 502 to determine whether there is a new request to classify a document. In response to detecting a new request to classify a document, the script 109 can forward the document to the conversion engine 103. If the document is not in a predetermined format (e.g., PDF format), the script 109 can convert the document to the predetermined format. The script 109 can send the document to the recognition service 503. The recognition service can include a learning engine 101 and a conversion engine 103. The conversion engine 103 can convert the document into an image file containing a single image stack, as described above. The conversion engine 103 can send the image file and a request to classify the document to the learning engine 101.

[0134] The API 102 can receive requests to classify image files and documents and forward the requests to classify the image files and documents to the first, second, and third learning models 104-108. The first, second, and third learning models 104-108 can use the image files to assign first, second, and third classifications to the documents, as described above with respect to FIGS. 1-3. The learning engine 101 can generate output 504. The output 504 can include the first, second, and third classifications, priorities based on the first, second, and third classifications, and confidence scores. The confidence scores indicate the estimated accuracy of the first, second, and third classifications. The learning engine 101 can forward the output 504 to an Interactive Response Technology (IRT) service 506.

[0135] The IRT service 506 may be a service configured to interact with the learning engine 101. For example, the IRT service 506 may be configured to collect classification information from the learning engine 101, such as the output 504. The IRT service 506 prioritizes documents based on the priorities assigned in the output 504. The IRT service 506 may reside on the server 100 or on a separate server.

[0136] The IRT service 506 can forward the output 504 to a transaction service 508. The transaction service 508 can request user input to validate the prioritization of documents based on the primary, secondary, and tertiary classifications. The transaction service 508 can reside on the server 100 or on a separate server.

[0137] The transaction service 506 can forward the output 504 to an analysis workflow 510. In the analysis workflow 510, a user (e.g., a subject matter expert) can verify the accuracy of the output 504 given the document. Specifically, the user can verify the accuracy of the priority and the first, second, and third classifications. In response to verifying the accuracy of the output 504, the learning engine 101 can be adjusted or retrained. For example, if the learning engine 101 incorrectly assigned the first, second, and third classifications or priorities, the learning engine 101 can be adjusted or retrained to improve its accuracy.

[0138] 6 is a block diagram illustrating a confusion matrix for case severity detection using a learning model, according to an example embodiment. The confusion matrix 600 can show the true positives, false positives, true negatives, and false negatives of the classifications assigned by the learning model. The confusion matrix 600 can be used to determine the effectiveness and accuracy of the learning model.

[0139] The confusion matrix 600 can be associated with classifications assigned to ICSR documents. Specifically, the classifier can be non-serious. That is, the learning model can be trained to classify documents as non-serious. The learning model assumes that documents not classified as non-serious are classified as serious.

[0140] The confusion matrix 600 may include a first axis 602 indicating the actual classification of the document and a second axis 604 indicating the classification predicted by the learning model. The first and second axes 602-604 may include labels for the document based on the classification. The labels may be serious and non-serious. A learning model configured to classify documents based on severity may be configured to identify documents that exhibit non-serious adverse effects. That is, documents containing "serious" adverse effects have negative truth values, and documents containing "non-serious" adverse effects have positive truth values.

[0141] Confusion matrix 600 may include quadrants 606, 608, 610, and 612. In the example of FIG. 5, quadrant 606 may indicate that the learning model correctly predicted that 9927 documents did not contain a non-serious adverse effect (true negative). Quadrant 608 may indicate that the learning model incorrectly predicted that 2379 documents contained a non-serious adverse effect (false positive). Quadrant 610 may indicate that the learning model incorrectly predicted that 2388 documents contained a non-serious adverse effect (false negative). Quadrant 612 may indicate that the learning model correctly predicted that 9918 documents contained a non-serious adverse effect (true positive).

[0142] Confusion matrices may be generated for efficacy and SREs. For example, Table 2 shows example experimental data from a confusion matrix for severity, efficacy, and SREs.

[0143] [Table 2]

[0144] As shown in Table 2, when training the learning model to assign classifications associated with document effectiveness, the learning model can generate 48,537 true negatives, 1076 false positives, 1199 false negatives, and 7100 true positives. Furthermore, when assigning classifications associated with adverse effect SREs, the learning model can generate 5431 true negatives, 1508 false positives, 1532 false negatives, and 5407 true positives.

[0145] 7 is an exemplary single image stack according to an exemplary embodiment. As described above, a conversion engine (e.g., conversion engine 103 shown in FIGS. 1-2) can convert a document into an image file (e.g., a JPEG image file). The image file can contain multiple pages. The conversion engine 103 can composite the pages of the image files to generate a single image stack 700. Thus, the single stack image 700 can be a composite of all pages of the image files.

[0146] For example, the transformation engine can use an image manipulation application such as ImageMagick, developed by ImageMagick Studio LLC (https: / / imagemagick.org / ). To generate a single image stack, the transformation engine can provide one or more of the following to the image manipulation application: a desired image resolution in dots per inch (e.g., 3000), a target image color space (e.g., gray), a desired width and height of the image files, and a desired background for the single stack image (e.g., white, light gray, transparent, slate gray1, dark gray).

[0147] Additionally, the transformation engine can provide instructions to the image manipulation application to overlay each page of the image file on top of one another by multiplying the content of each page of the image with a destination (e.g., the first page of the image file) and replacing the destination with the resulting product. For example, if an image file contains three pages, the content of page two can be multiplied with page one, and the resulting product of the content of page one and page two can replace page one. The product of the content of page one and page two can become the new first page of the image file. The content of the new first page can then be multiplied with the content of page three, and the result of the product of the content of page three with the content of the new first page can replace page one. The product of the content of page three with the content of the new first page can be a single page containing the content of pages one, two, and three. The multiplication operation allows the image manipulation application to overlay the content of multiple pages onto a single page. The exclusion, plus, add, and subtract operations can be used additionally or alternatively to overlay the content of multiple pages onto a single page.

[0148] The conversion engine may also provide instructions to merge pages of an image file using the darkest pixel in the stack for each page of the image file. An instruction to use the darkest pixel in the stack for each page of the image file may be "min". An instruction to use the lightest pixel in the stack for each page of the image file may be "max". An instruction to generate an average of the pages of the image file may be "mean". The instruction "min" may produce a clear, readable result.

[0149] In one embodiment, the transformation engine can instruct the image manipulation application to create a new image file by merging pages of the image files. That is, the transformation engine can instruct the image manipulation application to generate a composite of the pages of the image files and store the composite as a new image file. For example, a single image stack 700 can be stored as a new image file.

[0150] FIG. 8 is a vector space distribution according to an exemplary embodiment. The vector space distribution 800 can include positive and negative classes processed by a fully connected layer of a CNN algorithm. The CNN algorithm can classify documents as negative or positive. For example, if the CNN algorithm classifies a document as a valid document, the CNN algorithm has classified the document as positive. Alternatively, if the CNN algorithm classifies the document as invalid, the CNN algorithm has classified the document as negative. The vector space distribution 800 shows the accuracy of the CNN algorithm delineated between a positive class 802 and a negative class 804. Each dot can represent a document in the positive class and a document in the negative class 802-804.

[0151] 9 includes example receiver operating characteristic (ROC) graphs generated based on classifications assigned to documents by first, second, and third learning models, according to some embodiments. For example, ROC graphs 900, 902, and 904 can be used to determine whether the first, second, or third learning models are properly trained to classify documents.

[0152] As described above, the first, second, and third learning models can generate first, second, and third classifications of the document. As an example, the document may be an ICSR document. Further, the first classification may indicate the effectiveness of the ICSR document, the second classification may indicate the severity of the ICSR document, and the third classification may be associated with an SRE of an adverse effect contained in the ICSR document.

[0153] 8, ROC graph 900 corresponds to a first classification (e.g., efficacy), ROC graph 902 corresponds to a second classification (e.g., severity), and ROC graph 904 corresponds to a third classification (e.g., adverse effect SRE). Thus, ROC graphs 900-904 can be used to determine the effectiveness or accuracy of each learning model.

[0154] The ROC graphs 900-904 include an x-axis representing the false positive rate and a y-axis representing the true positive rate. Additionally, the ROC graphs 900-904 include an ROC curve for each classification. The ROC curve plots the true positive rate and the false positive rate based on a classification threshold. The ROC graphs 900-904 also display an area under the curve (AUC) value based on the ROC curve. The AUC value is a measure of the ability of each learning model to distinguish between classes. The AUC value may be a value between 0 and 1. The higher the AUC value, the better the ability of each learning model to distinguish between classes. An AUC value of 1 indicates that each learning model can perfectly classify documents. However, an AUC of 0 indicates that each learning model misclassifies all documents.

[0155] The figures represent values ​​obtained by the inventors testing the models across a sample set of approximately 500,000 ICSRs. In the example of Figure 9, ROC graph 900 shows an AUC value of 0.99 for the first trained model, ROC graph 902 shows an AUC value of 0.90 for the second trained model, and ROC graph 904 shows an AUC value of 0.87 for the third trained model.

[0156] FIG. 10 illustrates a feature map generated using a convolutional layer of a CNN algorithm according to an exemplary embodiment. As described above, the convolutional layer of a CNN algorithm can extract features from an image by applying a filter to the image. Multiple different filters can be applied to an image to generate different feature maps. The filters can attempt to extract given features. For example, a filter can be configured to extract all edges from the image. Alternatively, a filter can be configured to extract a given shape from the image. Additionally, multiple different convolutional layers can be included in a CNN algorithm. Thus, a feature map can be generated for each convolutional layer. The first feature map generated in the first convolutional layer of a CNN algorithm can retain most of the information of the image. Each feature map generated by a subsequent convolutional layer can detect fewer features, so the feature map may not look like an image but instead look like an abstract representation of the image.

[0157] As shown in the embodiment of Figure 10, feature maps 1000 and 1002 include light and dark regions. The light regions indicate activated regions. Activated regions can indicate that the filter has detected a pattern (e.g., an edge or shape) that it is configured to detect. The dark regions represent areas of the image where the filter did not detect a feature.

[0158] 11A to 13B show experimental results obtained by the inventors when training a learning model using a sample set of approximately 500,000 ICRs. The CNN algorithm used in the experiment used the LeNet-5 architecture. The CNN LeNet-5 architecture can be used to identify patterns of handwritten or typed characters. The CNN LeNet-5 architecture can be optimized using an increased data source image size, a modified model optimizer, an increased dropout rate, and an activation function to process various input image data. The experiment was conducted using a Linux instance on AWS with a 32-core CPU, 240 GB of RAM, and four Tesla V100 GPUs with 32 GB of memory for model training using the Keras / Tensorflow API.

[0159] 11A shows a graph 1100 illustrating a principal component analysis (PCA) of validity classification, according to an example embodiment. PCA can be used to reduce the dimensionality of features extracted from images using a CNN algorithm. Specifically, PCA can be used to identify features that account for a threshold amount of variance. Thus, PCA can identify features that do not need to be extracted from the image.

[0160] For example, graph 1100 may show a PCA of efficacy classification. In the example of FIG. 11A , which represents experimental values ​​obtained by the inventors across a sample set of approximately 500,000 ICSRs, graph 1100 shows that 40 features out of over 120 features account for 90% of the variance in efficacy classification. In this manner, the identified 40 features can be extracted from a given image to assign a efficacy classification to the document corresponding to the given image.

[0161] FIG. 11B shows a tensorboard output illustrating experimental results regarding the distribution of valid documents compared to invalid documents using roughly the sample set of FIG. 11A. The tensorboard output may be displayed as graph 1050. Graph 1050 shows the clustering of document classifications across three-dimensional space. Graph 1150 includes multiple views of the clustering of document classifications across three-dimensional space. Each dot or element represents a document classified using the learning model. Graph 1150 shows a clear distinction between documents classified as valid and those that are not valid. This indicates the accuracy with which the learning classified each document.

[0162] 12A shows a graph 1200 illustrating PCA of severity classification, according to some embodiments. In the example of FIG. 12A , which represents experimental values ​​obtained by the inventors across a sample set of approximately 500,000 ICSRs, graph 1200 shows that 25 features out of more than 120 features account for 90% of the variance in severity classification. In this manner, the identified 25 features can be extracted from a given image to assign a severity classification to the document corresponding to the given image.

[0163] FIG. 12B shows a Tensorboard output illustrating experimental results for the distribution of serious documents compared to non-serious documents using the sample set of FIG. 12A. The Tensorboard output may be displayed in graph 1250. Graph 1250 shows the clustering of document classifications across three-dimensional space. Graph 1250 includes multiple views of the clustering of document classifications across three-dimensional space. Each dot or element represents a document classified using the learning model. Graph 1250 shows a clear distinction between non-serious documents and documents classified as serious. This indicates the accuracy with which the learning classified each document.

[0164] 13A shows a graph 1300 illustrating PCA of SRE classification, according to some embodiments. In the example of FIG. 12A , which represents experimental values ​​obtained by the inventors across approximately 500,000 ICSRs, graph 1300 shows that 25 features out of over 120 features account for 90% of the variance in SRE classification. In this manner, the identified 25 features can be extracted from a given image to assign a classification associated with the SRE to the document corresponding to the given image. This indicates the accuracy with which the learning classified each document.

[0165] FIG. 13B shows a Tensorboard output illustrating experimental results of the distribution of documents with expected effects compared to documents with unexpected effects (e.g., SREs) using the sample set of FIG. 13A. The Tensorboard output may be displayed in graph 1250. Graph 1350 shows the clustering of document classifications across three-dimensional space. Graph 1350 includes multiple views of the clustering of document classifications across three-dimensional space. Each dot or element represents a document classified using the learning model. Graph 1350 shows a clear distinction between documents classified with unexpected effects (SRE classifications) and documents classified as having expected effects. This indicates the accuracy with which the learning classified each document.

[0166] 14 is a tensorboard graph illustrating a visualization of training of a learning model, according to an example embodiment. As described above, the learning models (e.g., the first, second, and third learning models 104-108) can implement a CNN algorithm. The graph 1400 can include various functions that are executed during training of the CNN algorithm. The functions can be part of the convolutional layers, max pooling layers, and fully connected layers of the CNN algorithm. For example, functions may include functions related to feature extraction and generating feature maps (e.g., conv2d_n), max pooling (e.g., max_pooling_n), dimensionality reduction (e.g., Squeeze_n), preventing overfitting (e.g., dropout_n), applying linear activation (e.g., dense_n), flattening a multi-vector array (e.g., flatten_n), determining how far a predicted value is from the actual value (e.g., loss_n), obtaining metrics for training a learning model (e.g., metrics), and calculating metrics (e.g., count and total).

[0167] The functions conv2d_n, Squeeze_n, and max_pooling_n may be part of the convolutional and max pooling layers of a CNN algorithm. The functions flatten_n and dense_n may be part of the fully connected layer of a CNN algorithm.

[0168] 15 is a flowchart illustrating a process for preparing an image file for training a learning model, according to one embodiment. Method 1500 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to practice the disclosure provided herein. Furthermore, as will be appreciated by those skilled in the art, some steps may be performed simultaneously or in a different order than that shown in FIG. 15.

[0169] The method 1500 is described with reference to Figure 1. However, the method 1500 is not limited to that example embodiment.

[0170] At operation 1502, the server 100 receives a request to train at least one of the first, second, or third learning models 104, 106, and 108 to classify documents. The request may be sent by the client device 110. The request may include a query to obtain a set of documents for training at least one of the first, second, or third learning models 104-108 and metadata corresponding to each of the documents.

[0171] At operation 1504, the transformation engine 103 retrieves the document set and metadata for each document in the document set from the database 120 using the query included in the request. The documents may include alphanumeric text. The document set may include documents in various formats. Furthermore, the documents in the document set may include a single page or multiple pages.

[0172] At operation 1505, the conversion engine 103 determines whether the document needs to be converted to a predetermined format. If the document is received from the reporter in an image format, method 1400 can proceed directly to operation 1408. For example, if the reporter sent the ICSR as a JPEG file, no further conversion is necessary and the method can proceed to operation 1408. However, if the document is not already in an image format, in embodiments, the document needs to be converted from the received format to an image format. In some embodiments, the document is converted directly from the received format to an image format, such as JPEG format. In some other embodiments, the document is first converted to a common format, such as PDF format, before being converted to an image format. Converting first to a common format before image conversion simplifies the image conversion process by allowing image conversion algorithms to operate on a single document type (e.g., PDF).

[0173] Thus, in optional operation 1506, the conversion engine 103 converts each document in the document set that is in a format different from the predetermined format into the predetermined format. The predetermined format may be, for example, PDF format. Thus, the conversion engine 103 can convert each document in PDF format into PDF format. Those skilled in the art will understand that operation 1506 can be skipped if the documents are converted directly, rather than indirectly, to image files.

[0174] In operation 1508, the conversion engine 103 converts each document in the document set into an image file, resulting in a set of image files, each of which may be, for example, a JPEG image.

[0175] In operation 1510, the conversion engine 103 combines each page of each image file that contains multiple pages into a single image stack. The single image stack may include an image representation of each page of the image file in a single JPEG image. If the image file is single-page, the single image stack may include an image representation of a single page of the image file. Thus, each image file in the set of image files may include a single image stack.

[0176] In operation 1512, the learning engine 101 can use the set of image files to train one or more of the first, second, and third learning models 104, 106, and 108. One or more of the first, second, and third learning models 104-108 can be trained to assign first, second, and third classifications, respectively.

[0177] 16 is a flowchart illustrating a process for training a learning model according to one embodiment. Method 1600 may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to practice the disclosure provided herein. Furthermore, as will be appreciated by those skilled in the art, some steps may be performed simultaneously or in a different order than that shown in FIG. 16.

[0178] The method 1600 is described with reference to Figure 1. However, the method 1600 is not limited to that example embodiment.

[0179] At operation 1602, the server 100 receives a request to train one or more models to classify documents using a document set. For example, the server 100 receives a request to train at least one of the first, second, or third learning models 104-108 to classify documents using the document set. The document set may be converted into a set of image files. Each image file may be an image representation of a document. Further, each image file may include a single image stack. The single image stack may be a composite of each page of the image files. The learning engine 101 may instruct at least one of the first, second, or third learning models 104-108 to run a CNN algorithm using the set of image files.

[0180] In operation 1604, the learning engine 101 extracts a set of features from each image file in the set of image files. For example, at least one of the first, second, or third learning models 104-108 extracts a set of features from each image file in the set of image files. At least one of the first, second, or third learning models 104-108 can extract features using a CNN algorithm. To extract features, a filter can be applied to each image file in the set of image files. The features can be edges, shapes, objects, or other features within a given image file. The features can be extracted in a convolutional layer of the CNN algorithm.

[0181] In operation 1606, the learning engine 101 generates a feature map for each image file in the set of image files. For example, at least one of the first, second, or third learning models 104-108 generates a feature map for each image file in the set of image files. The feature map for each image file may include a set of features extracted from the image file. For example, the feature map may include activation regions. The activation regions may indicate detected features (e.g., edges, shapes, objects, etc.). The feature map may include dark regions. The dark regions may indicate the absence of features.

[0182] At operation 1608, the learning engine 101 generates a reduced feature map for each image file in the set of image files. For example, at least one of the first, second, or third learning models 104-108 generates a reduced feature map for each image file in the set of image files by reducing the dimensional features of the feature map for each image file in the set of image files. The reduced feature map may be generated by applying max pooling or average pooling to the feature map.

[0183] At operation 1610, the learning engine 101 converts the reduced feature map for each image file into a vector. For example, at least one of the first, second, or third learning models 104-108 converts the reduced feature map for each image file in the set of image files into a vector. The reduced feature map for each image file may be converted into a vector with a fully connected layer. At least one of the first, second, or third learning models 104-108 may perform the linear conversion to a vector by applying weights and biases to the vector.

[0184] At operation 1612, the learning engine 101 performs a nonlinear transformation on the vectors of each image file to identify attributes associated with each image file. For example, at least one of the first, second, or third learning models 104-108 performs a nonlinear transformation on the vectors of each image file in the set of image files to identify attributes associated with each image file in the set of image files. The nonlinear transformation can be performed in a fully connected layer of a CNN algorithm.

[0185] At operation 1614, the learning engine 101 assigns a classification to each document based on the identified attributes. For example, at least one of the first, second, or third learning models 104-108 assigns a classification to each document in the document set based on the identified attributes of each image file in the image file set. The first, second, and third learning models may assign first, second, and third classifications, respectively.

[0186] At operation 1616, the learning engine validates the classification of each of the document subsets by comparing each document's metadata with the model's classification. For example, at least one of the first, second, or third learning models 104-108 validates the classification of each of the document subsets of the document set by comparing each document's metadata with the classification of each document in the document set. The metadata may include a classification assigned by a third party or SME. A validation result may also be generated. For example, at least one of the first, second, or third learning models 104-108 may generate a validation result. The validation result may indicate whether at least one of the first, second, or third learning models 104-108 correctly classified the document subset based on the metadata. At least one of the first, second, or third learning models 104-108 may validate the document subset using backpropagation.

[0187] Based on the validation results, gradient descent may be used to optimize the filters, weights, and biases, and operations 1602 through 1616 may be re-performed a predetermined number of times. For example, at least one of the first, second, or third learning models 104 through 108 may optimize the filters, weights, and biases after each iteration of operations 1602 through 1616 using a gradient descent operation.

[0188] 17 is a flowchart illustrating a process for classifying documents using a trained learning model. Method 1700 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It should be understood that not all steps are required to practice the disclosure provided herein. Furthermore, as will be appreciated by those skilled in the art, some steps may be performed simultaneously or in a different order than that shown in FIG. 17.

[0189] The method 1700 is described with reference to Figure 1. However, the method 1700 is not limited to that example embodiment.

[0190] In operation 1702, the server 100 receives a request to classify a document using a trained learning model. For example, the server 100 receives a request to classify a pharmacovigilance (PV) document using a trained learning model. The PV documents may be individual case safety reports (ICSRs) for various medications. An ICSR is a report of an adverse event experienced by a patient undergoing a particular treatment or taking a particular medication, which may potentially be related to that treatment or medication. The request may include a query to retrieve the PV documents. The conversion engine 103 may retrieve the PV documents from the database 120. The request may be to classify the PV documents using at least one of the first, second, or third learning models 104-108. The first, second, or third learning models 104-108 may be fully trained to assign a first, second, or third classification to the document, respectively.

[0191] In operation 1704, the conversion engine 103 converts the document to an intermediate format if it is not already in the intermediate format or an image file format. For example, if the document is not already in a predetermined format, the conversion engine 103 converts the PV document to a predetermined format. The predetermined format may be a PDF format.

[0192] In operation 1706, the conversion engine 103 converts the document to an image file format if it is not already in the image file format. For example, the conversion engine 103 converts the PV document to an image file. The image file may be a JPEG image. The image file may contain an image representation of the PV document.

[0193] In operation 1708, the conversion engine 103 composites the pages of the image files to generate a single stack image. The single stack image can be a composite of the pages of the image files. The image files can contain a single stack image.

[0194] In operation 1710, the conversion engine 103 sends the image file to the learning engine 101 for classification.

[0195] At operation 1712, the learning engine 101 identifies attributes associated with the image file by performing image analysis on the image file. For example, at least one of the first, second, or third learning models 104-108 identifies attributes associated with the image file by performing image analysis on the image file. In one embodiment, the image analysis may be performed by executing operations 1602-1612 of flowchart 1600 of FIG. 16. In some embodiments, the learning engine 101 may instruct all three of the first, second, or third learning models 104-108 to identify first, second, and third attributes associated with the image file.

[0196] At operation 1714, the learning engine 101 assigns a classification to the document based on the identified attributes. For example, at least one of the first, second, or third learning models 104-108 assigns a classification to the PV document using the identified attributes. As described above, the first, second, or third learning models 104-108 can assign a first, second, or third classification, respectively. The first classification corresponds to the effectiveness of the separate PV document. The second classification corresponds to the severity of an adverse effect described in the separate PV document. And, the third classification corresponds to whether an adverse effect described in the separate PV document is serious, relevant, or expected. The first, second, and third learning models 104-108 can implement a CNN algorithm to assign the first, second, and third classification, respectively.

[0197] At operation 1716, the learning engine 101 outputs the assigned classification to a client device, such as an interface. For example, the learning engine 101 outputs the document's assigned classification to the client device 110. The output may include a confidence score indicating the estimated accuracy of a given classification. For example, the output may indicate a 90% confidence score that the document is correctly classified as the first classification, the second classification, or the third classification. In some embodiments, confidence scores may be output for all three classifications. In some embodiments, the output is a priority or percentage that takes into account the outputs (which may include confidence levels) of all three learning models.

[0198] 18 is a flowchart illustrating a process for classifying documents using a trained learning model implemented on a user device, according to an exemplary embodiment. Method 1800 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps are required to practice the disclosure provided herein. Furthermore, as will be appreciated by those skilled in the art, some steps may be performed simultaneously or in a different order than that shown in FIG. 18.

[0199] The method 1800 is described with reference to Figure 1. However, the method 1800 is not limited to that example embodiment.

[0200] At operation 1802, the user classification application 134 receives an image of a document to be classified. The user classification application 134 may also receive an uploaded document to be classified. The user classification application 134 may convert the image or uploaded document into an image file containing a single image stack, as described in method 1500 of FIG. 15 . The user classification application 134 may implement a learning engine that is fully trained to classify documents. For example, the learning engine may be similar to learning engine 101. Additionally, the user classification application 134 may be configured to classify PV documents.

[0201] At operation 1804, the user classification application 134 assigns a classification to the document using the image file. For example, the user classification application 134 may assign the classification by performing steps 1704-1716 of method 1700. For example, the document may be a medical document, and the classification may indicate the severity of the content described in the medical document. The user classification application 134 may generate output that includes the classification, a recommendation whether the user should consult a medical professional, and a confidence score. The confidence score indicates the estimated accuracy of the classification. The user classification application 134 may assign the classification without identifying or extracting PII from the document.

[0202] In operation 1806, the user application 134 may cause the output to be displayed on the user device 130. The user classification application 134 may also send the output to the server 100 to store the output in the database 120. This allows a user to receive a classification for a document without sending the document to the server 100. In this regard, the user device 130 locally assigns a classification to the document. In doing so, the user's PII is protected from potential security risks associated with transmitting PII over a network.

[0203] Various embodiments may be implemented using one or more computer systems, such as, for example, computer system 1900 shown in Figure 9. Computer system 900 may be used to implement, for example, method 1500 of Figure 15, method 1600 of Figure 16, method 1700 of Figure 17, and method 1800 of Figure 18. Furthermore, computer system 1900 may be at least a portion of server 100, client device 110, database 120, and user device 130, as shown in Figure 1. For example, computer system 1900 routes communications to various applications. Computer system 1900 may be any computer capable of performing the functions described herein.

[0204] The computer system 1900 can be any known computer capable of performing the functions described herein.

[0205] Computer system 1900 includes one or more processors (also referred to as central processing units or CPUs), such as processor 1904. Processor 1904 is connected to a communication infrastructure or bus 1906.

[0206] Each of the one or more processors 1904 may be a graphics processing unit (GPU). In one embodiment, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. GPUs may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common in computer graphics applications, images, video, etc.

[0207] The computer system 1900 also includes user input / output devices 1903 , such as a monitor, keyboard, pointing device, etc., that communicate with a communications infrastructure 1906 via a user input / output interface 1902 .

[0208] The computer system 1900 may also include a main or primary memory 1908, such as random access memory (RAM). The main memory 1908 may include one or more levels of cache. The main memory 1908 stores control logic (i.e., computer software) and / or data.

[0209] The computer system 1900 may also include one or more secondary storage devices or memories 1910. The secondary memory 1910 may include, for example, a hard disk drive 1912 and / or a removable storage device or drive 1914. The removable storage drive 1914 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0210] The removable storage drive 1914 can interface with a removable storage unit 1918. The removable storage unit 1918 includes a computer-usable or readable storage device that stores computer software (control logic) and / or data. The removable storage device 1918 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and / or any other computer data storage device. The removable storage drive 1914 reads from and writes to the removable storage unit 1918 in a well-known manner.

[0211] According to an exemplary embodiment, secondary memory 1910 may include other methods, means, or approaches for allowing computer programs and / or other instructions and / or data to be accessed by computer system 1900. Such methods, means, or approaches may include, for example, removable storage unit 1922 and interface 1920. Examples of removable storage unit 1922 and interface 1920 may include a program cartridge and cartridge interface (e.g., found in a video game device), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick, a USB port, a memory card and associated memory card slot, and / or other removable storage unit and associated interface.

[0212] Computer system 1900 may further include a communications or network interface 1924. Communications interface 1924 enables computer system 1900 to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (individually and collectively referred to by reference numeral 1928). For example, communications interface 1924 may enable computer system 1900 to communicate with remote devices 1928 over communications path 1926, which may be wired and / or wireless and may include any combination of a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from computer system 1900 over communications path 1926.

[0213] In one embodiment, a tangible, non-transitory device or article of manufacture that includes a tangible, non-transitory computer-usable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 1900, main memory 1908, secondary memory 1910, removable storage units 1918 and 1922, and tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (e.g., computer system 1900), causes such data processing devices to operate as described herein.

[0214] Based on the teachings contained herein, it will be apparent to one skilled in the art how to make and use embodiments of the present disclosure using data processing devices, computer systems, and / or computer architectures other than those shown in Figure 19. In particular, embodiments may operate with software, hardware, and / or operating system implementations other than those described herein.

[0215] It should be understood that the Detailed Description section, and not the other sections, is intended to be used to interpret the claims, which may describe one or more example embodiments, but not all, as contemplated by the inventors, and are therefore not intended to limit the scope of the disclosure or the appended claims in any way.

[0216] While this disclosure describes exemplary embodiments for exemplary fields and applications, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereto are possible and are within the scope and spirit of the present disclosure. For example, without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and / or entities shown in the figures and / or described herein. Moreover, embodiments (whether or not explicitly described herein) have significant utility for fields and applications beyond the examples described herein.

[0217] Embodiments have been described herein using functional building blocks that illustrate implementation of specific functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for convenience of description. Alternative boundaries may be defined so long as the specified functions and relationships (or the like) are appropriately performed. Also, alternative embodiments may execute functional blocks, steps, operations, methods, etc. using an order different from that described herein.

[0218] References herein to “one embodiment,” “embodiment,” “exemplary embodiment,” or similar phrases indicate that the described embodiment includes a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, if a particular feature, structure, or characteristic is described in connection with an embodiment, it would be within the knowledge of one of ordinary skill in the art to incorporate such feature, structure, or characteristic into other embodiments, whether or not explicitly mentioned or described herein. Furthermore, some embodiments may be described using the terms “coupled” and “connected,” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments may be described using the terms “connected” and / or “coupled” to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term “coupled” may mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0219] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. 1. A method for automatically classifying documents, the method comprising: obtaining, by one or more computing devices, a document set and metadata for each document in the document set; converting, by the one or more computing devices, each document in the document set into an image file to generate a set of image files; training, by the one or more computing devices, a learning engine to classify each document of the document set using the metadata and the set of image files corresponding to each document of the document set; receiving, by the one or more computing devices, a request to classify another document; converting, by the one or more computing devices, the other document into a new image file; Combining, by the one or more computing devices, pages of separate image files into a single image stack; assigning, by the one or more computing devices, first, second, and third classifications to the other document based on the single image stack using a trained learning engine; determining, by the one or more computing devices, a priority value for the other document based on one or more of the first, second, or third classifications; A method comprising:

2. 2. The method of claim 1, wherein the first classification corresponds to the effectiveness of the other document, the second classification corresponds to an indicator of the severity of content in the other document, and the third classification corresponds to one or more factors corresponding to the severity of the content.

3. training the learning engine Recursively, performing, by the one or more computing devices, image analysis on each corresponding image file of the set of image files to identify attributes associated with each image file of the set of image files; assigning, by the one or more computing devices, the first, second, and third classifications to each document of the document set based on the identified attributes of each image file of the set of image files using the learning engine; verifying, by the one or more computing devices, the classification of each document subset of the document set by comparing the metadata of each document to the classification of each document in the document set; The method of claim 1 , comprising:

4. Performing image analysis is extracting, by the one or more computing devices, a set of features from each image file of the set of image files; generating, by the one or more computing devices, a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file; generating, by the one or more computing devices, a reduced feature map for each image file of the set of image files by reducing dimensional features of the feature map for each image file of the set of image files; converting, by the one or more computing devices, the reduced feature map for each image file in the set of image files into a vector; performing, by the one or more computing devices, a non-linear transformation on the vector of each image file of the image file set to identify the attributes associated with each image file of the image file set that correspond to each image file of the image file set; The method of claim 3, comprising:

5. The method of claim 1 , wherein the learning engine implements a supervised learning algorithm.

6. identifying, by the one or more computing devices, document types of documents in the document set that are different from a predetermined document type; converting, by the one or more computing devices, the document into the predetermined document type; The method of claim 1 further comprising:

7. Memory and a processor coupled to the memory, the processor comprising: obtaining metadata for a document set and for each document in the document set; converting each document of the document set into an image file to generate a set of image files; training a learning engine to classify each document of the document set using the metadata and the set of image files corresponding to each document of the document set; receives a request to classify another document, Converting the other document into a new image file; Combine pages from separate image files into a single image stack, using a trained learning engine to assign first, second, and third classifications to the other document based on the single image stack; determining a priority value for the other document based on one or more of the first, second, or third classifications; a processor configured to: A system comprising:

8. 8. The system of claim 7, wherein the first classification corresponds to the effectiveness of the other document, the second classification corresponds to an indicator of the severity of content in the other document, and the third classification corresponds to one or more factors corresponding to the severity of the content.

9. training the learning engine Recursively performing image analysis on each corresponding image file of the set of image files to identify attributes associated with each image file of the set of image files; using the learning engine to assign the first, second, and third classifications to each document in the document set based on the identified attributes of each image file in the set of image files; verifying the classification of each document subset of the document set by comparing the metadata of each document to the classification of each document of the document set; The system of claim 7 , comprising:

10. Performing image analysis is extracting a feature set from each image file of the set of image files; generating a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file; reducing dimensional features of the feature map for each image file in the set of image files to generate a reduced feature map for each image file in the set of image files; converting the reduced feature map for each image file in the set of image files into a vector; performing a non-linear transformation on the vectors of each image file of the image file set to identify the attributes associated with each image file of the image file set that correspond to each image file of the image file set; The system of claim 9 , comprising:

11. The system of claim 7 , wherein the learning engine implements a supervised learning algorithm.

12. The processor: identifying document types of documents in the document set that are different from a predetermined document type; converting the document to the predetermined document type; The system of claim 7 further configured to:

13. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors of a device, causing: obtaining metadata for a document set and for each document in the document set; converting each document of the document set into an image file to generate a set of image files; training a learning engine to classify each document of the document set using the metadata and the set of image files corresponding to each document of the document set; receiving a request to classify another document; converting the other document into a new image file; Combining pages from separate image files into a single image stack; assigning first, second, and third classifications to the other document based on the single image stack using a trained learning engine; a non-transitory computer-readable medium that causes the one or more processors to perform operations including:

14. 14. The non-transitory computer-readable medium of claim 13, wherein the first classification corresponds to an effectiveness of the other document, the second classification corresponds to an indicator of the severity of content in the other document, and the third classification corresponds to one or more factors that correspond to the severity of the content.

15. training the learning engine Recursively, performing image analysis on each corresponding image file of the set of image files to identify attributes associated with each image file of the set of image files; using the learning engine to assign the first, second, and third classifications to each document in the document set based on the identified attributes of each image file in the set of image files; verifying the classification of each document subset of the document set by comparing the metadata of each document to the classification of each document of the document set; 14. The non-transitory computer-readable medium of claim 13, comprising:

16. Performing image analysis is extracting a feature set from each image file of the set of image files; generating a feature map for each image file of the set of image files, the feature map for each image file including the set of features extracted from the image file; reducing dimensional features of the feature map for each image file in the set of image files to generate a reduced feature map for each image file in the set of image files; converting the reduced feature map for each image file in the set of image files into a vector; performing a non-linear transformation on the vectors of each image file of the image file set to identify the attributes associated with each image file of the image file set that correspond to each image file of the image file set; 16. The non-transitory computer-readable medium of claim 15, comprising:

17. The processor: identifying document types of documents in the document set that are different from a predetermined document type; converting the document to the predetermined document type; 16. The non-transitory computer-readable medium of claim 15, further configured to:

18. Memory and a processor coupled to the memory, the processor comprising: obtaining metadata for a document set and for each document in the document set; converting each document of the document set into an image file to generate a set of image files; training a learning engine to classify each document of the document set using the metadata and the set of image files corresponding to each document of the document set; receives a request to classify another document, Combining the pages of the separate documents into a single image stack; assigning a classification to the other document based on the single image stack using a trained learning engine; Displaying the classification; a processor configured to: A device comprising:

19. training the learning engine Recursively, performing image analysis on each corresponding image file of the set of image files to identify attributes associated with each image file of the set of image files; using the learning engine to assign the first, second, and third classifications to each document in the document set based on identified attributes of each image file in the set of image files; verifying the classification of each document subset of the document set by comparing the metadata of each document to the classification of each document of the document set; 20. The device of claim 18, comprising:

20. The processor: extracting a feature set from each image file of the set of image files; generating a feature map for each image file of the set of image files, the feature map including the set of features extracted from the image files; generating a reduced feature map for each image file of the set of image files by reducing a dimensional feature of the feature map for each image file of the set of image files; converting the reduced feature map for each image file in the set of image files into a vector; performing a non-linear transformation on the vectors of each image file of the image file set to identify attributes associated with each image file of the image file set that correspond to each image file of the image file set; 20. The device of claim 18 configured to:

21. 20. The device of claim 18, further comprising a camera coupled to the processor, the processor further configured to receive an image of the other document captured by the camera.

22. The device of claim 21 , wherein combining the pages of the separate documents into a single image stack comprises combining the pages of the images of the separate documents.

23. The device of claim 18 , wherein the classification indicates a severity of content of the other document.

24. The device of claim 18 , wherein the other document includes personally identifiable information (PII).

25. The device of claim 18 , wherein the processor is further configured to determine a suggestion to the user corresponding to the other document based on the classification.

26. 20. The device of claim 18, wherein the processor is configured to obtain the other document from an Internet of Things (IoT) device communicatively coupled to the device.

27. The device of claim 18 , wherein the learning engine resides on the device.

Images