Framework for document layout and information extraction

Abstract

Systems, apparatus, devices, methods, and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for extracting data from a file are provided herein. Embodiments described herein provide a framework for merging outputs of various models into an output file ingestible by a database or knowledge base, the outputs including information extracted from the file and its location information and an annotated region of interest.

Description

Background Art

[0001] Information extraction is an important aspect of creating a searchable knowledge base or database. Furthermore, information extraction and knowledge base creation is the ability to understand the data in a file and extract information from it. Information can be extracted from files such as documents, images, charts, graphs, etc. Files can be in various formats and have various layouts. Therefore, it may be difficult to accurately extract data files. Furthermore, it may be challenging to mine files at scale for information that can be processed automatically. Furthermore, conventional systems are unable to extract data from files in a manner similar to how humans read files. Summary of the invention

[0002] Provided herein are systems, devices, apparatus, methods, and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for extracting information from a file.

[0003] The given embodiment includes a method for extracting information. The method includes: receiving a file in a first format including information and a plurality of regions of interest (ROIs); and converting the file into an image. The method further includes generating a first output using a first model, the first output including a first set of information extracted from the image and a first set of coordinates of the first set of information in the image. The method includes generating a second output using a second model, the second output including a second set of coordinates in the image for each of the plurality of ROIs. The method includes generating a third output using a third model, the third output including a second set of information extracted from the image and a third set of coordinates of the second set of information in the image. The method further includes merging the first output and the third output to generate information and a plurality of coordinates included in the file. The plurality of coordinates include coordinates for the information in the image. The method includes generating an output file in a second format including a plurality of segments. Each of the plurality of segments corresponds to one of the plurality of ROIs, and each of the plurality of segments is included in the output file based on the coordinates of the ROI in the second set of coordinates corresponding to the corresponding segment. The method further includes populating each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on coordinates corresponding to the portion of the information and coordinates of the corresponding section. The second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

[0004] In some embodiments, the information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

[0005] In some embodiments, the third set of coordinates forms a bounding box around each ROI in the plurality of ROIs.

[0006] In some embodiments, the first output is generated using optical character recognition (OCR).

[0007] In some embodiments, the third output is generated using a neural network.

[0008] In some embodiments, the output files are used by one or more machine learning models to generate a knowledge base.

[0009] In some embodiments, the information is selectable in the output file.

[0010] In some embodiments, the method further comprises causing the output file and the image to be displayed.

[0011] In some embodiments, wherein the information includes words and / or images. Combining the first output with the third output to generate the information may include retaining the image included in the first output. The method may further include: identifying one or more words in the first output that share the same coordinates as one or more words in the third output; and determining a level of similarity between the one or more words in the first output and the one or more words in the third output.

[0012] The method may further include assigning a first priority value to the first output and assigning a second priority value to the third output. The method may further include including the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold. The method may further include excluding the one or more words from the first output in the plurality of words based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold. The method may further include identifying a first data type for the one or more words in the first output and a second data type for the one or more words in the second output. The method may further include including the one or more words from the first output in the plurality of words based on the first data type for the one or more words in the first output. The method may further include excluding the one or more words from the third output in the plurality of words based on the second data type for the one or more words in the third output.

[0013] Another embodiment is a system for extracting information. The system includes a memory and a processor coupled to the memory, the memory including instructions. The processor is configured to execute the instructions, and the instructions, when executed, cause the processor to: receive a file in a first format including information and a plurality of regions of interest (ROIs); and convert the file into an image. The instructions, when executed, cause the processor to: generate a first output using a first model, the first output including a first set of information extracted from the image and a first set of coordinates of the first set of information in the image. The instructions, when executed, further cause the processor to: generate a second output using a second model, the second output including a second set of coordinates in the image for each of the plurality of ROIs. The instructions, when executed, further cause the processor to: generate a third output using a third model, the third output including a second set of information extracted from the image and a third set of coordinates of the second set of information in the image. The instructions, when executed, further cause the processor to: merge the first output and the third output to generate information and a plurality of coordinates included in the file. The plurality of coordinates include coordinates for the information in the image. The instructions, when executed, further cause the processor to: generate an output file in a second format including a plurality of sections using the second output. Each of the plurality of sections corresponds to one of the plurality of ROIs, and each of the plurality of sections is included in the output file based on the coordinates of the ROI in the second coordinate set corresponding to the corresponding section. The instructions, when executed, further cause the processor to: populate each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on the coordinates corresponding to the portion of the information and the coordinates of the corresponding section. The second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

[0014] In some embodiments, the information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

[0015] In some embodiments, the third set of coordinates forms a bounding box around each ROI in the plurality of ROIs.

[0016] In some embodiments, the first output is generated using optical character recognition (OCR).

[0017] In some embodiments, the third output is generated using a neural network.

[0018] In some embodiments, the output files are used by one or more machine learning models to generate a knowledge base.

[0019] In some embodiments, the information is selectable in the output file.

[0020] In some embodiments, the instructions, when executed, further cause the processor to cause the output file and the image to be displayed.

[0021] In some embodiments, wherein the information includes words and / or images. Combining the first output with the third output to generate the information may include retaining the image included in the first output. The instructions, when executed, may further cause the processor to: identify one or more words in the first output that share the same coordinates as one or more words in the third output; and determine a level of similarity between the one or more words in the first output and the one or more words in the third output.

[0022] In some embodiments, the instructions, when executed, may further cause the processor to: assign a first priority value to the first output and assign a second priority value to the third output. The instructions, when executed, may further cause the processor to: include the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold. The instructions, when executed, may further cause the processor to: exclude the one or more words from the first output in the plurality of words based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold. The instructions, when executed, may further cause the processor to: identify a first data type for the one or more words in the first output and a second data type for the one or more words in the second output. The instructions, when executed, may further cause the processor to: include the one or more words from the first output in the plurality of words based on the first data type for the one or more words in the first output. The instructions, when executed, may further cause the processor to: exclude the one or more words from the third output in the plurality of words based on the second data type for the one or more words in the third output.

[0023] Another embodiment provided includes a non-transitory machine-readable medium having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations, the operations including. The operations include: receiving a file in a first format including information and a plurality of regions of interest (ROIs); and converting the file into an image. The operations further include generating a first output using a first model, the first output including a first set of information extracted from the image and a first set of coordinates of the first set of information in the image. The operations further include generating a second output using a second model, the second output including a second set of coordinates in the image for each of the plurality of ROIs. The operations further include generating a third output using a third model, the third output including a second set of information extracted from the image and a third set of coordinates of the second set of information in the image. The operations further include merging the first output and the third output to generate information and a plurality of coordinates included in the file. The plurality of coordinates include coordinates for the information in the image. The operations further include generating an output file in a second format including a plurality of sections using the second output. Each of the plurality of sections corresponds to one of the plurality of ROIs, and each of the plurality of sections is included in the output file based on the coordinates of the ROI in the second coordinate set corresponding to the corresponding section. The operation further includes populating each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on the coordinates corresponding to the portion of the information and the coordinates of the corresponding section. The second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

[0024] In some embodiments, the information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

[0025] In some embodiments, the third set of coordinates forms a bounding box around each ROI in the plurality of ROIs.

[0026] In some embodiments, the first output is generated using optical character recognition (OCR).

[0027] In some embodiments, the third output is generated using a neural network.

[0028] In some embodiments, the output files are used by one or more machine learning models to generate a knowledge base.

[0029] In some embodiments, the information is selectable in the output file.

[0030] In some embodiments, the operations further include causing the output file to be displayed.

[0031] In some embodiments, the information includes words and / or images. Combining the first output with the third output to generate the information includes retaining the image included in the first output. The operations may further include: identifying one or more words in the first output that share the same coordinates as one or more words in the third output; and determining a level of similarity between the one or more words in the first output and the one or more words in the third output.

[0032] In some embodiments, the operation may further include assigning a first priority value to the first output and assigning a second priority value to the third output. The operation may further include including the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold. The operation may further include excluding the one or more words from the first output in the plurality of words based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold. The operation may further include identifying a first data type for the one or more words in the first output and a second data type for the one or more words in the second output. The operation may further include including the one or more words from the first output in the plurality of words based on the first data type for the one or more words in the first output. The operation may further include excluding the one or more words from the third output in the plurality of words based on the second data type for the one or more words in the third output. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and to enable those skilled in the relevant art to make and use the present disclosure.

[0034] Figure 1 is a block diagram of a system for extracting information from a file, according to some embodiments.

[0035] Figure 2 is a block diagram of data flow in a system for extracting information from a file, according to some embodiments.

[0036] Figure 3An example image with multiple sections is shown according to some embodiments.

[0037] Figure 4 A bounding box surrounding a first segment of an image is shown according to some embodiments.

[0038] Figure 5 Output files according to some embodiments are shown.

[0039] Figure 6 Output files displayed on a graphical user interface (GUI) according to some embodiments are shown.

[0040] Figure 7 is a flow chart illustrating a process for extracting information from a file according to some embodiments.

[0041] Figure 8 is a block diagram of example components of a device according to an embodiment.

[0042] The drawing in which an element first appears is generally indicated by the leftmost digit(s) in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements. DETAILED DESCRIPTION

[0043] Provided herein are systems, devices, apparatus, methods, and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for extracting information from a file.

[0044] As described above, accurately extracting information from files can be an error-prone and time-consuming process. For example, building a database or knowledge base may require extracting and understanding information from files that may be hundreds or even thousands of pages long. These files may contain text, images, charts, graphics, etc. In addition, these files may be in different formats and in a variety of different layouts. In this regard, conventional systems are unable to extract data so that the data can be accurately and efficiently ingested seamlessly by databases and knowledge bases.

[0045] The embodiments described herein address these challenges by using an image-based process for merging the outputs of various models into output files that can be ingested by a database or knowledge base, wherein the output includes information extracted from the file and its position information and annotated regions of interest. This ensures that the output file includes information that is accurately mapped back to the original file. In addition, the information in the output file is searchable when it is presented on a GUI or stored in a data storage device. The information in the output file can also be used to extract knowledge from the information using a machine learning model, which can then be used to fill a database or ontology. In this way, the embodiments described herein provide an automated way to accurately extract information from a file so that it can be efficiently accessed.

[0046] In the given embodiment, the server receives a file of a first format including information and a plurality of regions of interest (ROIs). The server converts the file into an image. In addition, the server generates a first output using a first model, the first output including a first information set extracted from the image and a first coordinate set of the first information set in the image. The server generates a second output using a second model, the second output including a second coordinate set for each ROI in the image for a plurality of ROIs. In addition, the server generates a third output using a third model, the third output including a third coordinate set of the second information set extracted in the image and the second information set in the image. The server further merges the first output and the third output to generate information and a plurality of coordinates included in the file. The plurality of coordinates include coordinates for information in the image. The server generates an output file of a second format including a plurality of sections using the second output. Each section in the plurality of sections corresponds to an ROI in the plurality of ROIs, and each section in the plurality of sections is included in the output file based on the coordinates of the ROI in the second coordinate set corresponding to the corresponding section. In addition, the server fills each section in the plurality of sections in the output file with a portion of information, the portion being determined to correspond to the corresponding section based on the coordinates corresponding to the portion of information and the coordinates of the corresponding section. The second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

[0047] Embodiments described herein generate output files including information (e.g., text, images, charts, graphics) that are accurately extracted from files and segmented based on regions of interest from which the information is extracted. Embodiments described herein improve the accuracy of information extracted from files by organizing the information in logical segments extracted from files without existing organizational metadata. In addition, information is extracted in its expected order. In addition, additional information such as headers, footers, page numbers, copyrights, tables of contents, etc. can be excluded from the output files.

[0048] The output file can be displayed to the user. Additionally, the output file can be ingested by a database so that the information extracted from the file and its location in the file can be efficiently searched. Furthermore, other machine learning models can use the information in the output file to identify knowledge when generating a knowledge base.

[0049] Figure 1It is a block diagram of a system for extracting data from a file according to some embodiments. The system may include a server 100, a client device 110, and a data storage device 120. The equipment of the system may be connected via a network. For example, the equipment of the system may be connected via a wired connection, a wireless connection, or a combination of a wired and wireless connection. In an example embodiment, one or more parts of the network may be a self-organizing network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a metropolitan area network (MAN), a part of the Internet, a part of a public switched telephone network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, a network of any other type, or a combination of two or more such networks. Alternatively, the server 100, the client device 110, and the data storage device 120 may be located on a single physical machine or a virtual machine.

[0050] In some embodiments, the server 100 and the data storage device 120 may reside in a cloud computing environment. In other embodiments, the server 100 may reside in a cloud computing environment, while the data storage device 120 resides outside the cloud computing environment. Furthermore, in other embodiments, the server 100 may reside outside the cloud computing environment, while the data storage device 120 resides in the cloud computing environment.

[0051] The data storage device 120 may be one or more databases configured to store structured and / or unstructured data. The data storage device 120 may store files from which information is to be extracted, output files, training files for training models to extract page layouts, and the like.

[0052] In some embodiments, a user may use a client device 110 to transmit or upload files for information extraction to the server 100. Alternatively, the client device 110 may transmit a request to the server to extract information from one or more files stored in the database 120.

[0053] The server 100 may receive the file from the client device 110. Alternatively, the server 100 may retrieve one or more files from the database 120 as indicated in the request transmitted by the client device 110.

[0054] The server 100 may include an extraction engine 102. The extraction engine 102 may convert a file into an image. The extraction engine 102 may implement one or more models to extract information from a given file. In addition, the extraction engine 102 may combine the outputs from one or more models to generate an output file that includes the extracted information organized in logical sections extracted from the file. The output file may be stored in a database 120 so that the information in the output file may be searched. Figure 2 Describes the extraction and generation of output files in more detail.

[0055] Figure 2 is a block diagram of data flow in a system for extracting information from a file, according to some embodiments. Figure 2 Will refer to Figure 1 Give a description.

[0056] In some embodiments, the server 100 may receive a file (e.g., document) 200 and a request to extract information from the file 200. Alternatively, the server 100 may retrieve the file 200 from the database 120 in response to receiving a request to extract information from the file 200 from the client device 110. The file 200 may include information including text (e.g., words), images, charts, graphics, etc.

[0057] The extraction engine 102 may determine whether the file is in a universal format. The universal format may be, for example, but not limited to, a portable document format (PDF). If the file is not in a universal format, the extraction engine 102 may convert the file into a universal (e.g., PDF) format, resulting in the file 202. As a non-limiting example, the extraction engine 102 may use C# to convert a non-PDF file into a PDF format. Although the remainder of this disclosure refers to PDF as a universal format, those skilled in the art will appreciate that other universal formats may be used instead.

[0058] In addition, the extraction engine 102 can render / convert each page of the PDF file 202 into an image, resulting in the image 204. As a non-limiting example, the extraction engine 102 can implement PyMuPDF to render / convert each page of the PDF file 202 into an image. The extraction engine 102 also maintains a list of each image page in the PDF file 202. This allows an unlimited number of pages to be analyzed in the PDF file 202.

[0059] The extraction engine 202 may implement various models to extract information from the image 204. For example, the extraction engine 202 may implement a text extraction optical character recognition (OCR) model 206, a page layout analysis model 208, and a read PDF image model 210. The extraction engine 202 may run additional or fewer models to extract information from the image 204. The extraction engine 202 may transmit the image 204 to the text extraction OCR model 206, the page layout analysis model 208, and the read PDF image model 210 simultaneously or in parallel to extract information from the image 204.

[0060] The text extraction OCR model 206 can perform an OCR process on the image 204 to extract text from the image 204 and each word within the corresponding position of the text in the image 204. The text extraction OCR model 206 can extract the text and its corresponding position in the image 204. For example, the text extraction OCR model 206 can generate a bounding box around each word in the image 204. The text extraction OCR model 206 can identify the coordinates of the bounding box. As a non-limiting example, the text extraction OCR model 206 can implement one or more of AMAZON TEXTRACT, TESERACT, GOOGLE VISION, MICROSOFT AZURE READ API, etc. The text extraction OCR model 206 can generate a first output. The first output includes information (e.g., text) detected in the image 204 and a first set of coordinates identifying the position of the information detected in the image 204.

[0061] The page layout analysis model 208 may identify a ROI in the image 204. The ROI may be a section in the image 204, such as a title, an abstract, a chapter, a paragraph, a table, an image, etc. The page layout analysis model 208 may implement a neural network to identify the ROI in the image 204. As a non-limiting example, the neural network may be a convolutional neural network (CNN), such as fast R-CNN, faster R-CNN, Inception, etc.

[0062] The neural network can be trained to recognize ROIs in images with different page layouts. For example, the CNN can be trained in two stages: a forward stage and a backward stage. The forward stage can include one or more convolutional layers, one or more pooling layers, and a fully connected layer. To train the CNN, the extraction engine 102 can instruct the CNN to recognize the ROI of the input image of the training file retrieved from the database 120.

[0063] The convolution layer can apply filters to the input image to generate a feature map. Specifically, in the convolution layer, the CNN can perform feature extraction on the input image. The feature can include parts of the input image. For example, the feature can be different edges or shapes of the input image. CNN can extract different types of features to generate different types of feature maps. For example, the CNN can apply digital arrays (e.g., kernels) across different parts of the input image. The kernel can also be referred to as a filter. As indicated above, different types of filters can be applied to the input image to generate different feature maps. For example, the filter used to identify the shape in the input image can be different from the filter used for edge detection. Therefore, compared with edge detection, different kernels can be applied to identify the shape in the input image. Each kernel can include different digital arrays. The values ​​of the filter or kernel can be randomly assigned and optimized over time (e.g., using a gradient descent algorithm). The kernel can be applied as a sliding window across different parts of the input image. The kernel can be summed with a given part of the input image to generate an output value. The output value can be included in the feature map. The feature map can include output values ​​from different kernels applied to each part of the input image. The generated feature map can be a two-dimensional array.

[0064] The pooling layer can generate a reduced feature map. Specifically, in the pooling layer, the CNN can reduce the dimensionality of each feature map generated in the convolution layer. The CNN can extract parts of a given feature map and discard the rest. The pooled image retains important features while discarding unimportant features. For example, a feature map may include an activation region and a non-activation region. The activation region may include a detected feature, while the non-activation region may indicate that the part of the image does not include the feature. Pooling can remove the non-activation region. In this way, the size of the image is reduced. The CNN can use maximum or average pooling in the pooling layer to perform these operations. Maximum pooling retains the higher values ​​of parts of the feature map while discarding the remaining values. Average pooling retains the average value of different parts of the feature map. Therefore, the CNN can generate a reduced feature map for each feature map generated in the convolution layer.

[0065] The CNN may include additional convolutional layers. In the additional convolutional layers, the CNN may generate additional feature maps based on the reduced feature maps generated in the pooling layers. In addition, the CNN may include additional pooling layers. In the additional pooling layers, the CNN may generate further reduced feature maps based on the feature maps generated in the additional convolutional layers.

[0066] The convolutional layer may also apply a rectified linear unit (ReLU) function to the input image. The ReLU function is applied to the input image to remove linearity from the input image. For example, the ReLU function may remove all black elements from the input image and retain only gray and white. This results in more abrupt color changes in the input image, which removes linearity from the input image.

[0067] Convolutional layers and pooling layers can be used for feature learning. Feature learning allows the CNN to identify desired features in the input image and therefore accurately classify the input image. Therefore, by optimizing the convolutional layers and pooling layers, the CNN can apply the correct filters to the input image to extract the necessary features required to classify the input image.

[0068] The fully connected layer can then classify the features of the image using the weights and biases to generate an output. The CNN can identify the ROI in the input image. The fully connected layer can generate a bounding box around the identified ROI. In addition, the fully connected layer can identify the coordinates of the identified ROI in the input image.

[0069] Specifically, in the fully connected layer, the CNN can flatten the reduced feature map generated in the pooling layer into a one-dimensional array (or vector). The fully connected layer is a neural network. CNN can perform linear transformations on one-dimensional arrays. CNN can perform linear transformations by applying weights and biases to one-dimensional arrays. Initially, the weights and biases are randomly initialized and can be optimized over time. CNN can perform nonlinear transformations such as activation layer functions (e.g., softmax or sigmoid) to identify the ROI of the input image.

[0070] In the backward phase, the CNN can use backpropagation to determine whether the CNN can correctly identify the ROI. Backpropagation includes optimizing input parameters so that the CNN can more accurately classify the document. The input parameters may include kernel values, weights, biases, etc. Gradient descent can be used to optimize the parameters. Specifically, gradient descent can be used to optimize the CNN's recognition of the ROI in the input image.

[0071] Gradient descent is an iterative process for optimizing CNN. Gradient descent can update the parameters of CNN and enable CNN to identify each ROI in the input image based on the updated parameters and verify the identification of the ROI in the input image.

[0072] To this end, in some aspects, the CNN can use backpropagation to verify its classification of the input image. Specifically, in some aspects, a subject matter expert (e.g., a pathologist) can determine whether the model 104 correctly identifies the ROI in the input image. The subject matter expert can provide their feedback on the accuracy of the identified ROI. Alternatively or in addition to this feedback, the CNN can use metadata associated with each image to verify the identified ROI. For example, the metadata can be the ROI correctly identified for the input image.

[0073] The CNN can compare the ROI identified by the CNN in the input image with the information included in the metadata or feedback provided by the subject matter expert. Based on the comparison, the CNN can use a gradient descent algorithm to update the values ​​of filters, weights and biases and re-execute the forward stage on the input image.

[0074] The extraction engine 102 can instruct the CNN to identify the ROI of each image in the training file, including different versions of each image. The CNN iteratively optimizes its parameters, identifies the ROI of each image in the training file via the forward phase, and verifies the ROI via the backward phase until it reaches the desired accuracy threshold.

[0075] Once the CNN reaches a desired accuracy threshold, the CNN may be considered fully trained. Once the CNN is fully trained, the page layout analysis model 208 may use the CNN to identify ROIs in the image 204. The CNN may also identify and extract identifiers for the ROIs in the image 204. The page layout analysis model 208 may use the CNN to generate a second output including a second set of coordinates for each ROI in the image 204 and an identifier of the ROI. The CNN may be continuously refined after identifying the ROI for a given image based on feedback received from the user.

[0076] In some embodiments, the page layout analysis model 208 may use a faster R-CNN applied with image segmentation, which detects and locates ROIs in the image 204. The ROI may be a table, figure, paragraph, section title, etc.

[0077] The Read PDF Image model 210 may detect and extract text in the image 204. As a non-limiting example, the Read PDF Image model 210 uses PyMuPDF to extract text from the image 204. The Read PDF Image model 210 may output a bounding box for each detected word in the image 204. If the Read PDF Image model 210 encounters information in the image 204 that is not a word, the Read PDF Image model 210 extracts pixel information. The Read PDF Image model 210 generates a third output that includes a second set of words detected in the image 204 and a third set of coordinates for each corresponding word in a third set of words in the image 204.

[0078] The extraction engine 102 may also implement a merge analysis model 212. The merge analysis model 212 may be configured to merge the first output, the second output, and the third output to generate an output file. Specifically, the first output from the text extraction OCR model 206 and the third output from the read PDF image model 210 may include text from the image 204 and the corresponding coordinates of the text in the image 204. There may be some or significant overlap in the first output and the third output. That is, the text extraction OCR model 206 and the read PDF image model 210 extract information from the same location in the image 204. The merge analysis model 212 may need to determine which information to retain and which information to discard.

[0079] To this end, the merge analysis model 212 can assign priorities to the information in the first output and the third output. For example, if the merge analysis model 212 determines that the text extraction OCR model 206 and the read PDF image model 210 extract text from the same location in the image 204, the merge analysis model 212 can determine the similarity level of the text at the given location in the image 204 in the first output and the text at the given location in the image 204 in the third output. For example, the merge analysis model 212 can determine how similar the words in the text at the given location in the image 204 in the first output and the text at the given location in the image 204 in the third output are.

[0080] The merge analysis model 212 may also determine the data type of the extracted information in the first output and the second output. For example, if the text does not include any embedded fonts or images (e.g., the text is original text) and the similarity level is greater than a predetermined threshold, the merge analysis model 212 may retain the text from the third output (e.g., from the read pdf image model 210) and discard the text from the first output (e.g., from the text extraction OCR model 206). If the text includes embedded fonts or images, the merge analysis model 212 may retain the text from the first output (e.g., from the text extraction OCR model 206) and discard the text from the third output (e.g., from the read pdf image model 210). For any extracted information that may be an image, chart, graphic, etc., the merge analysis model 212 may retain the information from the first output (e.g., from the text extraction OCR model 206) and discard the information from the third output (e.g., from the read pdf image model 210). By merging the first and second outputs, the merge analysis model 212 may generate the complete content of the image 204 in the exact position or desired order.

[0081] The merge analysis model 212 can generate an output file including segments. Each segment corresponds to an identified ROI from the second output. The segments can be organized in the output file based on the coordinates of the ROI and the identifier of the ROI as indicated in the second output. For example, if the image 204 includes chapter 1 and chapter 2, the second output may include the coordinates of chapter 1 and chapter 2 in the image 204. The merge analysis model 212 can use the coordinates of chapter 1 and chapter 2 in the second output to generate the first segment and the second segment in the output file.

[0082] The merge analysis model 212 may populate the sections in the output file with information extracted from the image 204. That is, the merge analysis model 212 may populate each section of the output file with information determined to correspond to the corresponding section using information from the first output and the second output as merged by the merge analysis model 212, coordinates of the information in the image 204 as indicated in the first output and the second output, and coordinates of the ROI as indicated in the third output.

[0083] In some embodiments, the first output or the third output may include text from an image, chart, or table in images 204. The second output may include coordinates for the image, chart, or table. Merge analysis model 212 may populate the section of the output file corresponding to the image, chart, or table in the second output with text extracted from the image, chart, or table in the first output or the third output.

[0084] The output file may include information extracted from the image 204, organized in logical sections extracted from the image 204. The output file may also include coordinates of the information and logical sections as mapped to the image 204. The output file may also have symbols indicating spaces, paragraphs, tabs, indents, etc. included in the information. In some embodiments, the text extraction OCR model 206, the page layout analysis model 208, and the read PDF image model 210 may be configured to exclude headers, footers, tables of contents, and other information that may not be relevant to the user.

[0085] The output file can be a JavaScript Object Notation (JSON) file. In this way, the output file can be human readable. In addition, other machine learning models can use the output file for further processing. Specifically, the machine learning model can identify / extract knowledge from the output file and use the output file to build a knowledge base. The machine learning model can be on one or more of the named entity recognition (NER) model 214, the embedding model 216, the overview generation model 218, the classification model 220, etc.

[0086] In some embodiments, the output file can be stored in the data storage device 120. The information of the output file can be queried. Since the output file includes the coordinates of the information in the image 204, the user can also query the output file to locate the information in the image 204.

[0087] Figure 3 An example image with multiple segments is shown according to some embodiments. Figure 1 and Figure 2 To describe Figure 3 .

[0088] Image 300 may be a file that extraction engine 102 converts into image 300. Image 300 may include text divided into two sections: first section 304 and second section 306. First section 304 may be a section including text about dogs. Second section 306 may be a section including text about cats.

[0089] The extraction engine 102 may instruct the text extraction OCR model 206, the page layout analysis model 208, and the read PDF image model 210 to extract information from the image 300. In this scenario, the information may be text about dogs and cats.

[0090] Each of the text extraction OCR model 206 and the read PDF image model 210 can detect and extract text from the image 300. In addition, each of the text extraction OCR model 206 and the read PDF image model 210 can generate a bounding box around each detected word. For example, each of the text extraction OCR model 206 and the read PDF image model 210 can generate a bounding box 302 around the word "jump". In addition, each of the text extraction OCR model 206 and the read PDF image model 210 can determine the coordinates of the bounding box 302 in the image 300. The coordinates can be in the following format: (x, y), where x and y are pixels in the image 300. Each of the text extraction OCR model 206 and the read PDF image model 210 can extract the word "jump" and the coordinates of the bounding box 302. Each of the text extraction OCR model 206 and the read PDF image model 210 can include the extracted words and coordinates in the first output and the third output, respectively.

[0091] Figure 4 A bounding box around a first segment 304 of an image 300 is shown according to some embodiments. Figures 1 to 3 To describe Figure 4 .

[0092] As described above, the extraction engine 102 can instruct the page layout analysis model 208 to identify the ROI of the image 300. The ROI can be the first segment 304 and the second segment 306. The page layout analysis model 208 can use a neural network such as a CNN to identify the segments first segment 304 and second segment 306 in the image 300.

[0093] For example, the page layout analysis model 208 can generate a bounding box 400 around the first segment 304 in response to identifying the first segment 304. The page layout analysis model 208 can determine coordinates of the bounding box 400 in the image 300. Additionally, the page layout analysis model 208 can determine an identifier (e.g., "I. Dog") for the first segment 304. The page layout analysis model 208 can include the coordinates of the bounding box 400 and the identifier of the first segment 304 in the second output.

[0094] Figure 5 Output files according to some embodiments are shown. Figures 1 to 4 To describe Figure 5 .

[0095] The merge analysis model 212 can generate the output file 500 using the first output of the text extraction OCR model 206, the second output generated by the page layout analysis model 208, and the third output generated by the read PDF image model 210. Specifically, the merge analysis module 212 can merge the extracted text and the coordinates of the bounding box of the extracted text in the first output and the second output to generate the information contained in the image 300.

[0096] In addition, merge analysis model 212 may use the third output to generate segment 502 and segment 504 in output file 500. Segment 502 may correspond to first segment 304 in image 300. Segment 504 may correspond to second segment 306 in image 300. Merge analysis model 212 may generate segment 502 based on coordinates of a bounding box (e.g., bounding box 400) of first segment 304 and an identifier of first segment 304, and generate segment 504 based on coordinates of a bounding box of second segment 306 and an identifier of second segment 306.

[0097] The merge analysis model 212 can identify the text corresponding to the segment 502 from the merged text based on the coordinates of the text and the coordinates of the first segment 304. The merge analysis model 212 can fill the corresponding text into the segment 502. Similarly, the merge analysis model 212 can identify the text corresponding to the segment 504 from the merged text based on the coordinates of the text and the coordinates of the second segment 306. The merge analysis model 212 can fill the corresponding text into the segment 504.

[0098] Figure 6An output file displayed on a graphical user interface (GUI) according to some embodiments is shown. Figures 1 to 5 To describe Figure 6 .

[0099] The user may display the output file 500 on the GUI 600 using the client device 110. The GUI 600 may include a portion 602 and a portion 604. The portion 602 may include the image 300, and the portion 604 may include the output file 500.

[0100] The text in output file 500 may be selectable and searchable. In addition, output file 500 may include the coordinates of the text in image 300. In this way, the text in output file 500 may be mapped to the relevant portion of image 300. If the user selects the word "jump" 606 in output file 500, the corresponding word 608 may be highlighted in image 300 based on the coordinates of the word "jump" 608 in image 300. In this way, output file 500 may be used to effectively search image 300.

[0101] In another example, image 300 may be displayed on GUI 600, and output file 500 may not be displayed on GUI 600. However, when a user attempts to search for image 300, a search may be performed for output file 500, and relevant elements of image 300 may be highlighted based on the search results.

[0102] Figure 7 is a flow chart illustrating a process for extracting information from a file according to some embodiments. Method 700 may be performed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing device), or a combination thereof. It should be understood that not all steps may be required to perform the disclosure provided herein. In addition, as will be appreciated by one of ordinary skill in the art, some steps may be performed simultaneously or in different steps. Figure 7 Execute in the order shown.

[0103] Will refer to Figure 1 and Figure 2 700 is described below. However, method 700 is not limited to this example embodiment.

[0104] In operation 702, the server 100 receives a file 200 including information and a plurality of regions of interest (ROIs). The information may be text, images, graphics, charts, etc. The ROI may be a section of the file, such as a title, summary, chapter, section, etc. The file may be received from the client device 110 via a request to extract information from the file.

[0105] In operation 704, the extraction engine 102 converts the file into the image 204. The extraction engine 102 may determine whether the file 200 is a PDF. If not, the extraction engine 102 may convert the file 200 into a PDF file 202. In addition, the extraction engine 102 may convert the PDF file 202 into the image 204.

[0106] In operation 706, the extraction engine 102 generates a first output, which includes a first set of information extracted from the image 204 and a first set of coordinates of the first set of information in the image 204. The extraction engine 102 can instruct the text extraction OCR model 206 to generate the first output. The text extraction OCR model 206 can use OCR to extract the first set of information and the first set of coordinates of the first set of information. For example, the first set of coordinates can be a bounding box around each word, image, chart, graph, etc. The first set of coordinates indicates the location of the information in the image 204.

[0107] In operation 708, the extraction engine 102 generates a second output including a second set of coordinates for each ROI of the image 204. The extraction engine 102 can instruct the page layout analysis model 208 to detect and identify the set of coordinates for each ROI. The page layout analysis model 208 can implement a neural network to detect and identify the ROIs in the image 204. The page layout analysis model 208 can create a bounding box around each ROI in the image 204. The second set of coordinates can correspond to the bounding box in the image 204. The second set of coordinates indicates the location of the ROI in the image 204.

[0108] In operation 710, the extraction engine 102 generates a third output, which includes the second information set extracted from the image and a third coordinate set of the second information set in the image. The extraction engine 102 can instruct the PDF image model 210 to generate the third output. For example, the third coordinate set can be a bounding box around each word, image, chart, graph, etc.

[0109] In operation 712, the extraction engine 102 merges the first output and the third output to generate information and a plurality of coordinates included in the file. The plurality of coordinates includes coordinates for the information in the image. The extraction engine 102 may instruct the merge analysis model 212 to merge the first output and the third output to generate the information.

[0110] In operation 714, the extraction engine 102 generates an output file including the segments. The merge analysis model 212 can generate an output file. Each segment corresponds to an ROI of the image 204. Each segment is included in the output file based on the coordinates of the ROI corresponding to the respective segment.

[0111] In operation 716, the extraction engine 102 populates each section in the output file with a portion of the information that is determined to correspond to the corresponding section based on the coordinates corresponding to the portion of the information and the coordinates of the corresponding section. The merge analysis model 212 can populate the sections of the output file. The output file can include information extracted from the file and logically organized like the file. In addition, the output file can be searchable.

[0112] Various embodiments may be implemented, for example, using one or more computer systems such as Figure 8 The computer system 800 shown in FIG. 8 is implemented. The computer system 800 can be used to implement, for example Figure 7 In addition, the computer system 800 may be at least a part of the server 100, the client device 110, and the data storage device 120, such as Figure 1 As shown. For example, computer system 800 routes communications to various applications. Computer system 800 can be any computer capable of performing the functions described herein.

[0113] Computer system 800 may be any known computer capable of performing the functions described herein.

[0114] Computer system 800 includes one or more processors (also referred to as central processing units or CPUs), such as processor 804. Processor 804 is connected to a communication infrastructure or bus 806.

[0115] One or more processors 804 may each be a graphics processing unit (GPU). In an embodiment, a GPU is a processor that is a dedicated electronic circuit designed to process math-intensive applications. A GPU may have a parallel structure that is efficient for processing large blocks of data in parallel, such as math-intensive data common in computer graphics applications, images, videos, etc.

[0116] The computer system 800 also includes one or more user input / output devices 803 , such as a monitor, keyboard, pointing device, etc., that communicate with the communication infrastructure 806 through one or more user input / output interfaces 802 .

[0117] The computer system 800 also includes a main memory or primary storage 808, such as a random access memory (RAM). The main memory 808 may include one or more levels of cache. The main memory 808 stores control logic (ie, computer software) and / or data.

[0118] The computer system 800 may also include one or more secondary storage devices or memories 810. The secondary storage 810 may include, for example, a hard disk drive 812 and / or a removable storage device or drive 814. The removable storage drive 814 may be a floppy disk drive, a tape drive, an optical drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0119] The removable storage drive 814 may interact with a removable storage unit 818. The removable storage unit 818 includes a computer usable or readable storage device having computer software (control logic) and / or data stored thereon. The removable storage unit 818 may be a floppy disk, a magnetic tape, an optical disk, a DVD, an optical storage disk, and / or any other computer data storage device. The removable storage drive 814 reads and / or writes from the removable storage unit 818 in a known manner.

[0120] According to an exemplary embodiment, the secondary memory 810 may include other devices, tools, or other methods for allowing computer programs and / or other instructions and / or data to be accessed by the computer system 800. Such devices, tools, or other methods may include, for example, a removable storage unit 822 and an interface 820. Examples of removable storage unit 822 and interface 820 may include a program cartridge and cartridge interface (such as found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated sockets, a memory stick and USB port, a memory card, and associated memory card slot, and / or any other removable storage unit and associated interface.

[0121] The computer system 800 may further include a communication or network interface 824. The communication interface 824 enables the computer system 800 to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (collectively referred to individually by the reference numeral 828). For example, the communication interface 824 may allow the computer system 800 to communicate with a remote device 828 via a communication path 826, which may be wired and / or wireless and may include any combination of a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from the computer system 800 via the communication path 826.

[0122] In an embodiment, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer-usable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 800, primary memory 808, secondary memory 810, and removable storage units 818 and 822, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system 800), causes such data processing devices to operate as described herein.

[0123] Based on the teachings contained in this disclosure, it is obvious to a person skilled in the relevant art how to use the Figure 8 It will be clear that data processing devices, computer systems, and / or computer architectures other than those shown can be used to make and use the embodiments of the present disclosure. In particular, the embodiments can operate with software, hardware, and / or operating system implementations other than those described herein.

[0124] It should be understood that the detailed description section, rather than any other section, is intended to be used to interpret the claims. The other sections may set forth one or more but not all exemplary embodiments contemplated by one or more inventors, and therefore, are not intended to limit the present disclosure or the appended claims in any way.

[0125] Although the present disclosure describes exemplary fields and exemplary embodiments of applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereof are possible and are within the scope and spirit of the present disclosure. For example, without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware and / or entities shown in the drawings and / or described in this article. In addition, the embodiments (whether or not explicitly described in this article) have important utility to fields and applications outside the examples described herein.

[0126] Embodiments have been described herein by means of functional building blocks that demonstrate the implementation of specified functions and their relationships. For ease of description, the boundaries of these functional building blocks have been arbitrarily defined herein. As long as the specified functions and their relationships (or their equivalents) are properly performed, alternative boundaries may be defined. In addition, alternative embodiments may use a different order than that described herein to perform function blocks, steps, operations, methods, etc.

[0127] References to "one embodiment", "embodiment", "exemplary embodiment" or similar phrases herein indicate that the described embodiments may include specific features, structures or characteristics, but each embodiment does not necessarily include specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not explicitly mentioned or described herein, it is within the knowledge of a technician in the relevant field to combine such features, structures or characteristics in other embodiments. In addition, the expressions "coupled" and "connected" and their derivatives may be used to describe some embodiments. These terms are not necessarily synonymous with each other. For example, the terms "connected" and / or "coupled" may be used to describe some embodiments to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also refer to two or more elements that are not in direct contact with each other, but still cooperate or interact with each other.

[0128] The breadth and scope of the present disclosure should not be limited to any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A method for extracting information, the method comprising: receiving, by a processor, a file including information and a plurality of regions of interest (ROIs), wherein the file is in a first format; converting the file into an image by the processor; generating, by the processor, a first output using a first model, the first output comprising a first set of information extracted from the image and a first set of coordinates of the first set of information in the image; generating, by the processor, a second output using a second model, the second output comprising a second set of coordinates in the image for each of the plurality of ROIs; generating, by the processor, a third output using a third model, the third output comprising a second set of information extracted from the image and a third set of coordinates of the second set of information in the image; merging, by the processor, the first output and the third output to generate information and a plurality of coordinates included in the file, wherein the plurality of coordinates includes coordinates for the information in the image; generating, by the processor, an output file in a second format including a plurality of segments using the second output, wherein each segment of the plurality of segments corresponds to one of the plurality of ROIs, and each segment of the plurality of segments is included in the output file based on coordinates of the ROI in the second coordinate set corresponding to the corresponding segment; and populating, by the processor, each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on coordinates corresponding to the portion of the information and coordinates of the corresponding section, Wherein, the second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

2. The method according to claim 1, wherein: The information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

3. The method according to claim 1, wherein: The third set of coordinates forms a bounding box around each ROI of the plurality of ROIs.

4. The method according to claim 1, wherein: The first output is generated using optical character recognition (OCR).

5. The method according to claim 1, wherein: The third output is generated using a neural network.

6. The method according to claim 1, wherein: One or more machine learning models use the output files to generate a knowledge base.

7. The method according to claim 1, wherein: The information is optional in the output file.

8. The method of claim 1, further comprising causing, by the processor, the output file and the image to be displayed.

9. The method according to claim 1, wherein: The information includes words and / or images.

10. The method according to claim 9, wherein: Combining the first output with the third output to generate the information includes retaining an image included in the first output.

11. The method according to claim 9, further comprising: identifying, by the processor, one or more words in the first output that share common coordinates with one or more words in the third output; as well as A similarity level between the one or more words in the first output and the one or more words in the third output is determined by the processor.

12. The method according to claim 11, further comprising: assigning, by the processor, a first priority value to the first output and a second priority value to the third output; including, by the processor, the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold; as well as The one or more words from the first output are excluded from the plurality of words by the processor based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold.

13. The method according to claim 11, further comprising: identifying, by the processor, a first data type for the one or more words in the first output and a second data type for the one or more words in the second output; including, by the processor, the one or more words from the first output in the plurality of words based on a first data type of the one or more words in the first output; as well as The one or more words from the third output are excluded from the plurality of words by the processor based on the second data type of the one or more words in the third output.

14. A system for extracting information, the system comprising: a memory including instructions; and a processor coupled to the memory, wherein the processor is configured to execute the instructions, and the instructions, when executed, cause the processor to: receiving a file in a first format including information and a plurality of regions of interest (ROIs); converting the file into an image; generating a first output using a first model, the first output comprising a first set of information extracted from the image and a first set of coordinates of the first set of information in the image; generating a second output using a second model, the second output comprising a second set of coordinates in the image for each ROI in the plurality of ROIs; generating a third output using a third model, the third output comprising a second set of information extracted from the image and a third set of coordinates of the second set of information in the image; combining the first output and the third output to generate information and a plurality of coordinates included in the file, wherein the plurality of coordinates includes coordinates for the information in the image; generating an output file in a second format including a plurality of segments using the second output, wherein each segment in the plurality of segments corresponds to one ROI in the plurality of ROIs, and each segment in the plurality of segments is included in the output file based on coordinates of the ROI in the second coordinate set corresponding to the corresponding segment; and populating each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on coordinates corresponding to the portion of the information and coordinates of the corresponding section, Wherein, the second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

15. The system of claim 14, wherein: The information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

16. The system of claim 14, wherein: The third set of coordinates forms a bounding box around each ROI of the plurality of ROIs.

17. The system of claim 14, wherein: The first output is generated using optical character recognition (OCR).

18. The system of claim 14, wherein: The third output is generated using a neural network.

19. The system of claim 14, wherein: The output files are used by one or more machine learning models to generate a knowledge base.

20. The system of claim 14, wherein: The information is optional in the output file.

21. The system of claim 14, wherein: The instructions, when executed, further cause the processor to cause the output file and the image to be displayed.

22. The system of claim 14, wherein: The information includes words and / or images.

23. The system of claim 22, wherein: Combining the first output with the third output to generate the information includes retaining an image included in the first output.

24. The system of claim 22, wherein: The instructions, when executed, further cause the processor to: identifying one or more words in the first output that share the same coordinates as one or more words in the third output; and A level of similarity between the one or more words in the first output and the one or more words in the third output is determined.

25. The system of claim 24, wherein: The instructions, when executed, further cause the processor to: assigning a first priority value to the first output and assigning a second priority value to the third output; including the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold; and The one or more words from the first output are excluded from the plurality of words based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold.

26. The system of claim 24, wherein: The instructions, when executed, further cause the processor to: identifying a first data type for the one or more words in the first output and a second data type for the one or more words in the second output; including the one or more words from the first output in the plurality of words based on a first data type of the one or more words in the first output; and The one or more words from the third output are excluded from the plurality of words based on the second data type of the one or more words in the third output.

27. A non-transitory machine-readable medium having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: receiving a file in a first format including information and a plurality of regions of interest (ROIs); converting the file into an image; generating a first output using a first model, the first output comprising a first set of information extracted from the image and a first set of coordinates of the first set of information in the image; generating a second output using a second model, the second output comprising a second set of coordinates in the image for each ROI in the plurality of ROIs; generating a third output using a third model, the third output comprising a second set of information extracted from the image and a third set of coordinates of the second set of information in the image; combining the first output and the third output to generate information and a plurality of coordinates included in the file, wherein the plurality of coordinates includes coordinates for the information in the image; generating an output file in a second format including a plurality of segments using the second output, wherein each segment in the plurality of segments corresponds to one ROI in the plurality of ROIs, and each segment in the plurality of segments is included in the output file based on coordinates of the ROI in the second coordinate set corresponding to the corresponding segment; and populating each of the plurality of sections in the output file with a portion of the information, the portion being determined to correspond to the corresponding section based on coordinates corresponding to the portion of the information and coordinates of the corresponding section, Wherein, the second format allows the information in the output file to be searchable when it is presented on a graphical user interface (GUI) or stored in a data storage device.

28. The non-transitory machine-readable medium of claim 27, wherein: The information includes one or more words, and generating the first output or the third output includes generating a bounding box around each of the one or more words in the image.

29. The non-transitory machine-readable medium of claim 27, wherein: The third set of coordinates forms a bounding box around each ROI of the plurality of ROIs.

30. The non-transitory machine-readable medium of claim 27, wherein: The first output is generated using optical character recognition (OCR).

31. The non-transitory machine-readable medium of claim 27, wherein: The third output is generated using a neural network.

32. The non-transitory machine-readable medium of claim 27, wherein: The output files are used by one or more machine learning models to generate a knowledge base.

33. The non-transitory machine-readable medium of claim 27, wherein: The information is optional in the output file.

34. The non-transitory machine-readable medium of claim 27, wherein: The operations further include causing the output file and the image to be displayed.

35. The non-transitory machine-readable medium of claim 27, wherein: The information includes words and / or images.

36. The non-transitory machine-readable medium of claim 35, wherein: Combining the first output with the third output to generate the information includes retaining an image included in the first output.

37. The non-transitory machine-readable medium of claim 35, wherein: The operations further include: identifying one or more words in the first output that share the same coordinates as one or more words in the third output; and A level of similarity between the one or more words in the first output and the one or more words in the third output is determined.

38. The non-transitory machine-readable medium of claim 37, wherein: The operations further include: assigning a first priority value to the first output and assigning a second priority value to the third output; including the one or more words from the third output in the plurality of words based on the second priority value of the third output and based on the similarity level being greater than a predetermined threshold; and The one or more words from the first output are excluded from the plurality of words based on the first priority value of the first output and based on the similarity level being greater than the predetermined threshold.

39. The non-transitory machine-readable medium of claim 37, wherein: The operations further include: identifying a first data type for the one or more words in the first output and a second data type for the one or more words in the second output; including the one or more words from the first output in the plurality of words based on a first data type of the one or more words in the first output; and The one or more words from the third output are excluded from the plurality of words based on the second data type of the one or more words in the third output.

Images