Optical Character Recognition Error Correction Based on Vision and Textual Content

Abstract

A system for facilitating OCR error correction based on visual and textual content is described herein. According to an embodiment, the system may include using a neural network to divide the scanned content into blocks and labeling the blocks as textual content and image content. The system may further include converting the textual content into word feature vectors. The system may further include converting the image content into image feature vectors.

Description

Technical Field

[0001] The present disclosure relates generally to optical character recognition (OCR), and more particularly to OCR correctness based on visual and textual content. Background Art

[0002] Optical character recognition (OCR) converts typed images, handwritten or printed text into characters, such as ASCII. For typed images and printed text, OCR technology has improved significantly. However, for handwritten images, the technology still struggles. In current OCR systems, text information is extracted, while the visual content of the scanned document (such as photos, graphics, charts, etc.) is ignored. When processing documents with photos and text, OCR systems only analyze character symbols and then convert them into characters. However, today's OCR systems do not utilize all available information to accurately predict and adjust the proposed words. The output of handwritten OCR is still very poor (word-level accuracy is about 30%), while document-level accuracy is about 70%. This gap can be closed by including image analysis and deep learning.

[0003] The above background related to OCR technology is intended only to provide a contextual overview of some current issues and is not intended to be exhaustive. Additional contextual information may become more apparent by reading the following detailed description. Summary of the invention

[0004] A simplified overview of the specification is provided below to provide a basic understanding of certain aspects of the specification. This overview is not an exhaustive overview of the specification. It is neither intended to identify key or important elements of the specification nor to describe any scope of a particular embodiment of the specification or any scope of the claims. Its sole purpose is to present some concepts of the specification in a simplified form as a preface to a more detailed description presented later.

[0005] According to one embodiment, the system may include a memory storing a computer executable component and a processor executing the computer executable component stored in the memory. The computer executable component may include an image classification component that uses a neural network to divide the scanned content into blocks and mark the blocks as first content and second content. The computer executable component may further include a first content analysis component that converts the first content into a word feature vector. The computer executable component may further include a second content analysis component that converts the second content into an image feature vector.

[0006] In some aspects of the embodiment, the first content analysis component also includes a word construction component, which uses an optical character recognition component to recognize characters of the first content to construct words from the first content. In some aspects of the embodiment, the first content analysis component also includes a vector generation component, which converts the word into a word feature vector. In some aspects of the embodiment, the second content analysis component uses a convolutional neural network to convert the second content into an image feature vector. In some aspects of the embodiment, the computer executable component also includes an error correction component, which uses a multi-directional long short-term memory module to identify errors and correct errors. In some aspects of the embodiment, the computer executable component also includes a word generation component, which uses a word feature vector and an image feature vector to generate a final word. In some aspects of the embodiment, the word generation component uses a bidirectional long short-term memory module to learn the association between the first content and the second content to generate the final word.

[0007] According to another embodiment, a computer-implemented method may include: using a system including a processor, dividing the scanned content into blocks using a neural network, and marking the blocks as text content and image content. The computer-implemented method may further include converting the text content into a word feature vector by the system. The computer-implemented method may further include converting the image content into an image feature vector by the system.

[0008] According to another embodiment, a computer readable storage device includes instructions that, in response to being executed, cause a system including a processor to perform operations. The operations may include using a neural network to divide scanned content into blocks and labeling the blocks as text content and image content. The operations may further include converting the text content into word feature vectors. The operations may further include converting the image content into image feature vectors.

[0009] The following description and accompanying drawings set forth certain illustrative aspects of the specification. However, these aspects are only indicative of a few of the various ways in which the principles of the present specification may be employed. Other advantages and novel features of the present specification will become apparent from the following detailed description of the specification when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Non-limiting and non-exhaustive embodiments of the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0011] Figure 1 An example of a non-limiting OCR system for facilitating OCR error correction based on visual and textual content in accordance with one or more embodiments described herein is shown.

[0012] Figure 2A A document (eg, Figure 1 An example of non-limiting content of 102).

[0013] Figure 2B An example of non-limiting image processing according to one or more embodiments described herein is shown.

[0014] Figure 2C An example of non-limiting text processing according to one or more embodiments described herein is shown.

[0015] Figure 3 An example of a non-limiting bidirectional LSTM according to one or more embodiments described herein is shown.

[0016] Figure 4 A block diagram of an exemplary non-limiting system for facilitating OCR error correction based on visual and textual content in accordance with one or more embodiments described herein is shown.

[0017] Figure 5 A block diagram of an exemplary non-limiting system for facilitating OCR error correction based on visual and textual content in accordance with one or more embodiments described herein is shown.

[0018] Figure 6 A block diagram of an exemplary non-limiting system for facilitating OCR error correction based on visual and textual content in accordance with one or more embodiments described herein is shown.

[0019] Figure 7 A diagram depicts an exemplary non-limiting computer-implemented method for facilitating use of an OCR system in accordance with one or more embodiments described herein.

[0020] Figure 8 A diagram depicts an exemplary non-limiting computer-implemented method for facilitating use of an OCR system in accordance with one or more embodiments described herein.

[0021] Fig. 9 A diagram depicts an exemplary non-limiting computer-implemented method for facilitating use of an OCR system in accordance with one or more embodiments described herein.

[0022] Fig.10 A block diagram is shown of an exemplary non-limiting operating environment that can facilitate one or more embodiments described herein. DETAILED DESCRIPTION

[0023] The following detailed description is illustrative only and is not intended to limit the embodiments and / or the application or uses of the embodiments. In addition, it is not intended to be bound by any explicit or implicit information presented in the previous summary or detailed description.

[0024] One or more embodiments are now described with reference to the accompanying drawings, wherein like reference numerals are used to refer to like elements throughout the text. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a more thorough understanding of one or more embodiments. However, in various cases, it is apparent that one or more embodiments can be practiced without these specific details.

[0025] In several embodiments, a system and method are described herein that divides scanned content into chunks and labels the chunks as textual content and / or visual content. The textual content is processed by an OCR system to construct words by character-by-character recognition. The proposed words are converted into vectors using a word pair vector algorithm (e.g., word2vec). In parallel, the visual content is converted into feature vectors. The extracted vector list is further evaluated by a bidirectional algorithm to correct the proposed words using image, context, and historical knowledge (e.g., neural networks).

[0026] Figure 1 An example of a non-limiting optical character recognition (OCR) system 100 for facilitating OCR error correction based on visual and textual content according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of the same elements employed in the corresponding embodiments are omitted. According to some embodiments, a system and method are provided that divide content into blocks and mark the content as textual content or visual content. The OCR system then processes the textual content to recognize character by character to construct words. The proposed words are then converted into vectors using standard techniques (e.g., "word2vec"). In parallel, the visual content is converted into feature vectors. The extracted vector list is processed by a multi-directional long-term memory network (LSTM) to identify errors and repair such errors based on other or previously related vectors.

[0027] In some embodiments, a scanner 104 may be used to scan an original document 102. The scanned document is provided to a classification component 106 to localize, classify, or divide the scanned content into chunks using a neural network, and label the chunks as text content 108 and visual content 110. The text content 108 is segmented into words using one or more word segmentation techniques 112 using a long-term memory network LSTM-A 114. The LSTM architecture is designed for sequence prediction using spatial inputs such as text and images.

[0028] LSTM-A 114 utilizes a convolutional neural network (CNN) to perform character recognition 116 and then perform word recognition 118. After completing word recognition, for additional verification and confirmation, LSTM-B 122 (e.g., multi-directional LSTM-B) is used to perform context analysis 124 to eliminate errors based on analysis of previous vectors and new vectors. In some embodiments, context analysis 124 is a method of comparing words, paragraphs, images in the context of other words, paragraphs, or images. In some embodiments, the words generated by the CNN are analyzed for images contained in scanned documents. In some embodiments, the type of document is considered for correcting or verifying the accuracy of the proposed words. If the word is associated with other images, the generated word is more likely to be correct. For example, if the word generated from the handwritten portion of the scanned document is "ignitor", and the image contained in the document is fire, the generated word is more likely to be "ignitor".

[0029] In some embodiments, the visual content 110 is processed using a CNN 120 and additionally utilizing an LSTM B 122 to perform a contextual analysis 124. In some embodiments, the words are analyzed in conjunction with one or more images to increase the accuracy of the contextual analysis 124. In some embodiments, sentences and / or paragraphs are generated by collecting the words at 126. In one embodiment, when generating sentences or paragraphs, the words may be modified based on a relationship to a new word or new image processed as part of the document.

[0030] Figure 2A A document 202 (eg, Figure 1 For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted. According to some embodiments, the document 202 may include a text portion 204, a first image portion 206, and a second image portion 208. The text portion may include a handwritten portion and / or a typed portion.

[0031] Figure 2BAn example of non-limiting image processing according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements employed in the corresponding embodiments are omitted. According to some embodiments, the data block at 206 represents an image (e.g., image A 206). Image A 206 (e.g., an image portion of document 202) is provided to CNN 224 to be converted into an image feature vector. In some embodiments, the CNN analyzes the image and generates an image feature vector that includes various content-based information extracted from image A 206. For example, but not limited to, the content can be an image of an aircraft, an image of an engine or a portion of an engine, an image of a fire, an image of a person or a crowd, and / or an image of a pilot.

[0032] Figure 2C An example of non-limiting text processing according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements employed in corresponding embodiments are omitted. According to some embodiments, if the data block is textual, the block is provided to LSTM A 114. In some embodiments, a text block including a handwritten word "ignitor" is provided to CNN 234. CNN 234 converts the handwritten word into a proposed word "ignitor". The proposed word is provided to vector converter 238 to convert the proposed word into a word feature vector.

[0033] Figure 3 An example of a non-limiting bidirectional LSTM according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements used in the corresponding embodiments are omitted. According to some embodiments, all vectors generated from words and images are provided to a bidirectional LSTM 340 to learn the association / correlation between words and image content. In some embodiments, a word vector 240 ("n" represents the current vector) and an image vector are provided to a bidirectional LSTM 340 to generate a final word 338. The bidirectional LSTM analyzes the current word and image vector for the previous vector to correct errors. In some embodiments, the bidirectional LSTM 340 can analyze subsequent words and / or image vectors to modify the output word. For example, the bidirectional LSTM 340 can evaluate the previous image / word to determine whether there is a relationship between the current word and the previous image / word (e.g., vector 332 and vector 330), so as to make necessary corrections. The bidirectional LSTM 340 can store words and re-evaluate when generating subsequent images / words (e.g., vector 334 and vector 336).

[0034] Figure 4A block diagram of an exemplary non-limiting system 400 for facilitating OCR error correction based on visual and textual content according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements employed in corresponding embodiments are omitted. According to some embodiments, the OCR system 402 may include a memory 404, a controller component 406 (also referred to as a "processor") that executes computer executable components stored in the memory 404, or is associated with the memory 404 and the controller component 406. The OCR system 402 may further include a system bus 408, which may connect various components, including but not limited to an image classification component 410, a first content analysis component 412, and a second content analysis component 414.

[0035] Aspects of the systems (e.g., OCR system 402, etc.), devices, or processes explained in this disclosure may constitute machine-executable components embodied in a machine, for example, machine-executable components embodied in one or more computer-readable media associated with one or more machines. Such components, when executed by one or more machines (e.g., computers, computing devices, virtual machines, etc.), may cause the machines to perform the described operations.

[0036] It should be understood that the embodiments of the subject disclosure depicted in the various figures disclosed herein are for illustration only, and therefore, the architecture of such embodiments is not limited to the systems, devices and / or components depicted therein. For example, in some embodiments, the OCR system 402 may include the operating environment 1000 and the Fig.10 The various computer-based and / or computing elements described herein. In several embodiments, the implementation may be combined with Figure 4 One or more of the systems, devices and / or components shown and described in or other figures disclosed herein may use such computer-based and / or computing-based elements.

[0037] According to several embodiments, the memory 404 may store one or more computer and / or machine readable, writable and / or executable components and / or instructions that, when executed by the controller component 406, may facilitate the performance of operations defined by the executable components and / or instructions. For example, the memory 404 may store computer and / or machine readable, writable and / or executable components and / or instructions that, when executed by the controller component 406, may facilitate the performance of various functions described herein with respect to the image classification component 410, the first content analysis component 412, and the second content analysis component 414.

[0038] In several embodiments, memory 404 may include volatile memory (e.g., random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), etc.) and / or non-volatile memory (e.g., read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), etc.) that may employ one or more memory architectures. Fig.10 Other examples of memory 404 are described. Such examples of memory 404 may be used to implement any embodiments of the subject disclosure.

[0039] According to some embodiments, the controller component 406 may include one or more types of processors and / or electronic circuits that may implement one or more computer and / or machine readable, writable and / or executable components and / or instructions that may be stored in the memory 404. For example, the controller component 406 may perform various operations that may be specified by such computer and / or machine readable, writable and / or executable components and / or instructions, including but not limited to logic, control, input / output (I / O), arithmetic, etc. In some embodiments, the controller component 406 may include one or more central processing units, multi-core processors, microprocessors, dual microprocessors, microcontrollers, systems on a chip (SOCs), array processors, vector processors, and / or another type of processor.

[0040] In some embodiments, components of the OCR system 402, such as the controller component 406, the memory 404, the image classification component 410, the first content analysis component 412, and the second content analysis component 414, can be communicatively coupled, electrically coupled, and / or operably coupled to each other via a system bus 408 to perform the functions of the OCR system 402 and / or any components coupled thereto. In several embodiments, the system bus 408 can include one or more memory buses, memory controllers, peripheral buses, external buses, local buses, and / or another type of bus that can employ a variety of bus architectures. Reference is made below to the system bus 1018 and Fig.10 Other examples of system bus 408 are described. Such examples of system bus 408 may be used to implement any of the embodiments of the subject disclosure.

[0041] In several embodiments, the OCR system 402 may include one or more computer and / or machine readable, writable and / or executable components and / or instructions that, when executed by the controller component 406, may facilitate the performance of operations defined by such components and / or instructions. Additionally, in numerous embodiments, any component associated with the OCR system 402 may include one or more computer and / or machine readable, writable and / or executable components and / or instructions that, when executed by the controller component 406, may facilitate the performance of operations defined by such components and / or instructions, as described herein with or without reference to the various figures of the subject disclosure. For example, the image classification component 410 and / or any other component associated with (e.g., communicatively coupled, electrically coupled and / or operatively coupled to, and / or used by) the OCR system 402 may include such computer and / or machine readable, writable and / or executable components and / or instructions. Thus, according to many embodiments, the OCR system 402 and / or any components associated therewith may employ a controller component 406 to execute such computer and / or machine readable, writable and / or executable components and / or instructions to facilitate performance of one or more operations described herein with reference to the OCR system 402 and / or any such components associated therewith.

[0042] In some embodiments, the OCR system 402 can facilitate performing operations related to and / or performed by components of the OCR system 402, such as the controller component 406, the memory 404, the image classification component 410, the first content analysis component 412, and the second content analysis component 414. For example, as described in detail below, the OCR system 402 can facilitate: using a neural network to divide scanned content into blocks by a system including a processor (e.g., the image classification component 410) and label the blocks as text content and image content; converting text content into word feature vectors by the system (the first content analysis component 412); and converting image content into image feature vectors by the system (the second content analysis component 414).

[0043] In some embodiments, the image classification component 410 may include, but is not limited to, one or more electronic devices and processors to process scanned documents. In some embodiments, the image classification component 410 uses a neural network to divide the scanned content into blocks and mark the blocks as first content and second content. In some embodiments, the first content includes text data, which may be standard text (e.g., typed characters) or handwritten text. In some embodiments, the second content includes image blocks (e.g., photos, charts, graphics). In some embodiments, a bounding box (e.g., organizing the scanned document into several small boxes) may be used to identify a portion of the scanned document as an image. In some embodiments, depending on the scanned document, one or more blocks may be image content (e.g., second content). In some embodiments, a neural network may be utilized to divide the scanned document into blocks including image content and / or blocks including text content. Once a block is identified, the image classification component 410 identifies the type of content, such as a paragraph of text, a photo, a picture, a graphic, tabular data, a chart, etc. A neural network with a learning algorithm may be used to identify one or more blocks. In some embodiments, an LSTM with a recurrent neural network may be utilized to identify information blocks using an attention-based approach.

[0044] In some embodiments, the first content analysis component 412 may include, but is not limited to, one or more electronic devices and processors to analyze the blocks. In some embodiments, the first content analysis component 412 converts the first content into a word feature vector. Figure 2C As shown, the first content (e.g., a block of text content) is provided to the LSTM-CNN to predict a word. The word is provided to a vector generation engine (e.g., Doc2Vec) to generate a word feature vector. In some embodiments, the first content analysis component 412 includes a word construction component (not shown), which constructs words from the first content by using an optical character recognition component to recognize characters of the first content. In some embodiments, the first content analysis component 412 includes a vector generation component (not shown), which converts the word into a word feature vector. In some aspects of the embodiment, the first content analysis component 412 also includes a word construction component, which constructs words from the first content by using an optical character recognition component to recognize characters of the first content. In some aspects of the embodiment, the first content analysis component also includes a vector generation component, which converts the word into a word feature vector.

[0045] In some embodiments, the second content analysis component 414 may include, but is not limited to, one or more electronic devices and processors to analyze the block. In some embodiments, the second content analysis component 414 converts the second content into an image feature vector. Figure 2BAs shown, the second content (e.g., image block) is provided to a convolutional neural network that converts the second content into an image feature vector. In some embodiments, the CNN can generate the image feature vector using a previous analysis of an image from a current scanned document or a previously scanned document similar to the current document. In some aspects of the embodiment, the second content analysis component 414 uses a convolutional neural network to convert the second content into an image feature vector.

[0046] Figure 5 A block diagram of an exemplary non-limiting system 500 for facilitating OCR error correction based on visual and textual content according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements employed in corresponding embodiments are omitted. According to several embodiments, the system 500 may include an OCR system 402. In some embodiments, the OCR system 402 may further include an error correction component 516.

[0047] In some aspects of the embodiments, the error correction component 516 may include one or more sensors, memory, and one or more processors to facilitate correcting errors in the proposed word. In some embodiments, the error correction component 516 facilitates the use of a multi-directional long short-term memory module to identify errors and correct errors. In some embodiments, a multi-directional / bidirectional neural network (NN) analyzes one or more previous vectors (e.g., image feature vectors or word feature vectors) to determine contextual relationships to adjust the current word. In some embodiments, the NN checks the image feature vector to verify the currently generated word. For example, if the image is a fire or an engine, and the currently generated word is "ignnite" (e.g., a misspelling), the context analysis will result in correcting the misspelled word to "ignite". In some embodiments, the NN can analyze subsequent words or images for correction. For example, if a subsequent image indicates that the word should be "ignitor", the word "ignite" can be modified. In some embodiments, the error correction component 516 can also evaluate the document type to verify the accuracy of the proposed word. For example, if the document is related to aviation, the visual content is engine ignition, and the proposed word is "ignitor", the NN can determine that the proposed word is likely to be accurate. In another example, when a proposed word is not directly relevant to the document (e.g., "ignitor" for a medical document), the NN can continue to analyze other words and images and make adjustments to the proposed word.

[0048] Figure 6A block diagram of an exemplary non-limiting system 600 for facilitating OCR error correction based on visual and textual content according to one or more embodiments described herein is shown. For the sake of brevity, repeated descriptions of similar elements employed in corresponding embodiments are omitted. According to several embodiments, the system 600 may include an OCR system 402. In some embodiments, the OCR system 402 may further include a word generation component 618.

[0049] In some aspects of the embodiment, the word generation component 618 uses the word vector and the image vector to generate the final word. After performing the various contextual analyses discussed above (e.g., analysis of previous related words, subsequent words, images, document types, etc. compared by NN), the word generation component 618 facilitates generating the final word. In some embodiments, the word is generated by analyzing visual content and textual content.

[0050] Figure 7 A diagram depicting an exemplary non-limiting computer-implemented method for facilitating use of an OCR system 402 according to one or more embodiments described herein. For the sake of brevity, repeated descriptions of similar elements employed in other embodiments described herein are omitted. In some examples, flowchart 700 can be implemented by operating environment 1000 described below. It is understood that the operations of flowchart 700 can be implemented in an order different from that depicted.

[0051] In a non-limiting example embodiment, a computing device (or system) (e.g., computer 1012) is provided that includes one or more processors and one or more memories storing executable instructions that, when executed by the one or more processors, may facilitate the performance of the operations described herein, including Figure 7 A non-limiting method is shown in the flowchart.

[0052] Operation 702 divides the scanned content into blocks using a neural network by a system including a processor, and labels the blocks as text content and image content. Operation 704 describes converting the text content into word feature vectors by the system. Operation 706 depicts determining whether the scanned data includes image data (e.g., visual content, such as an image or photo). If the scanned data includes image data, operation 708 is performed. Otherwise, monitoring continues. Operation 708 depicts converting the image content into image feature vectors by the system.

[0053] Figure 8A diagram depicting an exemplary non-limiting computer-implemented method for facilitating use of an OCR system 402 according to one or more embodiments described herein. For the sake of brevity, repeated descriptions of similar elements employed in other embodiments described herein are omitted. In some examples, flowchart 800 can be implemented by operating environment 1000 described below. It is understood that the operations of flowchart 800 can be implemented in an order different from that depicted.

[0054] In a non-limiting example embodiment, a computing device (or system) (e.g., computer 1012) is provided that includes one or more processors and one or more memories storing executable instructions that, when executed by the one or more processors, may facilitate the performance of the operations described herein, including the following: Figure 8 A non-limiting method is shown in the flowchart.

[0055] Operation 802 divides the scanned content into blocks using a neural network by a system including a processor, and marks the blocks as text content and image content. Operation 804 depicts converting the text content into word feature vectors by the system. Operation 806 depicts determining whether the scanned data includes image data (e.g., visual content, such as an image or photo). If the scanned data includes image data, operation 808 is performed. Otherwise, monitoring continues. Operation 808 depicts converting the image content into an image feature vector by the system. Operation 810 depicts generating a final word by the system using the word vector and the image vector.

[0056] Fig. 9 A diagram depicting an exemplary non-limiting computer-implemented method for facilitating use of an OCR system 402 according to one or more embodiments described herein. For the sake of brevity, repeated descriptions of similar elements employed in other embodiments described herein are omitted. In some examples, flowchart 900 can be implemented by operating environment 1000 described below. It is understood that the operations of flowchart 900 can be implemented in an order different from that depicted.

[0057] In a non-limiting example embodiment, a computing device (or system) (e.g., computer 1012) is provided that includes one or more processors and one or more memories storing executable instructions that, when executed by the one or more processors, may facilitate the performance of the operations described herein, including: Fig. 9 A non-limiting method is shown in the flowchart.

[0058] Operation 902 divides the scanned content into blocks using a neural network by a system including a processor, and marks the blocks as text content and image content. Operation 904 depicts converting the text content into word feature vectors by the system. Operation 906 depicts determining whether the scanned data includes image data (e.g., visual content, such as an image or photo). If the scanned data includes image data, operation 908 is performed. Otherwise, monitoring continues. Operation 908 depicts converting the image content into image feature vectors by the system. Operation 910 depicts identifying errors by the system using a multi-directional long short-term memory module. Operation 912 depicts correcting errors by the system using a multi-directional long short-term memory module.

[0059] Fig.10A block diagram of an exemplary non-limiting operating environment 1000 that can facilitate one or more embodiments described herein is shown. The operating environment 1000 for implementing various aspects of the present disclosure may also include a computer 1012. The computer 1012 may also include a processing unit 1014, a system memory 1016, and a system bus 1018. The system bus 1018 couples system components including, but not limited to, the system memory 1016 to the processing unit 1014. The processing unit 1014 may be any of a variety of available processors. Dual microprocessors and other multi-processor architectures may also be used as the processing unit 1014. The system bus 1018 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any available bus architecture including, but not limited to, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), FireWire (IEEE 1094), and Small Computer System Interface (SCSI). The system memory 1016 may also include volatile memory 1020 and non-volatile memory 1022. A basic input / output system (BIOS) containing the basic routines to transfer information between various elements within the computer 1012, such as during startup, is stored in the non-volatile memory 1022. By way of illustration and not limitation, nonvolatile memory 1022 may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or nonvolatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). Volatile memory 1020 may also include random access memory (RAM) used as external cache memory. By way of illustration and not limitation, RAM comes in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM.

[0060] The computer 1012 may also include removable / non-removable, volatile / nonvolatile computer storage media. Fig.10For example, disk storage 1024 is shown. Disk storage 1024 may also include, but is not limited to, devices such as a magnetic drive, a floppy drive, a tape drive, a Jaz drive, a Zip drive, an LS-100 drive, a flash memory card, or a memory stick. Disk storage 1024 may also include storage media, alone or in combination with other storage media, including, but not limited to, optical drives, such as a compact disk ROM device (CD-ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), or a digital versatile disk ROM drive (DVD-ROM). To facilitate connecting disk storage 1024 to system bus 1018, a removable or non-removable interface, such as interface 1026, is typically used. Fig.10 Also depicted is software that acts as an intermediary between a user and the basic computer resources described in the appropriate operating environment 1001. Such software may also include, for example, an operating system 1028. The operating system 1028, which may be stored on disk storage 1024, is used to control and allocate the resources of the computer 1012. System applications 1030 utilize the management of resources by the operating system 1028 through program modules 1032 and program data 1034 (e.g., stored in system memory 1016 or on disk storage 1024). It should be understood that the present disclosure can be implemented with various operating systems or combinations of operating systems. The user enters commands or information into the computer 1012 through input devices 1036. Input devices 1036 include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, webcam, etc. These and other input devices are connected to the processing unit 1014 through the system bus 1018 via interface ports 1038. Interface ports 1038 include, for example, serial ports, parallel ports, game ports, and universal serial buses (USB). Output devices 1040 use some of the same types of ports as input devices 1036. Thus, for example, a USB port can be used to provide input to computer 1012, as well as output information from computer 1012 to output devices 1040. Output adapters 1042 are provided to illustrate that there are some output devices 1040, such as monitors, speakers, and printers, and other output devices 1040 that require special adapters. By way of illustration and not limitation, output adapters 1042 include video and sound cards that provide a means of connection between output devices 1040 and system bus 1018. It should be noted that other devices and / or device systems provide both input and output capabilities, such as remote computers 1044.

[0061] The computer 1012 can operate in a networked environment using logical connections to one or more remote computers (e.g., remote computer 1044). The remote computer 1044 can be a computer, server, router, network PC, workstation, microprocessor-based device, peer device or other public network node, etc., and can generally also include many or all of the elements described relative to the computer 1012. For simplicity, only the memory storage device 1046 with the remote computer 1044 is shown. The remote computer 1044 is logically connected to the computer 1012 through a network interface 1048 and then physically connected via a communication connection 1050. The network interface 1048 covers wired and / or wireless communication networks, such as local area networks (LANs), wide area networks (WANs), cellular networks, etc. LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet, token ring, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks (such as integrated services digital networks (ISDN) and variants thereon), packet switching networks, and digital subscriber lines (DSL). Communication connection 1050 refers to the hardware / software used to connect the network interface 1048 to the system bus 1018. Although the communication connection 1050 is shown to be internal to the computer 1012 for clarity of illustration, it can also be external to the computer 1012. For exemplary purposes only, the hardware / software used to connect to the network interface 1048 can also include internal and external technologies, such as modems including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0062] Embodiments of the present invention may be systems, methods, devices and / or computer program products at any possible level of integrated technical detail. The computer program product may include a computer-readable storage medium (or medium) having thereon computer-readable program instructions for causing a processor to perform various aspects of the present invention. The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media may also include the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device, such as a punch card or a raised structure in a groove, on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be understood as being a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted through wires.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device. The computer-readable program instructions for performing the operations of various aspects of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, and process programming languages ​​such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer or completely on a remote computer or server. In the latter case, the remote computer can be coupled to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, by using the Internet of an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs) can execute computer-readable program instructions by utilizing the state information of computer-readable program instructions to customize the electronic circuits, thereby performing various aspects of the present invention.

[0064] Various aspects of the present invention are described herein with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It will be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instruction is executed via the processor of the computer or other programmable data processing device to create a device for implementing the function / action specified in the flowchart and / or block diagram frame. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can instruct a computer, a programmable data processing device and / or other devices to act in a specific manner, so that the computer-readable storage medium in which instructions are stored includes a product, and the product includes instructions for implementing various aspects of the function / action specified in the flowchart and / or block diagram frame. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus to cause a series of operational actions to be performed on the computer, other programmable device, or other apparatus, thereby producing a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other apparatus implement the functions / actions specified in the flowchart and / or block diagram blocks.

[0065] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, fragment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the box may not occur in the order indicated in the figure. For example, depending on the functions involved, two boxes shown in succession may actually be executed substantially simultaneously, or sometimes the boxes may be executed in reverse order. It should also be noted that each box of the block diagram and / or flowchart and the combination of boxes of the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of hardware and computer instructions for special purposes.

[0066] Although the subject matter has been described above in the general context of computer executable instructions of a computer program product running on one and / or multiple computers, those skilled in the art will recognize that the present disclosure may also or may be implemented in combination with other program modules. Typically, program modules include routines, programs, components, data structures, etc. that perform tasks and / or implement abstract data types. In addition, those skilled in the art will appreciate that other computer system configurations may be utilized to practice the computer-implemented methods of the present invention, including single-processor or multi-processor computer systems, small computing devices, large computers and computers, handheld computing devices (e.g., PDAs, phones), microprocessor-based or programmable consumer or industrial electronics, etc. The described aspects may also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices linked through a communication network. However, some (if not all) aspects of the present disclosure may be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in local and remote memory storage devices.

[0067] As used in this application, the terms "component", "system", "platform", "interface", etc. may refer to and / or may include computer-related entities or entities related to an operable machine having one or more specific functions. The entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and / or a computer running on a processor. As an illustration, both an application running on a server and a server may be a component. One or more components may reside in a process and / or thread of execution, and a component may be located on one computer and / or distributed between two or more computers. In another example, the corresponding component may be executed from various computer-readable media having various data structures stored thereon. The component may communicate, for example, via local and / or remote processing according to a signal having one or more data packets (e.g., data from a component interacting with a local system, another component in a distributed system, and / or other systems via a network such as the Internet via a signal). As another example, a component may be a device having a specific function provided by a mechanical part operated by an electrical or electronic circuit, which is operated by a software or firmware application executed by a processor. In this case, the processor may be internal or external to the device and may execute at least a portion of a software or firmware application. As yet another example, the component may be a device that provides a specific functionality through electronic components without mechanical parts, where the electronic components may include a processor or other means to execute software or firmware that at least partially imparts the functionality to the electronic components. In one aspect, the component may emulate the electronic component via a virtual machine within, for example, a server computing system.

[0068] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X uses A or B" is intended to mean any natural inclusive arrangement. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the above cases. In addition, the articles "a" and "an" used in the subject specification and drawings should generally be interpreted to mean "one or more" unless otherwise specified or clear from the context to be in the singular form. As used herein, the terms "example" and / or "exemplary" are used to mean used as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to these examples. In addition, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor is it meant to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art.

[0069] As used in this subject specification, the term "processor" may refer to substantially any computational processing unit or device, including but not limited to a single-core processor; a single processor with software multithreaded execution capability; a multi-core processor; a multi-core processor with software multithreaded execution capability; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. In addition, a processor may refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. In addition, the processor may utilize nanoscale architectures, such as, but not limited to, transistors, switches, and gates based on molecules and quantum dots, to optimize space usage or enhance the performance of user equipment. The processor may also be implemented as a combination of computational processing units. In this disclosure, terms such as "storage", "data storage", "database", and substantially any other information storage component related to the operation and function of the component are used to refer to a "memory component", an entity embodied in a "memory", or a component containing a memory. It should be understood that the memory and / or memory components described herein may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. As an illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). Volatile memory may include RAM, for example, RAM may be used as external cache memory. As an illustration and not limitation, RAM has many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM) and Rambus dynamic RAM (RDRAM). In addition, the memory components of the system or computer-implemented method disclosed herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0070] What has been described above only includes examples of systems, computer program products, and computer-implemented methods. Of course, for the purpose of describing the present disclosure, it is not possible to describe every possible combination of components, products, and / or computer-implemented methods, but a person of ordinary skill in the art will recognize that many further combinations and permutations of the present disclosure are possible. In addition, to the extent that the terms "including", "having", "having", etc. are used in the detailed description, claims, appendices, and drawings, such terms are intended to be included in a manner similar to the interpretation of the term "including" when used as a transitional word in the claims. Descriptions of various embodiments have been given for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to a person of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, the practical application or technical improvement of the technology found on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0071] Further aspects of the invention are provided by the subject matter of the following clauses:

[0072] 1. A system, comprising: a memory having computer executable components stored thereon; a processor, the processor executing at least the following computer executable components: an image classification component, the image classification component dividing scanned content into blocks using a neural network, and marking the blocks as first content and second content; a first content analysis component, the first content analysis component converting the first content into a word feature vector; and a second content analysis component, the second content analysis component converting the second content into an image feature vector.

[0073] 2. The system according to any preceding clause, wherein the first content analysis component includes a word construction component, the word construction component using an optical character recognition component to recognize characters of the first content to construct words from the first content.

[0074] 3. A system according to any preceding clause, wherein the first content analysis component comprises a vector generation component, the vector generation component converting the word into the word feature vector.

[0075] 4. According to the system of any of the preceding items, the second content analysis component uses a convolutional neural network to convert the second content into the image feature vector.

[0076] 5. A system according to any of the preceding clauses, wherein the computer executable component further comprises: an error correction component, the error correction component using a multi-directional long short-term memory module to identify errors and correct the errors.

[0077] 6. The system according to any preceding clause, wherein the computer executable component further comprises: a word generation component, the word generation component using the word feature vector and the image feature vector to generate a final word.

[0078] 7. A system according to any of the preceding clauses, wherein the word generation component adopts a bidirectional long short-term memory module to learn the association between the first content and the second content to generate the final word.

[0079] 8. A method comprising: using a neural network to divide scanned content into blocks and mark the blocks as text content and image content through a system including a processor; converting the text content into a word feature vector through the system; and converting the image content into an image feature vector through the system.

[0080] 9. The method according to any of the preceding clauses, further comprising: generating a final word by the system using the word feature vector and the image feature vector.

[0081] 10. The method according to any preceding clause, wherein generating the final word comprises learning an association between the text content and the image content to generate the final word.

[0082] 11. A method according to any of the preceding clauses, wherein converting the text content into the word feature vector comprises using an optical character recognition model to recognize characters of the text content to construct words from the text content.

[0083] 12. A method according to any of the preceding clauses, wherein converting the image content into the image feature vector comprises converting the image content into the image feature vector using a convolutional neural network.

[0084] 13. The method according to any of the preceding clauses further comprises: identifying errors through the system using a multi-directional long short-term memory module; and correcting the errors through the system using the multi-directional long short-term memory module.

[0085] 14. According to the method of any of the preceding items, converting the text content into the word feature vector includes using a vector generation module to convert the text content into the word feature vector.

[0086] 15. A computer-readable storage device comprising instructions that, in response to execution, cause a system comprising a processor to perform operations comprising: using a neural network to divide scanned content into blocks and labeling the blocks as text content and image content; converting the text content into a word feature vector; and converting the image content into an image feature vector.

[0087] 16. A computer readable storage device according to any preceding clause, wherein the operation further comprises: using the word feature vector and the image feature vector to generate a final word.

[0088] 17. A computer-readable storage device according to any preceding clause, wherein generating the final word includes learning an association between the text content and the image content to generate the final word.

[0089] 18. A computer-readable storage device according to any of the preceding clauses, wherein the operations further comprise: identifying errors using a multi-directional long short-term memory module; and correcting the errors using the multi-directional long short-term memory module.

[0090] 19. A computer-readable storage device according to any of the preceding clauses, wherein converting the text content into the word feature vector comprises using an optical character recognition model to recognize characters of the text content to construct words from the text content.

[0091] 20. A computer-readable storage device according to any of the preceding clauses, wherein converting the image content into the image feature vector comprises converting the image content into the image feature vector using a convolutional neural network.

Claims

1. A system for optical character recognition, characterized in that: include: A memory having a computer executable component stored therein; A processor, the processor executing at least the following computer executable components: an image classification component that uses a neural network to divide the scanned content into blocks and labels the blocks as first content and second content; a first content analysis component, which converts the first content into a word feature vector; a second content analyzing component, wherein the second content analyzing component converts the second content into an image feature vector; a word generating component, the word generating component using the word feature vector and the image feature vector to generate a final word, wherein the word generating component adopts a bidirectional long short-term memory module to learn the association between the first content and the second content to generate the final word; an error correction component that uses a multi-directional long short-term memory module to identify errors and correct the errors and evaluate the document type to verify the accuracy of the proposed words; and A multi-directional / bidirectional neural network that analyzes one or more previous word feature vectors or image vectors to determine contextual relationships, thereby adjusting the current word.

2. The system according to claim 1, characterized in that in, The first content analysis component includes a word construction component that constructs words from the first content by recognizing characters of the first content using an optical character recognition component.

3. The system according to claim 2, characterized in that in, The first content analyzing section includes a vector generating section that converts the word into the word feature vector.

4. The system according to claim 2, characterized in that The second content analysis component converts the second content into the image feature vector using a convolutional neural network.

5. A method for optical character recognition, characterized in that: include: By a system including a processor, using a neural network to divide the scanned content into blocks and label the blocks as text content and image content; By means of the system, the text content is converted into a word feature vector; By means of the system, the image content is converted into an image feature vector; By means of the system, a final word is generated using the word feature vector and the image feature vector, wherein generating the final word includes learning the association between the text content and the image content to generate the final word; By means of the system, errors are identified using a multi-directional long short-term memory module; Correcting the error using the multi-directional long short-term memory module by the system; By means of the system, the document type is evaluated using an error correction component to verify the accuracy of the proposed words; and With the system, a multi-directional / bidirectional neural network is used to analyze one or more previous word feature vectors or image vectors to determine contextual relationships, thereby adjusting the current word.

6. The method according to claim 5, characterized in that The converting of the text content into the word feature vector includes using an optical character recognition model to recognize characters of the text content to construct words from the text content.

7. The method according to claim 5, characterized in that The converting the image content into the image feature vector includes converting the image content into the image feature vector using a convolutional neural network.

8. The method according to claim 5, characterized in that Converting the text content into the word feature vector includes converting the text content into the word feature vector using a vector generation module.

9. A computer-readable storage device, characterized in that: The computer readable storage device includes instructions that, in response to being executed, cause a system including a processor to perform operations including: Using a neural network to divide the scanned content into blocks, and labeling the blocks as text content and image content; Converting the text content into a word feature vector; Converting the image content into an image feature vector; Using the word feature vector and the image feature vector to generate a final word, wherein generating the final word includes learning the association between the text content and the image content to generate the final word; Use multi-directional long short-term memory modules to identify errors; Correcting the error using the multi-directional long short-term memory module; evaluating the document type using an error correction component to verify the accuracy of the proposed words; and A multi-directional / bidirectional neural network is used to analyze one or more previous word feature vectors or image vectors to determine contextual relationships and thus adjust the current word.

10. The computer-readable storage device according to claim 9, wherein: in, Converting the text content into the word feature vector includes using an optical character recognition model to recognize characters of the text content to construct words from the text content.

11. The computer-readable storage device according to claim 9, wherein: in, Converting the image content into the image feature vector includes converting the image content into the image feature vector using a convolutional neural network.

Images