Methods and systems for image content recognition

Abstract

A method for identifying image content includes applying a neural network to an image, the neural network including an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers. The method further includes applying a branch of the neural network to an output of at least one layer of the plurality of hidden layers, the branch being independent of the neural network and having an output layer for generating an output related to an estimated estimation error level; and generating a combined output representing the estimated image content and the estimation error level.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 560,275, filed September 19, 2017, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] In some embodiments, the present invention relates to image processing, and more specifically, but not exclusively, to image content recognition. Background Technology

[0004] Optical character recognition (OCR) typically involves converting text images into codes that represent actual text characters. OCR techniques based on the Latin alphabet are widely used and have a high success rate. Handwritten text generally presents different recognition challenges than typed text.

[0005] Recurrent neural networks (RNNs) and their extensions are known in the field for handwriting recognition techniques. Examples include long short-term memory (LSTM) networks, hidden Markov models (HMMs) and their combinations [SA Azeem and H. Ahmed]. Effective techniques for recognizing offline Arabic handwritten words using HMMs. International Journal on Document Analysis and Recognition (IJDAR), 16(4):399–412, 2013; T. Bluche, H. Ney and C. Kermorvant. Comparison of optical modeling for handwriting recognition using sequence-trained deep neural networks and recurrent neural networks. Statistical Language and Speech Processing, pp. 199–210, Springer, 2014; P. Doetsch, M. Kozielski and H. Ney. Fast and reliable recurrent neural networks for offline handwriting recognition. In the field of handwriting recognition (Frontiers in Handwriting Recognition, ICFHR), 14th International Conference, 2014, pp. 279-284, IEEE, 2014; H. El Abed and V. Margner. Icdar 2009 Arabic handwriting recognition competition. International Journal of Document Analysis and Recognition (IJDAR), 14(1): 3-13, 2011; F. Menasri, J. Louradour, A. Bianne-Bernard and C. Kermorvant. A2iA French handwriting recognition system in the Rimes-ICDAR2011 competition. In SPIE conference proceedings, Vol. 8297, 2012; F. Stahlberg and S. Vogel. qcri recognition system for handwritten Arabic. In Image Analysis and Processing ICIAP 2015, pp. 276–286. Springer, 2015].

[0006] Another approach, published by Almazán et al. [J. Almazan, A. Gordo, A. Fornes and E. Valveny. Word discovery and recognition with embedding attributes. IEEE Journal of Pattern Analysis and Machine Intelligence, (12): 2552–2566, 2014], encodes the input word image as Fisher Vectors (FVs), which can be viewed as an aggregation of the gradients of a Gaussian Mixture Model (GMM) on low-level descriptors. A set of linear Support Vector Machine (SVM) classifiers is then trained, one for each binary attribute contained in each word attribute. Canonical Correlation Analysis (CCA) is used to link the vectors of predicted attributes to the binary attribute vectors generated from the actual words.

[0007] Another approach published by Jaderberg et al. [M. Jaderberg, K. Simonyan, A. Vedaldi and A. Zisserman. Synthetic data and artificial neural networks for text recognition in natural scenes. arXiv preprint arXiv: 1406.2227, 2014] uses convolutional neural networks (CNNs) trained on synthetic data for scene text recognition.

[0008] The arXiv preprint arXiv:1507.05717 by Shi et al. discloses a neural network that integrates feature extraction, sequence modeling, and transcription into a unified framework. The network consists of convolutional layers, recurrent layers, and transcription layers. Summary of the Invention

[0009] According to one aspect of some embodiments of the present invention, a method for recognizing image content is provided. The method includes: applying a neural network to an image, the neural network comprising: an input layer for receiving the image; a plurality of hidden layers for processing the image; and an output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers. The method further includes applying a branch of the neural network to an output of at least one layer of the plurality of hidden layers, the neural network branch being independent of the neural network and having an output layer for generating an output related to an estimated estimation error level. The method further includes generating a combined output representing the estimated image content and the estimation error level.

[0010] According to some embodiments of the present invention, the neural network branch includes at least one recurrent layer that generates multiple output values. According to some embodiments of the present invention, the at least one recurrent layer is a Long Short-Term Memory (LSTM) layer. According to some embodiments of the present invention, the LSTM layer is a bidirectional layer.

[0011] According to one aspect of some embodiments of the present invention, a method for recognizing image content is provided. The method includes applying a neural network to the image. The neural network includes: an input layer for receiving the image, a plurality of hidden layers for processing the image, and an output layer for generating and estimating output related to the image content based on the outputs of the hidden layers. The method further includes calculating a confidence score by obtaining the output probabilities generated by the neural network to two most probable recognitions, and assigning a confidence score related to the ratio or difference of the two probabilities.

[0012] According to some embodiments of the present invention, the method further includes summing or averaging the plurality of output values ​​to provide the estimated error level.

[0013] According to some embodiments of the present invention, the neural network includes generating a set of convolutional neural subnetworks having multiple feature vectors, each feature vector corresponding to an application of the convolutional neural subnetwork to a different window in the image.

[0014] According to some embodiments of the present invention, the neural network includes a recurrent neural subnetwork that receives a set of the plurality of feature vectors from the convolutional neural subnetwork, and the output of at least one of the plurality of hidden layers is an output of at least one layer of the recurrent neural subnetwork. According to some embodiments of the present invention, the recurrent neural subnetwork includes a plurality of long short-term memory (LSTM) layers. According to some embodiments of the present invention, at least one of the plurality of long short-term memory layers is a bidirectional layer.

[0015] According to some embodiments of the present invention, the recurrent neural subnetwork generates a sequence of multiple vectors, the method comprising projecting one dimension of each vector in the sequence onto a predetermined dimension corresponding to a length of a symbol system to provide a sequence of multiple probability distributions on the symbol system, each probability distribution corresponding to a region within the image.

[0016] According to some embodiments of the present invention, the neural network includes a connected temporal classification (CTC) layer that receives sequences of the plurality of probability distributions and generates the estimates.

[0017] According to some embodiments of the present invention, the method further includes converting a height of the image to a predetermined height before the application of the neural network.

[0018] According to some embodiments of the present invention, the image is an image of a plurality of handwritten symbols.

[0019] According to some embodiments of the present invention, the image is an image of a plurality of machine-printed symbols.

[0020] According to some embodiments of the present invention, the image includes at least one region containing a plurality of handwritten symbols and at least one region containing a plurality of machine-printed symbols.

[0021] According to some embodiments of the present invention, the image is an image of a bank check.

[0022] According to some embodiments of the present invention, the bank check includes a plurality of bank check fields, and the method is repeated for at least two of the plurality of bank check fields respectively.

[0023] According to one aspect of some embodiments of the present invention, a computer software product is provided, comprising a computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to receive an input image and execute the method described above, and are optionally and preferably described in detail below.

[0024] According to one aspect of some embodiments of the present invention, a system for recognizing image content includes an imaging system for imaging an object to provide an image of the object; and an image processor configured to: receive the image; apply a neural network to the image, the neural network including: an input layer for receiving the image; a plurality of hidden layers for processing the image; and an output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers; apply a branch of the neural network to an output of at least one layer of the plurality of hidden layers, the branch of the neural network being independent of the neural network and having an output layer for generating an output related to an estimated estimation error level; and generate a combined output representing the estimated image content and the estimation error level.

[0025] According to one aspect of some embodiments of the present invention, a system for recognizing image content is provided, the system comprising a network interface for receiving an image via a communication network; and an image processor configured to: apply a neural network to the image, the neural network comprising: an input layer for receiving the image; a plurality of hidden layers for processing the image; and an output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers; applying a branch of the neural network to an output of at least one layer of the plurality of hidden layers, the neural network branch being independent of the neural network and having an output layer for generating an output related to an estimated estimation error level; and generating a combined output representing the estimated image content and the estimation error level.

[0026] Unless otherwise defined, all technical and / or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. While similar or equivalent methods and materials described herein may be used in the practice or testing of embodiments of the invention, exemplary methods and / or materials are described below. In case of conflict, the patent specification and its definitions shall prevail. Furthermore, the materials, methods, and embodiments are illustrative only and are not intended to be necessarily restrictive.

[0027] The implementation of the methods and / or systems of the embodiments of the present invention may involve performing or completing selected tasks manually, automatically, or in a combination thereof. Furthermore, the actual instruments and equipment of the embodiments of the methods and / or systems according to the present invention can use an operating system to implement several selected tasks through hardware, software, firmware, or a combination thereof.

[0028] For example, the hardware for performing a selected task according to embodiments of the present invention can be implemented as a chip or circuit. As software, the selected task according to embodiments of the present invention can be implemented as a plurality of software instructions executed by a computer using any suitable operating system. In exemplary embodiments of the present invention, one or more tasks according to exemplary embodiments of the methods and / or systems described herein are performed by a data processor, for example, a computing platform for executing a plurality of instructions. Optionally, the data processor includes volatile memory for storing instructions and / or data and / or non-volatile memory for storing instructions and / or data, such as a magnetic hard disk and / or removable media. Optionally, a network connection is also provided. A display and / or a user input device, such as a keyboard or mouse, may also be optionally provided. Attached Figure Description

[0029] Some embodiments of the invention are described herein by way of example only, with reference to the accompanying drawings. Referring now specifically to the drawings, it is emphasized that the details shown are by way of example and for the purpose of illustrative discussion of embodiments of the invention. In this regard, the description taken in conjunction with the drawings will make it apparent to those skilled in the art how embodiments of the invention can be practiced.

[0030] In the attached diagram:

[0031] Figure 1 This is a flowchart of a method suitable for recognizing image content according to various exemplary embodiments of the present invention.

[0032] Figure 2A and 2B This is a schematic diagram of representative examples of neural networks and neural network branches suitable for recognizing image content according to various exemplary embodiments of the present invention.

[0033] Figure 3 This is a schematic diagram of a computer system suitable for recognizing image content according to various exemplary embodiments of the present invention.

[0034] Figure 4A , Figure 4B and Figure 4C These are images showing examples of three datasets used in experiments conducted according to some embodiments of the present invention.

[0035] Figure 5 This is a schematic diagram of the training process of the error prediction branch used in experiments performed according to some embodiments of the present invention.

[0036] Figure 6 The receive operating characteristic (ROC) curves obtained in experiments conducted according to some embodiments of the present invention are shown.

[0037] Figure 7A and 7B The ROC curves shown are obtained from experiments using a testing instrument owned by Orbograph Ltd. of Yavne, Israel, according to some embodiments of the invention.

[0038] Figure 8A -D shows the ROC curves generated in experiments conducted using test sets referred to as IAM, RIMES, SVT, and IC13, respectively, according to some embodiments of the invention.

[0039] Figure 9 This displays examples of non-rejection images from a test set owned by Orbograph Ltd., which were used during experiments performed according to some embodiments of the invention. Detailed Implementation

[0040] In some embodiments, the present invention relates to image processing, and more specifically, but not exclusively, to image content recognition.

[0041] Before explaining at least one embodiment of the present invention in detail, it should be understood that the application of the present invention is not necessarily limited to the description in the following description and / or the details of the construction and arrangement of the elements and / or methods shown in the drawings and / or examples. The present invention can have other embodiments or can be practiced or performed in various ways.

[0042] Figure 1 This is a flowchart of a method for recognizing image content according to various exemplary embodiments of the present invention. It should be understood that, unless otherwise defined, the operations described below can be performed simultaneously or sequentially in many combinations or orders of execution. In particular, the order of the flowchart is not considered limiting. For example, two or more operations appearing in a particular order in the following description or flowchart may be performed in a different order (e.g., the reverse order) or substantially simultaneously. Furthermore, several operations described below are optional and may not be performed.

[0043] At least some of the operations described herein can be implemented by a data processing system, such as a dedicated circuit or a general-purpose computer, configured to receive data and perform the operations described below. At least some of the operations can be implemented by a cloud computing facility located in a remote location.

[0044] Computer programs implementing the methods of this embodiment can typically be transmitted via communication networks or on media such as, but not limited to, floppy disks, CD-ROMs, flash memory devices, and portable hard disks. Computer programs can be copied from communication networks or distribution media to hard disks or similar intermediate storage media. Computer programs can be run by loading code instructions from their distribution media or their intermediate storage media into the computer's execution memory and configuring the computer to operate according to the methods of the present invention. During operation, the computer can store data structures or values ​​obtained through intermediate computation in memory and retrieve these data structures or values ​​for subsequent operations. All of these operations are well known to those skilled in the art of computer systems.

[0045] The processing operations described herein can be performed using processor circuits such as DSPs, microcontrollers, FPGAs, ASICs, or any other conventional and / or special-purpose computing systems.

[0046] The method of this embodiment can be implemented in various forms. For example, it can be embodied in a tangible medium, such as a computer for performing the method operations. It can be embodied on a computer-readable medium comprising computer-readable instructions for performing the method operations. The invention can also be implemented in an electronic device with digital computing capabilities, which is arranged to run a computer program on a tangible medium or execute instructions on a computer-readable medium.

[0047] refer to Figure 1 As shown, the method begins at 10 and optionally and preferably continues to 11, at which point an image is received. The image typically contains symbols belonging to the alphabet, including but not limited to characters, accent marks, numbers, and / or punctuation marks. In some embodiments of the invention, the image contains handwritten symbols, in which case the letters are a set of handwritten symbols. In some embodiments of the invention, the image contains printed symbols, in which case the letters are a set of printed symbols. Images containing both handwritten and printed symbols are also considered. A representative example of such an image is an image of a bank check with multiple fields, some of which may include handwritten symbols (e.g., the number of digits and words, etc.), and some of which may include printed symbols (e.g., account number, name of the account holder, etc.).

[0048] The image is preferably a digital image and can be received from an external source (e.g., a storage device storing the image in a computer-readable form) and / or sent to a data processor performing the method on a communication network, such as the Internet, but not limited thereto.

[0049] The method optionally and preferably continues to step 12, where the size of the input image is adjusted. This operation is particularly useful when the image is received as input from an external source. Resizing may include stretching or shrinking the image along any axis to a predetermined width, predetermined length, and / or predetermined diagonal, as known in the art. The sizing along each axis can be done independently. For example, the method may transform only the height of the image to a predetermined height without changing the width, or vice versa, or transform both the height and width depending on the specific transformation scenario.

[0050] In 14, a neural network is applied to an image. Figure 2A and 2BThe diagram shows a representative example of a neural network 20 suitable for this embodiment. The neural network 20 typically includes an input layer 24 for receiving the image 22, a plurality of hidden layers 26 for processing the image, and an output layer 28 for generating an output related to estimating the image content based on the output of the hidden layers 26. Optionally and preferably, the neural network is pre-trained to estimate the image content based on a specific letter, a subset of which is contained in the image to which the neural network is designed to be applied.

[0051] In some embodiments of the present invention, the hidden layer 26 of the neural network 20 includes a convolutional neural subnetwork 30 (see [link]). Figure 2B The convolutional neural subnetwork 30 generates a set 32 ​​with multiple feature vectors, each feature vector corresponding to a different window within the image 22 to which the convolutional subnetwork 30 is applied. The set 32 ​​of multiple feature vectors can thus represent a window moving from one side of the image to the other. Optionally, but not necessarily, the convolutional neural subnetwork 30 is a fully convolutional neural subnetwork.

[0052] Unlike a fully connected network or subnetwork that associates a single value with each neuron of the network or subnetwork, a convolutional neural network or subnetwork operates by associating an array of values ​​with each neuron. Conceptually, this array can be viewed as a pixel block of the input image. The transformation of neuron values ​​in subsequent layers is from multiplication to convolution. This means that the connection strength is the convolution kernel rather than a scalar value. These more complex transformations involve more complex neural network matrices. Therefore, while the matrices in a fully connected network or subnetwork consist of arrays of numerical values, in a convolutional neural network or subnetwork, each matrix entry is a pixel block.

[0053] The neural network 20 may optionally include a recurrent neural subnetwork 34, which receives a set 32 ​​with multiple feature vectors from the convolutional neural subnetwork 30. The recurrent neural subnetwork 34 may optionally use one or more long short-term memory (LSTM) layers, and more preferably one or more bidirectional LSTM layers.

[0054] Recurrent neural networks, or subnetworks, are designed to accept sequences of varying lengths as input. The same set of weights is used in the processing of each sequence element. Such subnetworks are constructed hierarchically, such that each neuron in a layer receives activations from all neurons in the preceding layer as input, and furthermore, there are lateral connections with activations induced in the previous step of the sequence. Bidirectional neural networks, or subnetworks, use forward and backward layers. These two types of layers exist in parallel, and their activations (cascaded) serve as input to the next layer.

[0055] Long Short-Term Memory (LSTM) layers use memory cells instead of simple activations. Access to these memory cells is controlled by a multiplicative factor called gating. In each input state, a gating system determines the new input portion to be written into the memory cell, the portion of the cell's current content to be forgotten, and the portion of the content to be output. For example, if the output gate is closed (value 0), the neuron connected to the current neuron has a value of 0. If the output gate is partially open with a gate value of 0.5, the neuron can output half of the current value of the stored memory.

[0056] The recurrent neural subnetwork 34 typically generates a sequence 36 of multiple vectors. In these embodiments, the method optionally and preferably projects the dimension of each vector in the sequence 36 onto a predetermined dimension corresponding to the length of the letter, thus providing a series of probability distributions on the alphabet, where each probability distribution corresponds to a region within the image. In some embodiments of the invention, the output layer 28 includes a Connectionist Temporal Classification (CTC) layer [Graves et al., Connectionist Temporal Classification: Labeling Unsegmented Sequence Data Using Recurrent Neural Networks. Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 369–376, New York, USA, 2006. ACM]. The sequence of probability distributions is received, and an estimate of the image content is generated based on the letters.

[0057] The CTC layer can be used to train the network or to use the network to recognize image content. For the former, a loss function is used. For the latter, a CTC decoding scheme can be optionally and preferably used to obtain the output. Greedy decoding can be used to obtain the highest probability decoder, or a beam search scheme can be used to obtain n highest probability decoder values.

[0058] The method optionally and preferably proceeds to 14, where a neural network branch 38 is applied to the output of at least one of the hidden layers 26. The network branch 38 is applied to the output of any hidden layer 26. One or more recurrent layers of branch 38 can be, for example, LSTM layers, more preferably bidirectional LSTM layers. Typically, branch 38 is applied to the output of one layer in the recurrent neural subnetwork 34, such as the output of one of the LSTM layers, when such a layer is used. Preferably, but not necessarily, the network branch 38 is applied to the output of the first layer in the recurrent neural subnetwork 34, i.e., the output of the group 32 of multiple feature vectors generated by the convolutional neural subnetwork 30 fed to the layer.

[0059] Neural network branch 38 is optionally and preferably independent of neural network 20. Branch 38 has an output layer 40 that produces an output related to the estimated error level generated by layer 28. Optionally and preferably, neural network branch 38 includes one or more recurrent layers 42 that produce multiple output values. The method optionally and preferably sums the output values ​​to provide the estimated error level. Typically, the output values ​​provided by the recurrent layers 42 are vector values. In these embodiments, each vector value provided by the recurrent layers 42 is projected onto a scalar, thereby providing multiple scalars that can optionally and preferably be subsequently summed or averaged, so that the error level generated by the output layer 40 is also a scalar.

[0060] The method proceeds to step 15, where a combined output indicating the estimated image content and the estimated error level is generated. This combined output can be displayed on a display device, printed on a print medium, and / or transmitted to a remote computer via a communication network.

[0061] The method ends at 16.

[0062] Figure 3 This is a schematic diagram of a client computer 130 having a hardware processor 132, which typically includes input / output (I / O) circuitry 134, a hardware central processing unit (CPU) 136 (e.g., a hardware microprocessor), and hardware memory 138, typically including volatile and non-volatile memory. The CPU 136 communicates with the I / O circuitry 134 and the hardware memory 138. The client computer 130 preferably includes a graphical user interface (GUI) 142 that communicates with the processor 132. The I / O circuitry 134 preferably transmits information to and from the GUI 142 in a suitable configuration. A server computer 150 is also shown, which may similarly include a hardware processor 152, I / O circuitry 154, a hardware CPU 156, and hardware memory 158. The I / O circuitry 134 and 154 of the client computer 130 and server computer 150 can function as transceivers, communicating information with each other via wired or wireless communication. For example, client computer 130 and server computer 150 can communicate via network 140 such as a local area network (LAN), wide area network (WAN), or the Internet. In some embodiments, server computer 150 may be part of a cloud computing resource of a cloud computing facility that communicates with client computer 130 via network 140. Further shown is an imaging device 146, such as a camera or scanner, associated with client computer 130.

[0063] The GUI 142 and processor 132 can be integrated within the same housing, or they can be separate units that communicate with each other. Similarly, the imaging device 146 and processor 132 can be integrated within the same housing, or they can be separate units that communicate with each other.

[0064] GUI 142 can optionally and preferably be part of a system including a dedicated CPU and I / O circuitry (not shown) to allow GUI 142 to communicate with processor 132. Processor 132 publishes graphical and textual output generated by CPU 136 to GUI 142. Processor 132 also receives signals from GUI 142 in response to user input relating to control commands generated by GUI 142. GUI 142 can be of any type known in the art, such as, but not limited to, a keyboard and display, touchscreen, etc. In a preferred embodiment, GUI 142 is the GUI of a mobile device such as a smartphone, tablet, smartwatch, etc. When GUI 142 is the GUI of a mobile device and processor 132, the CPU circuitry of the mobile device can be used as processor 132 and can execute the code instructions described herein.

[0065] Client computer 130 and server computer 150 may further include one or more computer-readable storage media 144, 164, respectively. Media 144 and 164 are preferably non-transitory storage media storing computer code instructions, as described in further detail herein, and processors 132 and 152 execute these code instructions. The code instructions can be executed by loading the corresponding code instructions into the corresponding hardware memories 138 and 158 of the respective processors 132 and 152.

[0066] Each of storage media 144 and 164 may store program instructions that, when read by the corresponding processor, cause the processor to receive an input image and execute the methods described herein. In some embodiments of the invention, the input image is generated by imaging device 146 and transmitted to processor 132 by means of I / O circuitry 134. Processor 132 estimates the content of the image and the estimated error, as described above, and displays the combined output on GUI 142. Alternatively, processor 132 may send the image to server computer 150 via network 140. As described above, computer 150 receives the image, estimates the content of the image and the estimated error, and sends the combined output back to computer 130 via network 140. Computer 130 receives the combined output and displays it on GUI 142.

[0067] As used in this article, the term "about" refers to approximately 10%.

[0068] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments and / or as excluding the incorporation of features from other embodiments.

[0069] The term "optionally" is used herein to mean "provided in some embodiments but not in others". Any particular embodiment of the invention may include a number of "optional" features unless these features conflict.

[0070] The terms “comprises,” “comprising,” “includes,” “including,” “having,” and their common words mean “including but not limited to.”

[0071] The phrase "composed of" means "including and limited to".

[0072] The term "consistently made up of" means that the composition, method or structure may include additional ingredients, steps and / or portions, provided that the additional ingredients, steps and / or portions do not materially alter the basic and novel features of the claimed composition, method or structure.

[0073] As used herein, the singular forms “a,” “an,” and “the” include plural references unless the context clearly indicates otherwise. For example, the terms “a compound” or “at least one compound” may include multiple compounds, including mixtures thereof.

[0074] Throughout this application, various embodiments of the invention can be presented in a scope format. It should be understood that the scope format is for convenience and brevity only and should not be construed as an inflexible limitation on the scope of the invention. Therefore, the scope description should be considered as explicitly disclosing all possible sub-scopes and the individual numerical values ​​within those scopes. For example, a description of a scope from 1 to 6 should be considered as explicitly disclosing sub-scopes such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and the individual numbers within that scope, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the scope.

[0075] Whenever a range of numbers is indicated herein, it is intended to include any referenced number (fraction or integer) within the indicated range. The terms “ranging” or “ranges” between the first and second indicator numbers and “ranging” or “ranges” between the first and second indicator numbers are used interchangeably herein and are intended to include the first and second indicator numbers and all decimals and integers in between.

[0076] It should be understood that certain features of the invention described in the context of a single embodiment for clarity may also be provided in combination in a single embodiment. Conversely, for brevity, various features of the invention described in the context of a single embodiment may also be provided individually or in any suitable sub-combination or suitably in any other described embodiment of the invention. Certain features described in the context of various embodiments should not be considered essential features of those embodiments unless the embodiment is inoperable without those elements.

[0077] As described above, various embodiments and aspects of the present invention are experimentally supported in the following examples.

[0078] Example

[0079] The following embodiments, together with the foregoing description, illustrate some implementations of the invention in a non-limiting manner.

[0080] This example describes an optical character recognition (OCR) process applied to handwritten and printed bank checks. The applied process includes a method according to some embodiments of the invention, wherein a neural network is applied to an image to estimate image content based on the output of hidden layers, and a branch of the neural network is applied to the output of one of the hidden layers to estimate the error level of the image content estimation.

[0081] In this example, three different datasets are used, each containing a real-world image of a different field from a bank check. The datasets are named Account, CAR, and LAR, where Account corresponds to the numeric field of the account number, CAR corresponds to the dollar amount of the check in numeric form, and LAR corresponds to the dollar amount of the check in textual form. Figures 4A-4C Showing data from the LAR dataset ( Figure 4A (Displays "75 and 00"), CAR dataset ( Figure 4B (showing "486.37") and account dataset ( Figure 4C The example image shown is "161020417".

[0082] For each image, the dataset contains a reference transcription of its contents.

[0083] Network architecture

[0084] The network architecture is as follows. For the recognition part of the network, an end-to-end trainable network is used. The network is trained only on images and their transcriptions. In this example, the image height is normalized to 64 pixels, and the image width and the length of the transcription are arbitrary. No alignment information is used to train the network.

[0085] The input image is first passed through a VGG-style convolutional neural network, which consists of convolutional layers, batch normalization layers, and max pooling layers. In this example, ReLU is used as the activation function, but other activation functions can also be considered. In this example, the pooling layers reduce the image height from 64 to 1 and also reduce the image width by a factor of 8. This provides three 2×2 max pooling layers and three 2×1 max pooling layers.

[0086] Three convolutional layers are used between every two merge operations. In this example, the convolutional layers are 3×3 in shape with 1 padding. A batch normalization layer and an activation function are used after each convolutional layer.

[0087] The output of the convolutional stage is a series of feature vectors, each of which is the result of applying the convolutional network to a window of the original image. Therefore, the feature vectors correspond to the image with a sliding window from left to right.

[0088] The obtained feature vector sequence is fed as input to a recurrent subnetwork, which in this example comprises two bidirectional LSTM layers. Both bidirectional LSTM layers are implemented as concatenated LSTM layers of 1024 units each. After the convolutional and LSTM layers, a variable-length sequence of vectors of length 2048 is obtained. A linear projection from size 2048 is then applied to the desired letter length to obtain a sequence of probability distributions over the letters, each probability distribution corresponding to a rectangular receptive field in the original image.

[0089] The final transcription is given by applying a connectionist temporal classification (CTC) layer to the resulting sequence. Given a probability vector from the previous layer, this layer defines the probabilities for the output sequence. The output sequence with the highest probability vector is then defined. For training, the loss function is defined as the negative log-likelihood of the correct transcription given the probability distribution for each window.

[0090] Prediction error

[0091] The inventors discovered that CTC transcription probability performs poorly as a predictive metric for image error labeling. Therefore, they have attached an error prediction branch to the network. The input to the error prediction branch is a hidden layer in the middle of the network, and the output is a single number that can be used to predict the number of errors in transcription (edit distance) for a regression task, or as the log probability of correct transcription for a classification task.

[0092] Figure 5 This demonstrates the training process of the network's error prediction branch. For training, only the source images and correct transcriptions are used. The training base reality can be calculated by running the images in the sequence prediction network, calculating the observed error between the network output and the baseline reality, and training the error prediction branch based on this difference.

[0093] In this example, the error prediction branch takes the output of the first bidirectional LSTM layer as its input. Each vector of length 2048 is projected onto a smaller vector (128 units in this example). Another bidirectional LSTM layer (128 units in this example) is then applied to the projected vectors. Each output of the LSTM layer is projected onto a scalar, and these scalars are summed to provide a scalar output. Alternatively, a scalar output can be provided by averaging these scalars (instead of simply summing them), for example, to reduce the influence of image width on the results. The scalar output is defined as the output of the error prediction branch and is used as the log probability for the classification process.

[0094] Summarizing or averaging the projected scalars from the LSTM provides locality of error detection. This is advantageous for both classification (where the method predicts whether the detected label matches the baseline reality) and regression (where the method predicts the number of errors made by the network). For classification, when the network outputs large values ​​at two distinct locations, the final sum is larger, consistent with the observation of a large error probability. For regression, when the network makes multiple errors, the error detection process is expected to detect errors at multiple locations, and the higher and more numerous the location values, the higher the error prediction should be.

[0095] The sequence prediction network and the error prediction branch were not trained on the same dataset. This is because deep neural network models often overfit the training set. When overfitting occurs, the loss function and the perceived labeling error will reach 0 or be very close. If the error prediction branch is trained together with the network, the result is that the error predictor consistently outputs zero. Therefore, training is a two-stage operation where the network providing the transcription is trained on the training set and evaluated on the validation set; the error prediction branch is trained on the validation set and evaluated on the test set.

[0096] Experimental results

[0097] The system was trained on three datasets (CAR, LAR, and Account) and achieved good accuracy. The measured error was the total edit distance divided by the length of the correct string. The results are summarized in Table 1 below. In Table 1, the overall field accuracy results for LAR were measured after post-processing of the transcribed data. Measurements of accuracy for dollar amounts and cent amounts are provided separately.

[0098] Table 1

[0099]

[0100] To make error predictions, six classification models were trained, and their results were compared. Three of the models were binary classification models used to predict whether the resulting string was correct, while the other three were regression models that attempted to minimize the loss from the edit distance.

[0101] These three models are:

[0102] 1. LSTM summation - As mentioned above, LSTM, projection and summation.

[0103] 2. LSTM final state - Run a bidirectional LSTM, obtain its final state vector, and project it to obtain a scalar.

[0104] 3. LSTM Final State MLP - Run a bidirectional LSTM to obtain its final state vector, and run a small neural network model with hidden layers to obtain a scalar.

[0105] Figure 6The Receiver Operating Characteristic (ROC) curves for all six models are displayed, along with the reference CTC score on the CAR dataset. The ROC curves illustrate the trade-off between the false read rate and the overall system read rate. For example, a point with a false read rate of 0.2 and a read rate of 0.9 indicates that, by selecting a certain confidence threshold, the system will reject 10% of valid data and reduce the number of errors by 80%. The area under the curve (AUC) measure was used to compare the effectiveness of different confidence methods. Figure 6 As shown, the CTC score is a very poor measure of misprediction. Conversely, the LSTM and classification models in this embodiment significantly outperform all other models, with an AUC of 0.87.

[0106] Additional experiments were conducted using several benchmarks and on a dataset owned by Orbograph Ltd. in Yavne, Israel, including real cropped images of dollar amounts scanned by each method.

[0107] In the following experiments, two techniques were used to obtain confidence scores. In the first technique (hereinafter referred to as the "CTC ratio"), the probability ratio was used as the confidence score without using neural network branch 38. In this technique, the probabilities assigned to the first and second most likely readings of a given input image by the OCR model were obtained. A higher confidence score was assigned if the first reading was more likely than the second. If not, the sample was ambiguous, and therefore its confidence score was reduced. In the second technique (hereinafter referred to as "ErrPred"), an error prediction branch was added to the network to directly predict whether the OCR model would make a mistake given an input or whether the sample should be rejected. The error prediction branch formulated neural network branch 38 ( Figure 2A ).

[0108] Benchmarks include: ICDAR 2013 (IC13) [Karatzas et al., Icdar 2013 Robust Reading Competition, Document Analysis and Recognition (ICDAR), 12th International Conference 2013, pp. 1484–1493. IEEE, 2013], Street View Text (SVT) [Wang et al., End-to-End Scene Text Recognition. In Computer Vision (ICCV), 2011 IEEE International Conference, pp. 1457–1464. IEEE, 2011], IAM [Marti et al., “iam database: English sentence database for offline handwriting recognition, International Journal of Document Analysis and Recognition, 5(1): 39–46, 2002”]; and RIMES [Grosicki et al., Icdar 2009 Handwriting Recognition Competition, Document Analysis and Recognition, 2009. ICDAR'09]. The 10th International Conference, pp. 1398–1402, IEEE, 2009.

[0109] In the IC13 dataset, the test set contains 251 scene images with labeled bounding boxes. The test set is restricted to alphanumeric characters containing only three or more characters, resulting in approximately 900 cropped text images.

[0110] The SVT dataset contains 249 images collected from Google Street View. Word images were cropped from these images to obtain a test set of 647 samples.

[0111] The IAM dataset consists of handwritten English text scanned by 657 authors, providing a total of 115,320 labeled word images. After filtering out punctuation and short words, a training set of 40,526 samples was obtained.

[0112] The RIMES dataset consists of handwritten French text scanned by 1,300 people, totaling 67,000 words. Limited to sufficiently long words without punctuation, the training set size became 35,723 samples.

[0113] The Orbograph dataset consists of one million labeled images, automatically scanned and cropped to represent dollar amounts. This dataset was created by collecting scanned text samples of dollar amounts from actual printed documents and manually labeling them. Because this is unfiltered real-world data, it contains a number of rejected samples, which were used to train the method in this embodiment. Figure 7A and 7B The ROC curves on the Orbograph dataset test set are shown, where Figure 7A Display the ROC curves for all images in the test set. Figure 7B The ROC curves for the non-rejection images in the test set are shown. Misreading refers to receiving a rejection image with a high confidence score, or receiving a non-rejection image with a high confidence score that was erroneously transcribed.

[0114] The experiments were conducted in TensorFlow and trained on an NVIDIA Titan X and an NVIDIA K80. A search beam of size 100 was used for decoding with CTC. Training was performed using ADAM [Kingma et al., A Stochastic Optimization Approach, arXiv preprint arXiv: 1412.6980, 2014] with an initial learning rate of 10⁻⁴.

[0115] Table 2 summarizes the accuracy obtained for each benchmark and the AUC obtained using the baseline and method of this embodiment, and Table 3 summarizes the accuracy obtained for the digital dollar amount dataset and the AUC obtained using the baseline and method of this embodiment. In Table 3, the accuracy of Orbograph (all) is used to treat all rejected samples as misses, thus demonstrating the need for a confidence signal for accuracy.

[0116] Figure 8A -D respectively displays the confidence score based on the baseline CTC and the benchmark ROC curves of IAM, RIMES, SVT and IC13 of the method in this embodiment.

[0117] Table 2

[0118]

[0119] Table 3

[0120]

[0121] Tables 2, 3 and Figure 8A -D demonstrates that the method of this embodiment is effective in predicting and preventing OCR errors. The probability ratio technique has proven highly effective across all datasets without sample removal. Both the probability ratio technique and the technique employing the error prediction branch yielded convincing results when trained on large datasets. Without wishing to be bound by any particular theory, the relatively lower accuracy obtained using the error prediction branch is explained by a lack of data.

[0122] Orbograph's dataset provides better benchmark conditions because it is not synthetic and contains a large number of samples. Furthermore, the dataset simulates the real-world conditions and requirements of OCR systems by including defective samples. Figure 7A and 7B As shown, both techniques perform well when measuring the accuracy of error prediction without considering defective products, with the probability ratio method having a slight advantage. However, when defective samples are also considered, the technique using the error prediction branch yields excellent results. The latter technique can reduce the error rate by up to 99.7% while rejecting only 1% of valid samples.

[0123] Figure 9This section shows examples of non-rejection images at different confidence levels from the Orbograph dollar amount dataset for qualitative evaluation. It is expected that for an OCR model trained with cropped printed dollar amounts, it will be difficult to read samples at the lower confidence percentile. Indeed, one sample is handwritten, one includes surrounding boxes due to improper cropping, one contains irrelevant text and markings, and two others are blurry. The text in the fifth percentile is completely legible. Note that the top percentile is entirely dominated by samples labeled "$0.00". This is because the dataset includes many such samples, allowing the neural network in this example to learn to recognize them with high confidence.

[0124] The examples demonstrate that the technology of this embodiment significantly outperforms the proposed baseline and proves applicable to printed text OCR, handwritten text OCR, and scene text recognition.

[0125] Although the invention has been described in conjunction with specific embodiments thereof, it will be apparent to those skilled in the art that many alternatives, modifications, and variations will be readily apparent. Therefore, it is intended to cover all such alternatives, modifications, and variations falling within the spirit and broad scope of the appended claims.

[0126] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference in their entirety to the same extent as if each individual publication, patent, or patent application were specifically and individually indicated to be incorporated herein by reference. Furthermore, any reference or designation of any reference in this application should not be construed as an admission that such reference is available as prior art to the invention. The use of the headings in this section should not be construed as an inherent limitation.

Claims

1. A method for recognizing image content, comprising: A neural network is applied to an image, the neural network comprising: an input layer for receiving the image; Multiple hidden layers are used to process the image; and a first output layer for generating an output related to an estimated image content based on the multiple outputs of the multiple hidden layers; A neural network branch is applied to the output of at least one of the plurality of hidden layers, the neural network branch being independent of the neural network and having a second output layer for generating an output related to an estimated level of estimation error, the neural network branch being trained based on the difference between the baseline reality and the output generated by the first output layer of the neural network, the second output layer being decoupled from the first output layer; and A combined output is generated, which represents the estimated image content and the estimation error level; The neural network is pre-trained to estimate the image content based on a single letter, a subset of which is contained in the image.

2. The method for recognizing image content as described in claim 1, wherein: The neural network branch contains at least one recurrent layer that produces multiple output values.

3. The method for recognizing image content as described in claim 2, wherein: The at least one cyclic layer is a long short-term memory layer.

4. The method for recognizing image content as described in claim 3, wherein: The long short-term memory layer is a bidirectional layer.

5. The method for recognizing image content as described in any one of claims 2-4, wherein: The method also includes summing or averaging the plurality of output values ​​to provide the estimated error level.

6. The method for recognizing image content as described in claim 1, wherein: The neural network comprises generating a set of convolutional neural subnetworks with multiple feature vectors, each feature vector corresponding to an application of the convolutional neural subnetwork to a different window in the image.

7. The method for recognizing image content as described in any one of claims 2-4, wherein: The neural network comprises a convolutional neural subnetwork that generates a set of multiple feature vectors, each feature vector corresponding to an application of the convolutional neural subnetwork to a different window in the image.

8. The method for recognizing image content as described in claim 6, wherein: The neural network includes a recurrent neural subnetwork that receives a set of the plurality of feature vectors from the convolutional neural subnetwork, and the output of at least one of the plurality of hidden layers is an output of at least one layer of the recurrent neural subnetwork.

9. The method for recognizing image content as described in claim 7, wherein: The neural network includes a recurrent neural subnetwork that receives a set of the plurality of feature vectors from the convolutional neural subnetwork, and the output of at least one of the plurality of hidden layers is an output of at least one layer of the recurrent neural subnetwork.

10. The method for recognizing image content as described in claim 8, wherein: The recurrent neural subnetwork contains multiple long short-term memory layers.

11. The method for recognizing image content as described in claim 9, wherein: The recurrent neural subnetwork contains multiple long short-term memory layers.

12. The method for recognizing image content as described in claim 10, wherein: At least one of the multiple long short-term memory layers is a bidirectional layer.

13. The method for recognizing image content as described in claim 11, wherein: At least one of the multiple long short-term memory layers is a bidirectional layer.

14. The method for recognizing image content as described in claim 8, wherein: The recurrent neural subnetwork generates a sequence of multiple vectors, the method comprising projecting one dimension of each vector in the sequence onto a predetermined dimension corresponding to a length of a symbol system to provide a sequence of multiple probability distributions on the symbol system, each probability distribution corresponding to a region within the image.

15. The method for recognizing image content as described in claim 9, wherein: The recurrent neural subnetwork generates a sequence of multiple vectors, the method comprising projecting one dimension of each vector in the sequence onto a predetermined dimension corresponding to a length of a symbol system to provide a sequence of multiple probability distributions on the symbol system, each probability distribution corresponding to a region within the image.

16. The method for recognizing image content as described in claim 14, wherein: The neural network includes a connected temporal classification layer that receives sequences of the plurality of probability distributions and generates the estimates.

17. The method for recognizing image content as described in claim 15, wherein: The neural network includes a connected temporal classification layer that receives sequences of the plurality of probability distributions and generates the estimates.

18. The method for recognizing image content as described in claim 1, wherein: The method further includes converting a height of the image to a predetermined height before the application of the neural network.

19. The method for recognizing image content as described in any one of claims 2-4, wherein: The method further includes converting a height of the image to a predetermined height before the application of the neural network.

20. The method for recognizing image content as described in claim 1, wherein: The image is an image of multiple handwritten symbols.

21. The method for recognizing image content as claimed in any one of claims 2-4, wherein: The image is an image of multiple handwritten symbols.

22. The method for recognizing image content as described in claim 1, wherein: The image is an image of multiple machine-printed symbols.

23. The method for recognizing image content as described in any one of claims 2-4, wherein: The image is an image of multiple machine-printed symbols.

24. The method for recognizing image content as described in claim 1, wherein: The image includes at least one region containing multiple handwritten symbols and at least one region containing multiple machine-printed symbols.

25. The method for recognizing image content as described in any one of claims 2-4, wherein: The image includes at least one region containing multiple handwritten symbols and at least one region containing multiple machine-printed symbols.

26. The method for recognizing image content as described in claim 1, wherein: The image is an image of a bank check.

27. The method for recognizing image content as claimed in any one of claims 2-4, wherein: The image is an image of a bank check.

28. The method for recognizing image content as described in claim 26, wherein: The bank check contains multiple bank check fields, and the method is repeated for at least two of the multiple bank check fields respectively.

29. The method for recognizing image content as described in claim 27, wherein: The bank check contains multiple bank check fields, and the method is repeated for at least two of the multiple bank check fields respectively.

30. A computer software product comprising a computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to receive an input image and perform the method according to any one of claims 1-4.

31. A system for recognizing image content, comprising: An imaging system for imaging an object to provide an image of the object; and An image processor configured for: Receive the image; A neural network is applied to an image, the neural network comprising: an input layer for receiving the image; a plurality of hidden layers for processing the image; and a first output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers; A neural network branch is applied to the output of at least one of the plurality of hidden layers, the neural network branch being independent of the neural network and having a second output layer for generating an output related to an estimated level of estimation error, the neural network branch being trained based on the difference between the baseline reality and the output generated by the first output layer of the neural network, the second output layer being decoupled from the first output layer; and A combined output is generated, which represents the estimated image content and the estimation error level; The neural network is pre-trained to estimate the image content based on a single letter, a subset of which is contained in the image.

32. A system for recognizing image content, comprising: A network interface for receiving an image through a communication network; and An image processor configured for: A neural network is applied to an image, the neural network comprising: an input layer for receiving the image; a plurality of hidden layers for processing the image; and a first output layer for generating an output related to an estimated image content based on a plurality of outputs of the plurality of hidden layers; A neural network branch is applied to the output of at least one of the plurality of hidden layers, the neural network branch being independent of the neural network and having a second output layer for generating an output related to an estimated level of estimation error, the neural network branch being trained based on the difference between the baseline reality and the output generated by the first output layer of the neural network, the second output layer being decoupled from the first output layer; and A combined output is generated, which represents the estimated image content and the estimation error level; The neural network is pre-trained to estimate the image content based on a single letter, a subset of which is contained in the image.

Images