Positional embedding for document processing

Abstract

Disclosed herein are system, method, and computer program product embodiments for generating document tags using position embeddings. In an embodiment, a tagging system may identify tags, such as words, for a document image. The tagging system may apply a position vector neural network to the document image to analyze pixels and determine position embedding vectors corresponding to words. The tagging system may then combine the position embedding vector with the corresponding word vector for use as input to a neural network trained to generate document tags. This combination may embed the position information into the corresponding word information in a serialized manner for processing by the document tagging neural network. Using this formatting, the tagging system may generate document tags in a lightweight and fast manner while still preserving the spatial relationships between words.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to systems, methods, and non-transitory computer-readable devices for document processing. Background Art

[0002] Optical character recognition (OCR) has been used to identify characters in documents. In addition, machine learning algorithms have been developed to identify characters and words in document images based on applied OCR techniques. For example, characters, words, or symbols can be extracted from document images. However, these techniques may not retain positional information associated with the symbols. For example, these techniques may not consider the layout of document symbols. The lack of such positional information may cause difficulties in analyzing the content of the document. If this positional information is not considered during document processing, even document data files that include text and positional information may produce inaccurate and / or incorrect results. Summary of the invention

[0003] At least one embodiment of the present disclosure provides a computer-implemented method. The method may include: receiving a document image; identifying one or more tokens of the document image; applying a first neural network to the document image to identify a position embedding sequence corresponding to the one or more tokens; applying a second neural network to one or more combinations of the one or more tokens and corresponding position embeddings of the position embedding sequence; and generating one or more document tags in response to applying the second neural network.

[0004] At least one embodiment of the present disclosure also provides a system. The system may include a memory and at least one processor. The at least one processor is coupled to the memory and may be configured to: receive a document image; identify one or more tags of the document image; apply a first neural network to the document image to identify a position embedding sequence corresponding to the one or more tags; apply a second neural network to one or more combinations of the one or more tags and corresponding position embeddings of the position embedding sequence; and generate one or more document tags in response to applying the second neural network.

[0005] At least one embodiment of the present disclosure also provides a non-transitory computer-readable device. The non-transitory computer-readable device has instructions stored thereon, which, when executed by at least one computing device, can cause the at least one computing device to perform operations, which operations may include: receiving a document image; identifying one or more tags of the document image; applying a first neural network to the document image to identify a position embedding sequence corresponding to the one or more tags; applying a second neural network to one or more combinations of the one or more tags and corresponding position embeddings of the position embedding sequence; and generating one or more document tags in response to applying the second neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings are incorporated herein and constitute a part of the specification.

[0007] Figure 1A A block diagram of a document processing environment is depicted in accordance with some embodiments.

[0008] Figure 1B A block diagram of a document processing flow is depicted in accordance with some embodiments.

[0009] Figure 2 Depicted are example network equations according to some embodiments.

[0010] Figure 3 A flow chart illustrating a method for generating document tags according to some embodiments is depicted.

[0011] Figure 4 Example serialization techniques in accordance with some embodiments are depicted.

[0012] Figure 5 An example computer system is depicted that facilitates implementing various embodiments.

[0013] In the drawings, like reference numbers generally refer to the same or similar elements. Additionally, the leftmost digit(s) of a reference number generally identifies the drawing in which the reference number first appears. DETAILED DESCRIPTION

[0014] Provided herein are systems, devices, equipment, methods and / or computer program product embodiments, and / or combinations and sub-combinations thereof, for generating document tags using position embedding. Embodiments disclosed herein can analyze document images to identify position parameter sequences of symbols or tokens of document images. These position parameters can be used to retain layout information of tags in document images during tag generation, and can provide increased accuracy. For example, if the tag generation process uses a serialized machine learning or artificial intelligence format, the position parameters can be embedded with the symbols to retain position information. Using these embeddings, labels can be generated using position information to achieve higher accuracy in the case of an accelerated learning process.

[0015] In an embodiment, a tagging system may receive a document and / or a document image and generate tags using position embedding. Tags may identify specific values ​​and / or symbols from the document and / or document image used when processing the document. For example, a document may be an invoice, and a tag may be generated by identifying a symbol or mark representing the total amount of the invoice based on document processing. Tags may also identify key-value pairs, named entities, and / or other semantic information from the document. The tagging system may generate these tags and / or identify specific symbols or marks using position information to provide increased accuracy.

[0016] Positional embedding information can help with document analysis. For example, some information can produce patterns based on word location. To illustrate an example embodiment, a document may be an invoice with a "First name" field. The input for this field may be "John", but may be arranged in a variety of ways. For example, the input may be arranged below the field label, to the right of the field label, or above the field label. These three configurations are shown below: (1)

[0018] First name

[0019] John (2)

[0021] First name: John (3)

[0023] John

[0024] First name

[0025] Analyzing several documents and the spatial positions of these input values ​​can help more accurately identify and label the correct value. By using position embeddings, such as horizontal and / or vertical positions, machine learning models can better identify values ​​and analyze documents.

[0026] However, a problem with some machine learning models can be the use of serialized tokens. For example, some natural language processing techniques can analyze documents as one-dimensional sequences of characters or words. This type of processing can result in the loss of two-dimensional information and / or relationships between words or tokens in the document.

[0027] In this way, the labeling system described herein can perform position embedding to retain position information. The position embedding can retain the two-dimensional coordinates corresponding to the position of the word or mark of the document. In some embodiments, the labeling system can receive a document and / or a document image as a data file. The labeling system can identify the grouping of characters, words and / or other characters as the mark of the document. For example, the labeling system can perform an optical character recognition (OCR) process to identify characters and / or words. In some embodiments, the document can be a data file containing formatted text, such as HTML, Word text, plain text, formatted text and / or other text data files. In this case, the labeling system can skip the OCR process and directly identify text and / or position information from the document data file. For example, the labeling system can use a file reader or converter.

[0028] The labeling system may also use a first neural network to identify position embeddings corresponding to characters and / or words. The first neural network may identify a sequence of position embeddings. The sequence may be combined with a sequence of input word vectors and applied to a second neural network. The second neural network may generate labels based on a combination of the word vectors and the position embedding vectors. As will be further explained below, the first neural network and the second neural network may be subnetworks of the combined neural network. Similarly, the first neural network and the second neural network may be trained jointly or sequentially.

[0029] By combining position embeddings with vector values ​​of words, the labeling system can retain position information when generating labels. When generating labels, the labeling system can utilize position information in the second neural network to generate more accurate results. This configuration can further retain position information for use even when the second neural network uses one-dimensional and / or sequential formatting. For example, this configuration can extract data from a table and retain the table organization. In this way, the two-dimensional information of the document can be retained and utilized even in a one-dimensional language model or neural network. This configuration can also be less computer resource intensive and can be more efficient when training machine learning models. This process can speed up the machine learning process and produce higher accuracy. In addition, the neural network configuration can use fewer layers to allow less resource-intensive processing. In this way, when classifying characters and / or words of a document, the configuration can be lightweight and fast while still capturing the position embedding of each character or word.

[0030] Various embodiments of these features will now be discussed with reference to the corresponding drawings.

[0031] Figure 1A A block diagram of a document processing environment 100A according to some embodiments is depicted. The document processing environment 100A may include a labeling system 110, a downstream processing system 130, and a document 120. The document 120 may include handwritten and / or typed text. The document 120 may be an image of a document and / or other type of data file with characters that can be recognized. The document 120 may include characters and / or groupings of characters. In some embodiments, the document 120 may include different languages ​​and / or symbols representing different alphabets.

[0032] The tagging system 110 can receive the document 120 and generate position embeddings and / or tags to identify the value of the document 120. The tagging system 110 can include a position vector network 112, a tag processor 114, and / or a tag network 116 for processing the document 120. The tag processor 114 can manage the position vector network 112 and / or the tag network 116. The tag processor 114 can include one or more processors and / or memories configured to implement a neural network or machine learning algorithm. The position vector network 112 can be a neural network and / or other machine learning model configured to recognize the position embeddings of characters, words, symbols, and / or tokens of the document 120. The tag network 116 can use these position embeddings and / or word vector values ​​to generate tags. The tag processor 114 can control this process.

[0033] In an embodiment, the labeling system 110 may receive the document 120 from an external source (such as a transmission from an external computer), and / or may retrieve the document 120 from an accessible memory. The label processor 114 may analyze the document 120 to identify characters and / or character groups (such as words). The label processor 114 may perform an optical character recognition (OCR) process to recognize the characters. Based on the programming of the label processor 114, the label processor 114 may recognize the character grouping as a symbol or mark for labeling. For example, the label processor 114 may recognize a word as a mark. As will be described in detail in the reference to FIG. Figure 2 To further explain, the tag processor 114 may further convert “w i ” is identified as a sequence of input word vectors. For example, each word vector can represent a word of document 120.

[0034] In addition to identifying a tag for a document 120, the tagging system 110 may use the position vector network 112 to identify a position embedding vector corresponding to the tag. Figure 2 To further explain, the position vector network 112 can be i” is identified as a sequence of position embedding vectors. In an embodiment, position vector network 112 may determine the position embeddings using a two-dimensional image-based or character grid-based convolutional neural network. For example, a layer (such as the last layer) of a convolutional neural network may be serialized into a one-dimensional, vector sequence to determine the two-dimensional position embeddings. To identify these position embeddings, position vector network 112 may analyze the pixels of document 120. Position vector network 112 may operate as an encoder of two-dimensional position embeddings to identify complex two-dimensional position embeddings that may change according to the text layout of document 120.

[0035] In an embodiment, the position vector network 112 may perform serialization by mapping pixels and / or position embeddings using a space-filling curve, such as a Morton or Hilbert-Lebesgue curve. Figure 4 Depicted are serialization techniques according to some embodiments. For example, a Morton curve 410 and / or a Hilbert-Lebesgue curve 420 may be used to generate a serialization of a sequence of two-dimensional embeddings. In some embodiments, the Hilbert-Lebesgue curve 420 may provide better spatial locality. Although the accompanying drawings may depict a feature map channel, each element may correspond to a vector. The vector may be a position embedding. As will be further explained below, a serialization or space filling curve may be used as the last encoding layer of the position vector network 112 to identify the position embedding.

[0036] During training or operation, the position vector network 112 may be operated to identify position embeddings of characters, words, and / or tokens of the document 120. In this case, the last feature layer of the position vector network 112 may utilize a padding mask to obtain a sufficient number of pixels to map the sequence of tokens. In the event that the sequence is shorter than the number of available position embeddings, the position vector network 112 may use the padding mask to perform deep learning and / or natural language processing.

[0037] After identifying the position embedding, the tag processor 114 may combine the position embedding with the input word vector as the input to the tag network 116. For example, for each input word vector “w i "Identification position embedding vector" x i These position embedding vectors can map information about the position of a particular tag of a two-dimensional document into a higher dimensional space. The dimension can be the same as or different from the dimension of the input word vector. The tag processor 114 can combine and / or append the position embedding vector with the input word vector and provide the combination as input to the tag network 116. In this way, the combination can preserve the two-dimensional layout information within the tag generation process.

[0038] The label network 116 can be a transformer neural network that can generate a sequence of labels as an output. In some embodiments, the label network 116 can be a neural network that does not retain or rely on the spatial positionality and / or spatial ordering of the label. As previously described, the label network 116 can generate labels using a sequence of two-dimensional embedding vectors combined with each word vector. The label network 116 can have been pre-trained to identify labels based on word vector values ​​and position embedding. In the case where the label network 116 is a sequence labeling machine learning model based on a converter, each element of the sequence (such as a word or character) can be compared with each other element in each layer of the label network 116. In this way, this comparison can identify the disordered relationship between any elements of the input sequence. By including position embedding information when using these converter-based systems, the spatial relationship between elements can be retained. For example, the spatial relationship between words can be retained. This information can further help identify and generate accurate labels.

[0039] Although the position vector network 112 and the label network 116 are depicted as separate neural networks in some embodiments, they can be implemented using a single neural network. Similarly, the position vector network 112 and the label network 116 can be trained jointly or sequentially. In order to train the networks separately, the labeling system 110 can freeze the position vector network 112 when training the label network 116. In some embodiments, the position vector network 112 can be pre-trained by predicting the sequence of words included in the document 120. In this way, the labeling system 110 can train different sub-networks or train a joint neural network to recognize position embeddings and / or labels. Similarly, when analyzing a document, the labeling system 110 can use either configuration.

[0040] Reference now Figure 2 , Figure 2 Depicted are network equations 210, 220, and 230 in accordance with some embodiments. These network equations 210-230 further illustrate the operation of the tagging system 110, the position vector network 112, and the tagging network 116.

[0041] The position vector network 112 may use the position vector equation 220 to determine the position embedding vector “x i ". This value can be the position embedding vector of the i-th word in the document 120. The function "g" can represent a two-dimensional convolutional encoder network based on a two-dimensional image or a character grid. As mentioned above, the layers of the network can be serialized into a one-dimensional vector sequence. For example, the last layer of the position vector network 112 can be serialized to generate the position embedding. In the position vector equation 220, the operand of the function can be an "I" that can represent a pixel of the document 120.kl ”. The operands may represent the kth and lth pixels in a document. The pixels may be character and / or word pixels of a document image and / or a character grid. By applying the function to each group of pixels, the position vector network 112 may determine a position embedding vector for each character, word, or token of the document 120.

[0042] The label prediction equations 210 and 230 can be used to generate the label “ i ". The tag network 116 can be a transformer network using the tag prediction equation 210 and the tag prediction equation 230. The tag prediction equations 210 and 230 can use the function "f" to represent the transformer network. The operand of the transformer network can be a combination of a tag vector and a position embedding vector. In some embodiments, the tag vector can be a word vector "w i ”, and the position embedding vector may be “w i ”. The combination can be generated by appending the position embedding vector to each word vector representing each term of the document 120. In this way, the label network 116 can generate labels using the position embedding and word vectors output from the position vector network 112 as shown in the label prediction equation 210 and the label prediction equation 230.

[0043] As previously described, the position vector network 112 and the label network 116 can be separate neural networks managed by the label processor 114. The label prediction equations 210 and 230 can illustrate the interaction between the position vector network 112 and the label network 116. For example, as shown in the label prediction equation 230, the label network 116 can merge the output from the position vector network 112. In some embodiments, the position vector network 112 and the label network 116 can be implemented using the same neural network. In this case, the common neural network can implement the label prediction equation 230 and perform pixel analysis to identify the position embedding. The position embedding can then be used to generate a label.

[0044] Back to Figure 1A After the tagging system 110 has generated tags, the tagging system 110 can store the tags and / or utilize the tags in downstream document processing tasks. For example, the tagged version of the document 120 can be stored in a database for indexing and / or archiving. The tag values ​​can be extracted and / or further manipulated in downstream processing. In some embodiments, the tagging system 110 can transmit the tags and / or the document 120 to a downstream processing system 130 for further processing. The downstream processing system 130 can be a system external to the tagging system 110 and can be connected via a wired or wireless connection.

[0045] To illustrate an example embodiment of tag generation, we will now discuss Figure 1B . Figure 1B A block diagram of a document processing flow 100B according to some embodiments is depicted. Document 120 may be a data file of a document image. The document image may depict an invoice having text characters "Invoice" and "Total: $100". In an embodiment, labeling system 110 may apply an optical character recognition (OCR) process to recognize characters of document 120. In some embodiments, for example, document 120 may be a data file containing formatted text, such as HTML, Word text, plain text, formatted text, and / or other text data files. In this case, labeling system 110 may skip the OCR process and recognize text and / or location information directly from document 120. For example, labeling system 110 may use a file reader or converter.

[0046] The tagging system 110 can also identify tokens (such as words) from the document 120 by identifying groupings of characters. For example, the tagging system 110 can group characters based on identifying spaces between groups of characters based on pixel analysis. These groupings can be words that correspond to word vectors used during the tag generation process. For example, the word "Invoice" can be represented as word vector "w1", while "Total:" and "$100" can be represented as word vectors "w2" and "w3", respectively. For these word vectors, the tagging system 110 can identify position embedding vectors.

[0047] To identify the position embedding vector, the tagging system 110 can apply an OCR process to obtain a sequence of word tags. This can generate a tokenized document 140. The tokenized document 140 can have C channels, a height H, and a weight W. The tag processor 114 can generate the tokenized document 140 and pass the tokenized document 140 to the position vector network 112. The position vector network 112 can be a two-dimensional encoder network, such as, for example, a deep convolutional neural network. The neural network can generate a vector 150. The vector 150 can then be serialized for combination with a word vector. As shown in FIG. Figure 4 As depicted and described, the encoder feature graph of the network can be serialized using a Morton curve 410 or a Hilbert-Lebesgue curve 420. This analysis can be performed for each word to identify a corresponding positional embedding vector and produce a sequence of two-dimensional embeddings. The serialization curve can "unroll" the vector 150 and / or flatten the feature graph in a way that preserves the positionality of the elements. This positionality can help preserve the relative distance of words, and whether words are close or far from other words.

[0048] The tag processor 114 may then embed the serialized position into the vector "x i ” and word vector “w i ” combination. In some embodiments, the position embedding vector can be attached to the word vector. These combinations 160 can be generated for each token or word of the document 120. For example, for the token "$100", the corresponding combination can be "w3+x3". The combination 160 can then be applied to the labeling network 116. As previously described, the labeling network 116 can be a transformer network configured to recognize tags.

[0049] The tag network 116 can be trained to recognize one or more tags. For example, the tag network 116 may have been previously trained to recognize the total amount of an invoice based on training data using position embeddings. In this way, the tag network 116 can output a tag sequence 170 indicating a specific category description as a tag. If the total amount is the third word identified as "w3", the tag can identify the third word as the desired total amount category. The tag system 110 can then use the tag to extract the value for additional document processing. In some embodiments, tagging can help extract values ​​and / or classify documents 120. Depending on the configuration of the tag network 116, other words can be given null or zero values ​​in the tag sequence 170, where null or zero values ​​can indicate that other words do not belong to the desired total amount category. In some embodiments, each word can be given a tag based on its classification.

[0050] The tagging system 110 may also perform process 100B to perform translation of words into different languages. For example, the tagging system 110 may be applied to translations where the languages ​​have characters or words of different lengths. The tagging system 110 may not require a one-to-one translation, but may adapt to different sequence lengths between different languages. The generated tags and / or extracted information may be used to translate the document 120. Similarly, tag generation may help summarize two-dimensional documents. Identification of tags may provide quick summarization by identifying key values.

[0051] Figure 3 A flow chart illustrating a method 300 for generating document tags according to some embodiments is depicted. Figure 1A Method 300 is described; however, method 300 is not limited to this example embodiment.

[0052] In an embodiment, the tagging system 110 may use one or more neural networks to analyze the document 120 to identify the document tag. Although the method 300 is described with reference to the tagging system 110, the method 300 may be implemented on any computing device (such as, for example, a computer). Figure 5The described computer systems and / or processing logic may include hardware (eg, circuitry, dedicated logic, programmable logic, microcode, etc.), software (eg, instructions executed on a processing device), or a combination thereof.

[0053] It should be understood that not all steps are required to perform the disclosure provided herein. In addition, as will be appreciated by those of ordinary skill in the art, some steps may be performed simultaneously or in different order. Figure 3 Execute in the order shown.

[0054] At 305, the labeling system 110 may receive a document image. The document image may be a document 120. The labeling system 110 may access the document image from a storage location in a computer memory and / or may receive the document image from an external computer system. The document image may be a text, document, and / or image data file. The document image may include handwritten and / or typed text.

[0055] At 310, the tagging system 110 may identify one or more tags of the document image. To identify the one or more tags, the tagging system 110 may apply an optical character recognition (OCR) service. The OCR service may provide a mapping to numeric index values ​​for characters detected in the document image. The OCR service may perform pixel analysis to determine these character values. In some embodiments, the OCR service may identify tags as character groupings. For example, tags may be words, phrases, sentences, paragraphs, or other character organizations. The tagging system 110 may identify tags from one or more languages. In some languages, characters may be designated as tags, such as, for example, Chinese. The tagging system 110 may identify these tags to generate one or more word vectors "w i ”.

[0056] In some embodiments, the document image may be a data file containing formatted text, such as, for example, HTML, Word text, plain text, formatted text, and / or other text data files. In this case, the labeling system 110 may skip the OCR process and identify the text and / or location information directly from the document data file. For example, the labeling system 110 may use a file reader or converter. The labeling system 110 may identify one or more tags directly from the document data file.

[0057] At 315, the tagging system 110 may apply a first neural network to the document image to identify a sequence of position embeddings corresponding to the tags. The first neural network may be a position vector network 112. As previously described above, the first neural network may identify position embeddings for words of the document image. The position embeddings may be generated based on pixel recognition performed by a two-dimensional encoder network or a deep convolutional neural network. The feature map generated by the encoder may be serialized to produce a two-dimensional position embedding sequence corresponding to the tags of the document image. The serialization may be performed using a Morton curve, a Hilbert-Lebesgue curve, and / or other serialization methods to preserve spatial positionality. This flattening may produce the position embedding sequence.

[0058] At 320, the tagging system 110 may apply a second neural network to one or more combinations of the one or more tags and corresponding position embeddings of the position embedding sequence. The second neural network may be the tagging network 116. The combination of the one or more tags and corresponding position embeddings may include appending the position embedding vectors to their corresponding tags identified from the document image. This configuration may arrange the one or more combinations into a one-dimensional sequence for analysis by the tagging network 116. The tagging network 116 may then process the combined sequence to identify tags based on the configuration and previous training of the tagging network 116.

[0059] At 325, the tagging system 110 may generate one or more document tags. These document tags may be generated by the tagging network 116 based on a combination of one or more tags and corresponding position embeddings. The document tag may identify a specific value for data extraction from a document image, for document classification, and / or for other document processing. In some embodiments, the document tag may identify an expected value. In some embodiments, document tags may be provided for multiple tags of a document image to provide additional document tags. The tagging system 110 may use and / or store these tags with the document image for further document processing. In some embodiments, the tagging system 110 may transmit the one or more document tags to a downstream processing system 130. The downstream processing system 130 may be an external system that performs additional processing on the document image using the one or more document tags.

[0060] For example, one or more well-known computer systems (such as Figure 5 The computer system 500 shown in the figure implements various embodiments. For example, one or more computer systems 500 can be used to implement any of the embodiments discussed herein, as well as combinations and sub-combinations thereof.

[0061] Computer system 500 may include one or more processors (also referred to as central processing units or CPUs), such as processor 504. Processor 504 may be connected to a communications infrastructure or bus 506.

[0062] The computer system 500 may also include user input / output device(s) 503 (such as a monitor, keyboard, pointing device, etc.) that may communicate with the communication infrastructure 506 through the user input / output interface(s) 502 .

[0063] One or more of the processors 504 may be a graphics processing unit (GPU). In an embodiment, a GPU may be a processor that is a dedicated electronic circuit designed to process arithmetic-intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as arithmetic-intensive data commonly found in computer graphics applications, images, videos, etc.

[0064] The computer system 500 may also include a main memory or primary storage 508, such as a random access memory (RAM). The main memory 508 may include one or more levels of cache. The main memory 508 may have stored therein control logic (ie, computer software) and / or data.

[0065] The computer system 500 may also include one or more secondary storage devices or secondary memories 510. The secondary memories 510 may include, for example, a hard disk drive 512 and / or a removable storage device or drive 514. The removable storage drive 514 may be a floppy disk drive, a tape drive, an optical drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0066] Removable storage drive 514 can interact with removable storage unit 518. Removable storage unit 518 may include a computer usable or readable storage device on which computer software (control logic) and / or data has been stored. Removable storage unit 518 may be a floppy disk, a magnetic tape, a compact disk, a DVD, an optical storage disk, and / or any other computer data storage device. Removable storage drive 514 can read from and / or write to removable storage unit 518.

[0067] Secondary memory 510 may include other devices, apparatuses, components, means, or other methods that allow computer system 500 to access computer programs and / or other instructions and / or data. Such devices, apparatuses, components, means, or other methods may include, for example, a removable storage unit 522 and an interface 520. Examples of removable storage unit 522 and interface 520 may include a program cartridge and cartridge interface (such as found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and / or any other removable storage unit and associated interface.

[0068] The computer system 500 may further include a communication or network interface 524. The communication interface 524 may enable the computer system 500 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference numeral 528). For example, the communication interface 524 may allow the computer system 500 to communicate with an external or remote device 528 via a communication path 526, which may be wired and / or wireless (or a combination thereof) and may include any combination of a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from the computer system 500 via the communication path 526.

[0069] The computer system 500 may also be any of a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet computer, a smart phone, a smart watch or other wearable device, part of the Internet of Things, and / or an embedded system, to name a few non-limiting examples, or any combination thereof.

[0070] Computer system 500 can be a client or server accessing or hosting any application and / or data through any delivery paradigm, where the delivery paradigms include, but are not limited to: remote or distributed cloud computing solutions; local or on-premise software ("on-premise" cloud-based solutions); "as a service" models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (SaaS), managed software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and / or hybrid models including any combination of the foregoing examples or other services or delivery paradigms.

[0071] Any applicable data structures, file formats, and schemas in computer system 500 may be derived from standards, including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representation, alone or in combination. Alternatively, proprietary data structures, formats, or schemas may be used alone or in combination with known or open standards.

[0072] In some embodiments, a tangible, non-transitory apparatus or article of manufacture including a tangible, non-transitory computer-usable or readable medium having stored thereon control logic (software) may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 500, main memory 508, auxiliary memory 510, and removable storage units 518 and 522, and tangible articles of manufacture embodying any combination of the foregoing. When executed by one or more data processing devices (such as computer system 500), such control logic may cause such data processing devices to operate as described herein.

[0073] Based on the teachings contained in this disclosure, it is obvious to a person skilled in the relevant art how to use the Figure 5 It will be apparent that data processing devices, computer systems, and / or computer architectures other than those shown can be used to implement and use embodiments of the present disclosure. In particular, the embodiments can operate with software, hardware, and / or operating system implementations other than those described herein.

[0074] It should be understood that the detailed description section, but not any other section, is intended to be used to interpret the claims. The other sections may set forth one or more but not all example embodiments contemplated by the inventor(s), and therefore, are not intended to limit the present disclosure or the appended claims in any way.

[0075] Although the present disclosure describes example embodiments of example fields and applications, it should be understood that the present disclosure is not limited thereto. Other embodiments and modifications thereto are possible and within the scope and spirit of the present disclosure. For example, without limiting the generality of this paragraph, the embodiments are not limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. In addition, the embodiments (whether or not explicitly described herein) have significant utility for fields and applications outside the examples described herein.

[0076] Embodiments have been described herein by means of functional building blocks that illustrate the implementation of specific functions and relationships thereof. For ease of description, the boundaries of these functional building blocks are arbitrarily defined herein. As long as the specified functions and relationships (or their equivalents) are properly performed, alternative boundaries may be defined. In addition, alternative embodiments may use sequential execution function blocks, steps, operations, methods, etc. different from those described herein.

[0077] References to "one embodiment", "embodiment", "example embodiment" or similar phrases herein indicate that the described embodiments may include specific features, structures or characteristics, but each embodiment does not necessarily include specific features, structures or characteristics. In addition, such phrases do not necessarily refer to the same embodiment. In addition, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not explicitly mentioned or described herein, incorporating such features, structures or characteristics into other embodiments is within the knowledge of (multiple) technical personnel in the relevant field. In addition, some embodiments may be described using the expressions "coupled" and "connected" and their derivatives. These terms are not necessarily synonymous with each other. For example, some embodiments may be described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also represent that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0078] The breadth and scope of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method, include: receiving a document image; Recognize one or more marks of the document image, wherein the one or more marks are characters and / or words and / or other character groupings of the document image; applying a first neural network to the document image to identify a sequence of position embeddings corresponding to the one or more tokens, wherein the first neural network determines the position embeddings using a two-dimensional image-based or character grid-based convolutional neural network, and wherein a space filling curve is applied to a final layer of the convolutional neural network to serialize the position embedding vectors to identify the sequence of position embeddings; identifying a sequence of input word vectors, each input word vector representing a word of the document image; applying a second neural network to one or more combinations of the sequence of input word vectors and the sequence of position embedding vectors, wherein the one or more combinations are generated by appending each position embedding vector to a corresponding one of the input word vectors and the one or more combinations are used as input to the second neural network; and In response to applying the second neural network, generating one or more document labels based on the one or more combinations, The one or more combinations are arranged in a one-dimensional sequence so as to be used by the second neural network to identify one or more document tags based on the one-dimensional sequence.

2. The computer-implemented method of claim 1 , wherein the first neural network and the second neural network are subnetworks of a single neural network.

3. A system, include: Memory; and at least one processor coupled to the memory and configured to: receiving a document image; Recognize one or more marks of the document image, wherein the one or more marks are characters and / or words and / or other character groupings of the document image; applying a first neural network to the document image to identify a sequence of position embeddings corresponding to the one or more tokens, wherein the first neural network determines the position embeddings using a two-dimensional image-based or character grid-based convolutional neural network, and wherein a space filling curve is applied to a final layer of the convolutional neural network to serialize the position embedding vectors to identify the sequence of position embeddings; identifying a sequence of input word vectors, each input word vector representing a word of the document image; applying a second neural network to one or more combinations of the sequence of input word vectors and the sequence of position embedding vectors, wherein the one or more combinations are generated by appending each position embedding vector to a corresponding one of the input word vectors and the one or more combinations are used as input to the second neural network; and In response to applying the second neural network, generating one or more document labels based on the one or more combinations, The one or more combinations are arranged in a one-dimensional sequence so as to be used by the second neural network to identify one or more document tags based on the one-dimensional sequence.

4. The system of claim 3, wherein the first neural network and the second neural network are sub-networks of a single neural network.

5. A non-transitory computer-readable device having instructions stored thereon, which when executed by at least one computing device causes the at least one computing device to perform operations, the operations include: receiving a document image; Recognize one or more marks of the document image, wherein the one or more marks are characters and / or words and / or other character groupings of the document image; applying a first neural network to the document image to identify a sequence of position embeddings corresponding to the one or more tokens, wherein the first neural network determines the position embeddings using a two-dimensional image-based or character grid-based convolutional neural network, and wherein a space filling curve is applied to a final layer of the convolutional neural network to serialize the position embedding vectors to identify the sequence of position embeddings; identifying a sequence of input word vectors, each input word vector representing a word of the document image; applying a second neural network to one or more combinations of the sequence of input word vectors and the sequence of position embedding vectors, wherein the one or more combinations are generated by appending each position embedding vector to a corresponding one of the input word vectors; and In response to applying the second neural network, generating one or more document labels based on the one or more combinations, The one or more combinations are arranged in a one-dimensional sequence so as to be used by the second neural network to identify one or more document tags based on the one-dimensional sequence.

Images