System, method, and computer program product for structured information extraction from documents

Abstract

Systems, methods, and computer program products are provided for structured information extraction from documents that may include an artificial intelligence (AI)-driven system for extracting structured information from documents that have information requested but do not follow a pre-defined template. The system may reduce a number of false positive and false negative errors by combining outputs of an image segmentation model and an optical character recognition (OCR) model. The system may include an approach for reducing large language model (LLM) hallucinations by using a fact-checking approach and / or auto-correcting OCR errors by using context of data fields.

Description

Attorney Docket No. 08223-2502439 (9305W001)SYSTEM, METHOD, AND COMPUTER PROGRAM PRODUCT FOR STRUCTURED INFORMATION EXTRACTION FROM DOCUMENTSCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 680,655 filed August 8, 2024, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND1 . Technical Field

[0002] This disclosure relates generally to information extraction from documents and, in non-limiting embodiments or aspects, to systems, methods, and computer program products for structured information extraction from documents using generative artificial intelligence (Al) or machine learning.2. Technical Considerations

[0003] In an exchange, such as a payment exchange, there is often a need to extract structured information from documents that do not follow a pre-defined template.SUMMARY

[0004] Accordingly, provided are improved systems, methods, and computer program products for structured information extraction from documents.

[0005] According to some non-limiting embodiments or aspects, provided is a system, including: at least one processor configured to: obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first66E6779.DOCX Page 1 of 56Attorney Docket No. 08223-2502439 (9305W001) confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second subbounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0006] In some non-limiting embodiments or aspects, the at least one processor is further configured to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text.

[0007] In some non-limiting embodiments or aspects, the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one66E6779.DOCX Page 2 of 56Attorney Docket No. 08223-2502439 (9305W001) merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

[0008] In some non-limiting embodiments or aspects, the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: anonymize, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: de-anonymize, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

[0009] In some non-limiting embodiments or aspects, the at least one processor is further configured to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof.

[0010] In some non-limiting embodiments or aspects, the at least one processor is configured to extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair; the one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof is determined based on at least one previous key-value pair used to extract, using at least one LLM, at least one previous data element from at least one previous document, and wherein each of the66E6779.DOCX Page 3 of 56Attorney Docket No. 08223-2502439 (9305W001) at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0011] In some non-limiting embodiments or aspects, the at least one processor is configured to generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first subbounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.66E6779.DOCX Page 4 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0012] According to some non-limiting embodiments or aspects, provided is a method including: obtaining, with at least one processor, at least one image associated with at least one page of at least one document; generating, with the at least one processor, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first OCR technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; applying, with the at least one processor, at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generating, with the at least one processor, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extracting, with the at least one processor, using at least one LLM, based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0013] In some non-limiting embodiments or aspects, the method further includes: receiving, with the at least one processor, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determining, with the at least one processor, whether the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response66E6779.DOCX Page 5 of 56Attorney Docket No. 08223-2502439 (9305W001) to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verifying, with the at least one processor, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete, with the at least one processor, the at least one data element extracted from the merged text.

[0014] In some non-limiting embodiments or aspects, the method further includes: before using the at least one LLM to extract the at least one data element: identifying, with the at least one processor, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determining, with the at least one processor, whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintaining, with the at least one processor, the at least section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, removing, with the at least one processor, the at least one section from the at least one document.

[0015] In some non-limiting embodiments or aspects, the method further includes: before using the at least one LLM to extract the at least one data element: anonymizing, with the at least one processor, using a PI tokenization technique, personal information included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: deanonymizing, with the at least one processor, using the PI tokenization technique, the66E6779.DOCX Page 6 of 56Attorney Docket No. 08223-2502439 (9305W001) personal information in the merged text included in the at least one data element extracted from the merged text.

[0016] In some non-limiting embodiments or aspects, the method further includes: automatically modifying, with the at least one processor, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text, the at least one data element, or any combination thereof.

[0017] In some non-limiting embodiments or aspects, extracting, with the at least one processor, using the at least one LLM, the at least one data element includes providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair, one or more expected formats for at least one value of the at least one key-value pair, or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair, the one or more expected formats for the at least one value of the at least one key-value pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0018] In some non-limiting embodiments or aspects, generating, with the at least one processor, the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, includes: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the66E6779.DOCX Page 7 of 56Attorney Docket No. 08223-2502439 (9305W001) first sub-bounding boxes combined therein; in response to the size of the first subbounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0019] According to some non-limiting embodiments or aspects, provided is a computer program product including at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first OCR technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and first text associated with the at least first one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated66E6779.DOCX Page 8 of 56Attorney Docket No. 08223-2502439 (9305W001) with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one LLM, based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0020] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one keyvalue pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text.

[0021] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated66E6779.DOCX Page 9 of 56Attorney Docket No. 08223-2502439 (9305W001) with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

[0022] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: before using the at least one LLM to extract the at least one data element: anonymize, using a PI tokenization technique, personal information included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: de-anonymize, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

[0023] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof, wherein the program instructions, when executed by the at least one processor, cause the at least one processor to extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair, the one or more expected formats for the at least one value of the at least one key-value pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0024] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, cause the at least one processor to66E6779.DOCX Page 10 of 56Attorney Docket No. 08223-2502439 (9305W001) generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first subbounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first subbounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0025] Further non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0026] Clause 1 : A system, comprising: at least one processor configured to: obtain at least one image associated with at least one page of at least one document;66E6779.DOCX Page 1 1 of 56Attorney Docket No. 08223-2502439 (9305W001) generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second subbounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0027] Clause 2: The system of clause 1 , wherein the at least one processor is further configured to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds66E6779.DOCX Page 12 of 56Attorney Docket No. 08223-2502439 (9305W001) to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text.

[0028] Clause 3: The system of clause 1 or 2, wherein the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

[0029] Clause 4: The system of any of clauses 1 -3, wherein the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: anonymize, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: deanonymize, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

[0030] Clause 5: The system of any of clauses 1 -4, wherein the at least one processor is further configured to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof.

[0031] Clause 6: The system of any of clauses 1 -5, wherein the at least one processor is configured to extract, using the at least one LLM, the at least one data66E6779.DOCX Page 13 of 56Attorney Docket No. 08223-2502439 (9305W001) element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair; the one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof is determined based on at least one previous key-value pair used to extract, using at least one large language model (LLM), at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0032] Clause 7: The system of any of clauses 1 -6, wherein the at least one processor is configured to generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second subbounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first subbounding box with that first sub-bounding box until a size of the combined first subbounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined66E6779.DOCX Page 14 of 56Attorney Docket No. 08223-2502439 (9305W001) therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second subbounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0033] Clause 8: A method, comprising: obtaining, with at least one processor, at least one image associated with at least one page of at least one document; generating, with the at least one processor, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first subbounding box is associated with a first confidence score; applying, with the at least one processor, at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generating, with the at least one processor, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extracting, with the at least one processor, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.66E6779.DOCX Page 15 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0034] Clause 9: The method of clause 8, further comprising: receiving, with the at least one processor, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determining, with the at least one processor, whether the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verifying, with the at least one processor, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one keyvalue pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, deleting, with the at least one processor, the at least one data element extracted from the merged text.

[0035] Clause 10: The method of clause 8 or 9, further comprising: before using the at least one LLM to extract the at least one data element: identifying, with the at least one processor, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determining, with the at least one processor, whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintaining, with the at least one processor, the at least section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, removing, with the at least one processor, the at least one section from the at least one document.66E6779.DOCX Page 16 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0036] Clause 1 1 : The method of any of clauses 8-10, further comprising: before using the at least one LLM to extract the at least one data element: anonymizing, with the at least one processor, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: de-anonymizing, with the at least one processor, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

[0037] Clause 12: The method of any of clauses 8-1 1 , further comprising: automatically modifying, with the at least one processor, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text, the at least one data element, or any combination thereof.

[0038] Clause 13: The method of any of clauses 8-12, wherein extracting, with the at least one processor, using the at least one LLM, the at least one data element includes providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair, one or more expected formats for at least one value of the at least one key-value pair, or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair, the one or more expected formats for the at least one value of the at least one key-value pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using at least one large language model (LLM), at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0039] Clause 14: The method of any of clauses 8-13, wherein generating, with the at least one processor, the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, includes: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size66E6779.DOCX Page 17 of 56Attorney Docket No. 08223-2502439 (9305W001) of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first subbounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0040] Clause 15: A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text66E6779.DOCX Page 18 of 56Attorney Docket No. 08223-2502439 (9305W001) associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second subbounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0041] Clause 16: The computer program product of clause 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, deleting the at least one data element extracted from the merged text.

[0042] Clause 17: The computer program product of clause 15 or 16, wherein the program instructions, when executed by the at least one processor, further cause the66E6779.DOCX Page 19 of 56Attorney Docket No. 08223-2502439 (9305W001) at least one processor to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

[0043] Clause 18: The computer program product of any of clauses 15-17, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: before using the at least one LLM to extract the at least one data element: anonymize, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: deanonymize, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

[0044] Clause 19: The computer program product of any of clauses 15-18, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof, wherein the program instructions, when executed by the at least one processor, cause the at least one processor to extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair,66E6779.DOCX Page 20 of 56Attorney Docket No. 08223-2502439 (9305W001) the one or more expected formats for the at least one value of the at least one keyvalue pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

[0045] Clause 20: The computer program product of any of clauses 15-19, wherein the program instructions, when executed by the at least one processor, cause the at least one processor to generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second subbounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first subbounding box with that first sub-bounding box until a size of the combined first subbounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second subbounding box or the combined second sub-bounding boxes within a threshold66E6779.DOCX Page 21 of 56Attorney Docket No. 08223-2502439 (9305W001) tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0046] These and other features and characteristics of the present disclosure, as well as the methods of operation and functions of the related elements of structures and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Additional advantages and details are explained in greater detail below with reference to the non-limiting, exemplary embodiments that are illustrated in the accompanying schematic figures, in which:

[0048] FIG. 1 is a schematic diagram of an electronic payment processing network, according to some non-limiting embodiments or aspects;

[0049] FIG. 2 is a schematic diagram of example components of one or more devices of FIG. 1 , according to some non-limiting embodiments or aspect;

[0050] FIGS. 3A-3C are flow diagrams of a method for structured information extraction from documents, according to some non-limiting embodiments or aspects;

[0051] FIG. 4 is a block diagram of components of a system for structured information extraction from documents, according to some non-limiting embodiments or aspects;

[0052] FIG. 5 depicts an example of tables and text columns identified in a document, according to some non-limiting embodiments or aspects; and

[0053] FIG. 6 depicts an example of merging outputs from an image segmentation technique and an optical character recognition (OCR) technique, according to some non-limiting embodiments or aspects.66E6779.DOCX Page 22 of 56Attorney Docket No. 08223-2502439 (9305W001)DETAILED DESCRIPTION

[0054] For purposes of the description hereinafter, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and derivatives thereof shall relate to the embodiments as they are oriented in the drawing figures. However, it is to be understood that the present disclosure may assume various alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Hence, specific dimensions and other physical characteristics related to the embodiments or aspects disclosed herein are not to be considered as limiting.

[0055] Some non-limiting embodiments or aspects are described herein in connection with thresholds. As used herein, satisfying a threshold may refer to a value being greater than the threshold, more than the threshold, higher than the threshold, greater than or equal to the threshold, less than the threshold, fewer than the threshold, lower than the threshold, less than or equal to the threshold, equal to the threshold, etc.

[0056] No aspect, component, element, structure, act, step, function, instruction, and / or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more” and “at least one.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, and / or the like) and may be used interchangeably with “one or more” or “at least one.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based at least partially on” unless explicitly stated otherwise. In addition, reference to an action being “based on” a condition may refer to the action being “in response to” the condition. For example, the phrases “based on” and “in response to” may, in some non-limiting embodiments or aspects, refer to a condition for automatically triggering an action (e.g., a specific operation of an electronic device, such as a computing device, a processor, and / or the like).66E6779.DOCX Page 23 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0057] As used herein, the term “communication” may refer to the reception, receipt, transmission, transfer, provision, and / or the like of data (e.g., information, signals, messages, instructions, commands, and / or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and / or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and / or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and / or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) that includes data. It will be appreciated that numerous other arrangements are possible.

[0058] As used herein, the term “computing device” may refer to one or more electronic devices configured to process data. A computing device may, in some examples, include the necessary components to receive, process, and output data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or standard cellular phone), a portable computer, a wearable device (e.g., watches, glasses, lenses, clothing, and / or the like), a personal digital assistant (PDA), and / or other like devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0059] As used herein, the term “server” may refer to or include one or more computing devices that are operated by or facilitate communication and processing for multiple parties in a network environment, such as the Internet, although it will be appreciated that communication may be facilitated over one or more public or private network environments and that various other arrangements are possible. Further, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile66E6779.DOCX Page 24 of 56Attorney Docket No. 08223-2502439 (9305W001) devices, etc.) directly or indirectly communicating in the network environment may constitute a “system.”

[0060] As used herein, the term “system” may refer to one or more computing devices or combinations of computing devices (e.g., processors, servers, client devices, software applications, components of such, and / or the like). Reference to “a device,” “a server,” “a processor,” and / or the like, as used herein, may refer to a previously-recited device, server, or processor that is recited as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and the claims, a first device, a first server, or a first processor that is recited as performing a first step or a first function may refer to the same or different device, server, or processor recited as performing a second step or a second function.

[0061] As used herein, the term “real-time” refers to performance of a task or tasks during another process or before another process is completed. For example, a realtime inference may be an inference that is obtained from a model before a payment transaction is authorized, completed, and / or the like.

[0062] Documents processed by exchanges are often scanned-in images and have textual and tabular data. For example, in a payment network, there are issuer and acquirer banks representing a customer and a merchant, respectively, in a payment transaction. Even though a small percentage of transactions are disputed by the customer, a large payment network may see hundreds of thousands of disputes a day. Acquirers and issuers often exchange documents while they are resolving disputes via the payment network. The documents may contain information, such as receipts, transaction details, proof of delivery, etc.

[0063] A document receiver (e.g., an issuer system, etc.) may want to see the information in a structured format. However, a document provider (e.g., an acquirer system, etc.) may only have the information in image or text format.

[0064] Existing information extraction from documents may involve optical character recognition (OCR) and / or large language models (LLMs). Multiple libraries are available for the OCR that have different accuracy and latency characteristics. There are two main types of OCR errors - omissions, where the OCR fails to read information that is in the document, and recognition errors, where the OCR reads the text incorrectly or inserts characters that do not exist in the document (e.g., 1 is read as T, etc.) Additionally, the OCR libraries, in general, do not understand tables very66E6779.DOCX Page 25 of 56Attorney Docket No. 08223-2502439 (9305W001) well. For example, the OCR libraries may not understand column headings versus each row and may treat each row as an independent line of text. Hence, the information about the type of each cell in a table may not be captured by the OCR.

[0065] Multiple approaches for information extraction are possible, including using LLMs, Named Entity Recognition (NER) and Regular Expression, each of which have their pluses and minuses. LLMs do not require much training or fine-tuning, but can suffer from hallucinations (where the information presented by the LLM was not from the source document) and omission errors. Personal data going into the LLM may also need to be anonymized. Regular Expression and NER models do not have hallucination issues, but require documents to follow a rigid grammar, which is not practical in this setting. Documents not following the rigid grammar may be subject to false rejections.

[0066] Non-limiting embodiments or aspects of the present disclosure provide systems, methods, and computer program products for structured information extraction from documents that obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first OCR technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination66E6779.DOCX Page 26 of 56Attorney Docket No. 08223-2502439 (9305W001) thereof; and extract, using at least one LLM, based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0067] In this way, non-limiting embodiments or aspects of the present disclosure may provide improved structured information extraction from documents accurately and in real-time to save a receiver the effort of information extraction from the documents and that enables detecting tables in document images and representing the tables in a structured format, reducing errors during an OCR process by combining the output from at least two different types of OCR models, auto-correcting the OCR by using a context of the data field, and / or reducing LLM hallucinations by using a fact checking approach. Accordingly, non-limiting embodiments or aspects of the present disclosure may achieve high information extraction accuracy by reducing a number of false positive and false negative errors in the process of extracting information from image and text documents while meeting a processing response time requirement (e.g., of a few seconds, etc.). For example, false positive errors may include ‘hallucinations’, where the structured information sent to the receiver was not in the sender’s document and recognition errors (e.g., a ‘1 ’ in the document recognized as an T, etc.). False negative errors may arise when information in the document is not extracted, and the document is incorrectly rejected by the payment network for not having the required information.

[0068] Referring now to FIG. 1 , FIG. 1 shows electronic payment processing network 100, according to non-limiting embodiments or aspects. The payment processing network may be used in conjunction with the systems and methods described herein. It will be appreciated that the particular arrangement of electronic payment processing network 100 shown is for example purposes only, and that various arrangements are possible. Transaction processing system 101 (e.g., a transaction handler) is shown to be in communication with one or more issuer systems (e.g., such as issuer system 106) and one or more acquirer systems (e.g., such as acquirer system 108). Although only a single issuer system 106 and a single acquirer system 108 are shown, it will be appreciated that transaction processing system 101 may be in communication with a plurality of issuer systems and / or acquirer systems. In some non-limiting embodiments or aspects, transaction processing system 101 may also operate as an issuer system, such that both transaction processing66E6779.DOCX Page 27 of 56Attorney Docket No. 08223-2502439 (9305W001) system 101 and issuer system 106 are a single system and / or controlled by a single entity.

[0069] In some non-limiting embodiments or aspects, transaction processing system 101 may communicate with merchant system 104 directly through a public or private network connection. Additionally or alternatively, transaction processing system 101 may communicate with merchant system 104 through payment gateway 102 and / or acquirer system 108. In some non-limiting embodiments or aspects, acquirer system 108 associated with merchant system 104 may operate as payment gateway 102 to facilitate the communication of transaction requests from merchant system 104 to transaction processing system 101. Merchant system 104 may communicate with payment gateway 102 through a public or private network connection. For example, merchant system 104, that includes a physical POS device, may communicate with payment gateway 102 through a public or private network to conduct card-present transactions. As another example, merchant system 104, that includes a server (e.g., a web server), may communicate with payment gateway 102 through a public or private network, such as a public Internet connection, to conduct card-not-present transactions.

[0070] In some non-limiting embodiments or aspects, transaction processing system 101 , after receiving a transaction request from merchant system 104 that identifies an account identifier of a payor (e.g., such as an account holder) associated with issued consumer device 1 10, may generate an authorization request message to be communicated to issuer system 106 that issued consumer device 1 10 and / or the account identifier. Issuer system 106 may then approve or decline the authorization request and, based on the approval or denial, generate an authorization response message that is communicated to transaction processing system 101. Transaction processing system 101 may communicate an approval or denial to merchant system 104. When issuer system 106 approves the authorization request message, it may then clear and settle the payment transaction between issuer system 106 and acquirer system 108.

[0071] The number and arrangement of systems and devices shown in FIG. 1 are provided as an example. There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or differently arranged systems and / or devices than those shown in FIG. 1. Furthermore, two or more systems or devices shown in FIG. 1 may be implemented within a single system or66E6779.DOCX Page 28 of 56Attorney Docket No. 08223-2502439 (9305W001) device, or a single system or device shown in FIG. 1 may be implemented as multiple, distributed systems or devices. Additionally or alternatively, a set of systems (e.g., one or more systems) or a set of devices (e.g., one or more devices) of electronic payment processing network 100 may perform one or more functions described as being performed by another set of systems or another set of devices of electronic payment processing network 100.

[0072] Referring now to FIG. 2, shown is a diagram of example components of device 200, according to non-limiting embodiments or aspects. Device 200 may correspond to transaction processing system 101 , payment gateway 102, merchant system 104, issuer system 106, acquirer system 108, and / or consumer device 110, as an example. In some non-limiting embodiments or aspects, such systems or devices may include at least one device 200 and / or at least one component of device 200. The number and arrangement of components shown are provided as an example. In some non-limiting embodiments or aspects, device 200 may include additional components, fewer components, different components, or differently arranged components than those shown. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0073] As shown in FIG. 2, device 200 may include bus 202, processor 204, memory 206, storage component 208, input component 210, output component 212, and communication interface 214. Bus 202 may include a component that permits communication among the components of device 200. In some non-limiting embodiments or aspects, processor 204 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.) that can be programmed to perform a function. Memory 206 may include random access memory (RAM), read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0074] With continued reference to FIG. 2, storage component 208 may store information and / or software related to the operation and use of device 200. For66E6779.DOCX Page 29 of 56Attorney Docket No. 08223-2502439 (9305W001) example, storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, a solid-state disk, etc.) and / or another type of computer-readable medium. Input component 210 may include a component that permits device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). Additionally or alternatively, input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). Output component 212 may include a component that provides output information from device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 214 may include a transceiver-like component (e.g., a transceiver, a separate receiver and transmitter, etc.) that enables device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 214 may permit device 200 to receive information from another device and / or provide information to another device. For example, communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, and / or the like.

[0075] Device 200 may perform one or more processes described herein. Device 200 may perform these processes based on processor 204 executing software instructions stored by a computer-readable medium, such as memory 206 and / or storage component 208. A computer-readable medium may include any non-transitory memory device. A memory device includes memory space located inside of a single physical storage device or memory space spread across multiple physical storage devices. Software instructions may be read into memory 206 and / or storage component 208 from another computer-readable medium or from another device via communication interface 214. When executed, software instructions stored in memory 206 and / or storage component 208 may cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, embodiments described herein are not limited to any specific combination of hardware circuitry and software. The term “configured to,” as used herein, may refer to an arrangement of software, device(s), and / or hardware66E6779.DOCX Page 30 of 56Attorney Docket No. 08223-2502439 (9305W001) for performing and / or enabling one or more functions (e.g., actions, processes, steps of a process, and / or the like). For example, “a processor configured to” may refer to a processor that executes software instructions (e.g., program code) that cause the processor to perform one or more functions.

[0076] Referring now to FIGS. 3A-3C, shown is a flow diagram for method 300 for structured information extraction from documents, according to some non-limiting embodiments or aspects. The steps shown in FIGS. 3A-3C are for example purposes only. It will be appreciated that additional, fewer, different, and / or a different order of steps may be used in some non-limiting embodiments or aspects. In some non-limiting embodiments or aspects, a step may be automatically performed in response to performance and / or completion of a prior step.

[0077] As shown in FIG. 3A, step 302 of method 300 includes obtaining at least one image associated with at least one page of at least one document. For example, transaction processing system 101 (e.g., using image segmenter module 1 of FIG. 4, etc.) may obtain at least one image associated with at least one page of at least one document (e.g., an image of a document, a PDF file, a GIF file, a JPEG file, a BMP file, etc.).

[0078] As shown in FIG. 3A, step 304 of method 300 includes generating, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof. For example, and referring also to FIG. 5, transaction processing system 101 (e.g., using image segmenter module 1 of FIG. 4, etc.) may generate, using at least one image segmentation technique (e.g., a thresholding technique, an edge detection technique, a region-based technique, a clustering-based technique, a neural network-based technique, etc.), based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof.

[0079] The at least one image segmentation technique may apply to at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box. The first66E6779.DOCX Page 31 of 56Attorney Docket No. 08223-2502439 (9305W001) text associated with the at least one first sub-bounding box may be associated with a first confidence score (e.g., a first confidence score generated by the at least one first OCR technique, etc.). For example, the at least one image segmentation technique may generate a bounding box for each table, the table layout (e.g., landscape or portrait layout, etc.) of each table, and the sub-bounding box and the first text for each cell within each table and / or a bounding box and first text for each line of text in each text column.

[0080] The at least one image segmentation technique may include an image segmentation machine learning model trained to segment document images by identifying non-overlapping tables and other contiguous sections of text (e.g., text columns, etc.) from the document images. For example, the at least one image segmentation technique may identify two bounding boxes in a page with two columns of text (e.g. a bounding box for each column of text, etc.). As an example, the at least one image segmentation technique may identify a page with one column of text with one table in the middle of the column of text as a first bounding box for the cell column before the table, a second bounding box for the table, and a third bounding box for the cell column after the table. An output of the at least one image segmentation technique (e.g. the bounding box(es), sub-bounding box(es), first text associated therewith, and first confidence score associated therewith, etc.) may be automatically compared with a labeled image set where the text and bounding boxes are available to automate checking or verifying the output of the at least one image segmentation technique.

[0081] As shown in FIG. 3A, step 306 of method 300 includes applying at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and second text associated with the at least one second sub-bounding box. For example, transaction processing system 101 (e.g., using OCR engine module 2 of FIG. 4, etc.) may apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and second text associated with the at least one second sub-bounding box. The second text associated with the at least one second subbounding box may be associated with a second confidence score (e.g., a second confidence score generated by the at least one second OCR technique, etc.). As an example, the at least one second OCR technique may process each bounding box66E6779.DOCX Page 32 of 56Attorney Docket No. 08223-2502439 (9305W001) identified by the at least one image segmentation technique (e.g., the image for each table and text column, etc.) to generate the at least one second sub-bounding box associated with that bounding box, the second text associated with that second subbounding box, and the second confidence score for the second text associated with that second sub-bounding box (e.g., associated with that table cell or text line in a column of text, etc.). In this way, by using two or more different techniques or algorithms for the OCR, some non-limiting embodiments or aspects of the present disclosure may reduce a chance of missing some text entirely and / or increase an overall accuracy of the text recognition.

[0082] In some implementations, if the at least one document is completely in text form, the second OCR technique may be omitted and the at least one image segmentation technique may generate the bounding boxes per line from the text document and provide the output directly to page scorer module 4 of FIG. 5.

[0083] As shown in FIG. 3A, step 308 of method 300 includes generating, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof. For example, transaction processing system 101 (e.g., using merge module 3 of FIG. 4, etc.) may generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof.

[0084] Transaction processing system 101 (e.g., using merge module 3 of FIG. 4, etc.) may generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size66E6779.DOCX Page 33 of 56Attorney Docket No. 08223-2502439 (9305W001) of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first sub-bounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first subbounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

[0085] For example, and referring also to FIG. 6, transaction processing system 101 (e.g., using merge module 3 of FIG. 4, etc.) may initialize a merged box list as empty. Transaction processing system 101 (e.g., using merge module 3 of FIG. 4, etc.) may initialize a first OCR list OCR1 that includes the output from the at least one image segmentation technique including the at least one first OCR technique and a second OCR list OCR2 that includes the output from the at least one second OCR technique. Each of the first OCR list OCR1 and the second OCR list OCR2 may be sorted by bounding box and / or sub-bounding box starting from a top left of the at least one page to a bottom right of the at least one page. Transaction processing system 101 (e.g., using merge module 3 of FIG. 4, etc.) may iteratively compare the bounding boxes and / or sub-bounding boxes from the first OCR list OCR1 and the second OCR list OCR2.66E6779.DOCX Page 34 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0086] If two compared boxes overlap completely, the box of the two compared boxes with the higher confidence score may be added to the merged list and the iterative comparison may move to the next boxes in the first OCR list OCR1 and the second OCR list OCR2.

[0087] If two compared boxes do not overlap, each box of the two compared boxes may be added to the merged list and the iterative comparison may move to the next boxes in the first OCR list OCR1 and the second OCR list OCR2.

[0088] If two compared boxes partially overlap, the next boxes from each of the first OCR list OCR1 and the second OCR list OCR2 may be iteratively combined until the two combined boxes match completely. For example, if a current box from the first OCR list OCR1 is bigger than a current box from the second OCR list OCR2, the next box(es) from the second OCR list OCR2 is added to the current box from the second OCR list OCR2 until a combined size is equal to or greater than the size of the current box from the first OCR list OCR1 . If the combined size from the second OCR list OCR2 is greater than the current box from the first OCR list, the next box(es) from the first OCR list OCR1 is added to the current box from the first OCR list OCR1 until a combined size is equal to or greater than the combined size from the second OCR list OCR2 or an end of the first OCR list is reached. An average confidence score for each combined box is then calculated and the combined box with the higher score is added to the merged list and the iterative comparison may move to the next boxes in the first OCR list OCR1 and the second OCR list OCR2.

[0089] Accordingly, starting from the top left and moving to the bottom right of each page, merge module 4 may compare box 1 from OCR1 and box 2 from OCR2. If box 1 and box 2 the same size (e.g., within a threshold tolerance), the box with the higher confidence score is used. If box 1 from OCR1 can be fully contained within box 2 from OCR2, the next boxes in OCR1 may be merged until the combined size is equal to or greater than the size of box 2 from OCR 2. A new confidence score of a combined box may be the weighted average (e.g., weight based on area of the box, etc.) of the scores of the constituent boxes of the combined box. If the combined box from OCR 1 is now larger than box 2 from OCR2, the next boxes in OCR2 may be merged until a combined size is equal to or greater than the combined size of the boxes from OCR1 . This process may be repeated until an end of each line is reached. Once the sizes are equal, the box with the higher confidence score may be used.66E6779.DOCX Page 35 of 56Attorney Docket No. 08223-2502439 (9305W001)

[0090] In some implementations, at least one threshold may be used to compare whether boxes are overlapping as the at least one image segmentation technique and the at least one second OCR technique may generate or draw the bounding boxes or sub-bounding boxes differently (e.g. one library may draw slightly larger boxes than the other, etc.). For example, bounding boxes or sub-bounding boxes may be identified as overlapping if an overlap between the bounding boxes or sub-bounding boxes satisfies the at least one threshold. As an example, if the at least one threshold is set to 90%, bounding boxes or sub-bounding boxes may be identified as overlapping if an overlap between the bounding boxes or sub-bounding boxes is at least 90%. A structured representation (e.g., a markdown, etc.) for each table and text column, that includes the table header, column headers, and each cell in each row for each table, may then be generated as the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof.

[0091] As shown in FIG. 3A, step 310 of method 300 includes, identifying, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text. For example, transaction processing system 101 (e.g., using page scorer module 4 of FIG. 4, etc.) may identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text. The at least one section may include one or more merged bounding boxes of the at least one merged bounding box. The at least one machine learning model may generate at least one relevance score associated with the at least one section.

[0092] The at least one machine learning model may include an NER model and / or a Bidirectional Encoder Representations from Transformers (BERT) model that is trained to identity if a section and / or a page of a document includes relevant information (e.g., relevant to a key-value pair, etc.) and calculate a relevance score for each section and / or page. Each section or page in the merged output may be scored in parallel and only sections or pages that have a relevance score that satisfies a threshold relevance score may be selected. The selected sections or pages may be merged together and provided to the at least one LLM as described herein with respect to step 320 of method 300. The structured text and bounding and / or sub-bounding boxes may be included in the information sent to the LLM. For example, providing the66E6779.DOCX Page 36 of 56Attorney Docket No. 08223-2502439 (9305W001) bounding and / or sub-bounding boxes may improve a fact checking step described herein with respect to steps 324-328 of method 300. Filtering the set of pages sent to the LLM may provide a safeguard against omissions and hallucinations by restricting the LLM’s context, address token length constraints that may be imposed by existing LLMs, and / or improve latency because only relevant sections or pages may be sent to the LLM.

[0093] As shown in FIG. 3A, step 312 of method 300 includes determining whether the at least one relevance score satisfies at least one threshold relevance score. For example, transaction processing system 101 (e.g., using page scorer module 4 of FIG. 4, etc.) may determine whether the at least one relevance score satisfies the at least one threshold relevance score.

[0094] As shown in FIG. 3B, step 314 of method 300 includes, in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score in step 312, maintaining the at least one section in the at least one document. For example, transaction processing system 101 (e.g., using page scorer module 4 of FIG. 4, etc.) may, in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document. As an example, the at least one section may be maintained in the merged output or information sent to the LLM.

[0095] As shown in FIG. 3B, step 316 of method 300 includes, in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score in step 312, removing the at least one section from the at least one document. For example, transaction processing system 101 (e.g., using page scorer module 4 of FIG. 4, etc.) may, in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document. As an example, the at least one section may be removed from the merged output or information before sending the merged output or information to the LLM.

[0096] As shown in FIG. 3B, step 318 of method 300 includes anonymizing, using a personal information (PI) tokenization technique, personal information included in the merged text in the at least one merged bounding box. For example, transaction processing system 101 (e.g., using PI anonymizer module 5 of FIG. 4, etc.) may66E6779.DOCX Page 37 of 56Attorney Docket No. 08223-2502439 (9305W001) anonymize, using a PI tokenization technique, personal information included in the merged text in the at least one merged bounding box (e.g., before sending the merged text in the at least one merged bounding box to LLM module 6 of FIG. 4, etc.). In some implementations, step 318 may be performed only if the LLM being used is hosted in the cloud as opposed to locally. For example, the PI may not be anonymized if the LMM is implemented by transaction processing system 101 .

[0097] As shown in FIG. 3B, step 320 of method 300 includes extracting, using at least one LLM, based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text. For example, transaction processing system 101 (e.g., using LLM module 6 of FIG. 4, etc.) may extract, using at least one LLM, based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

[0098] Transaction processing system 101 (e.g., using LLM module 6 of FIG. 4, etc.) may extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof. At least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair; the one or more expected formats for the at least one value of the at least one key-value pair; or any combination thereof, may be determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document. Each of the at least one key-value pair and the at least one previous key-value pair may be associated with a same client.

[0099] For example, the target meta-data (e.g., the key-value pairs of interest, etc.) may be provided with the merged text and the bounding boxes of the identified pages of interest to the at least one LLM. As an example, if only the first two pages of a document are identified as of interest, only text and meta-data for those pages may be provided with the LLM, which may reduce a chance of hallucination by the LLM. Further, to provide the LLM with additional context and prevent false negatives, for each key-value pair, the LLM may be provided with the following: common synonyms66E6779.DOCX Page 38 of 56Attorney Docket No. 08223-2502439 (9305W001) for each key, additional natural language descriptions / prompts, and / or expected formats for each value, which may help augment a general purpose LLM with specific domain knowledge it may not have been heavily trained on (e.g., payments or business-specific terminology, etc.), and / or examples of previously analyzed documents and filled meta-data to help the LLM with context. These can be document supplier specific historical examples for exchanges where different suppliers have different templates / styles of documents.

[0100] As shown in FIG. 3B, step 322 of method 300 includes receiving, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text. For example, transaction processing system 101 may receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text. As an example, the at least one LLM may generate, based on the at least one key-value pair of interest and the at least one document including the at least one page including the at least one merged bounding box including the merged text, the at least one further bounding box associated with the at least one data element extracted from the merged text and provide the at least one further bounding box associated with the at least one data element extracted from the merged text to transaction processing system 101. For example, the at least one LLM may provide a bounding box for each extracted data element.

[0101] As shown in FIG. 3B, step 324 of method 300 includes determining whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text. For example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text. As an example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may determine if a bounding box provided for each extracted data element exists in the merged output.

[0102] As shown in FIG. 30, step 326 of method 300 includes, in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document66E6779.DOCX Page 39 of 56Attorney Docket No. 08223-2502439 (9305W001) including the at least one page including the at least one merged bounding box including the merged text, verifying, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest. For example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may, in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest. As an example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may check if the data element label and value corresponds to or matches the merged text in the bounding box using a fuzzy match (e.g., E-mail, Electronic Mail, Email, Email Address may each correspond to or match to the keyword ‘email’, etc.) which may reduce a chance of hallucination in the output.

[0103] As shown in FIG. 30, step 328 of method 300 includes, in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, deleting the at least one data element extracted from the merged text. For example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may, in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text. As an example, transaction processing system 101 (e.g., using fact checker module 7 of FIG. 4, etc.) may not fill the meta-data element if a bounding box provided for an extracted data element does not exist in the merged output.

[0104] As shown in FIG. 30, step 330 of method 300 includes de-anonymizing, using the PI tokenization technique, the PI in the merged text included in the at least one data element extracted from the merged text. For example, transaction processing system 101 (e.g., using PI de-anonymizer module 8 of FIG. 4, etc.) may deanonymize, using the PI tokenization technique, the PI in the merged text included in66E6779.DOCX Page 40 of 56Attorney Docket No. 08223-2502439 (9305W001) the at least one data element extracted from the merged text. In some implementations, step 330 may be performed only if the PI in the merged text included in the at least one data element extracted from the merged text was anonymized before being provided to the at least one LLM.

[0105] As shown in FIG. 30, step 332 of method 300 includes automatically modifying, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof. For example, transaction processing system 101 (e.g., using PI deanonymizer module 8 of FIG. 4, etc.) may automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof. As an example, transaction processing system 101 (e.g., using PI de-anonymizer module 8 of FIG. 4, etc.) may automatically correct the merged OCR output based on context using a smaller and / or local LLM model. In such an example, in the E-mail value field, a comma probably means and may be replaced with a dot when considered in context by the LLM, and in an IP value field, an T probably means ‘1 ’, ‘D’ probably means and may be replaced with a ‘0’ when considered in context by the LLM.

[0106] Accordingly, some non-limiting embodiments or aspects of the present disclosure may provide an Al or machine learning driven system for extracting structured information from documents that include requested information but may not follow a pre-defined template. Non-limiting embodiments or aspects of the present disclosure may reduce a number of false positive and false negative errors by combining outputs of an image segmentation technique or model and an OCR technique or model using an improved approach for merging outputs. Non-limiting embodiments or aspects of the present disclosure may reduce LLM hallucinations by using a fact-checking approach and auto-correcting OCR recognition errors by using the context of the data field.

[0107] Although embodiments have been described in detail for the purpose of illustration, it is to be understood that such detail is solely for that purpose and that the disclosure is not limited to the disclosed embodiments or aspects, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the66E6779.DOCX Page 41 of 56Attorney Docket No. 08223-2502439 (9305W001) present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect can be combined with one or more features of any other embodiment or aspect.66E6779.DOCX Page 42 of 56

Claims

Attorney Docket No. 08223-2502439 (9305W001)WHAT IS CLAIMED IS:1 . A system, comprising: at least one processor configured to: obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second subbounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second subbounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.66E6779.DOCX Page 43 of 56Attorney Docket No. 08223-2502439 (9305W001)2. The system of claim 1 , wherein the at least one processor is further configured to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text.

3. The system of claim 1 , wherein the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score;66E6779.DOCX Page 44 of 56Attorney Docket No. 08223-2502439 (9305W001) in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

4. The system of claim 1 , wherein the at least one processor is further configured to: before using the at least one LLM to extract the at least one data element: anonymize, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: de-anonymize, using the PI tokenization technique, the PI in the merged text included in the at least one data element extracted from the merged text.

5. The system of claim 1 , wherein the at least one processor is further configured to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof.

6. The system of claim 1 , wherein the at least one processor is configured to extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at66E6779.DOCX Page 45 of 56Attorney Docket No. 08223-2502439 (9305W001) least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair; the one or more expected formats for the at least one value of the at least one key-value pair; or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

7. The system of claim 1 , wherein the at least one processor is configured to generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first subbounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a66E6779.DOCX Page 46 of 56Attorney Docket No. 08223-2502439 (9305W001) combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

8. A method, comprising: obtaining, with at least one processor, at least one image associated with at least one page of at least one document; generating, with the at least one processor, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first subbounding box within the at least one bounding box and a first text associated with the at least one first sub-bounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; applying, with the at least one processor, at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generating, with the at least one processor, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second66E6779.DOCX Page 47 of 56Attorney Docket No. 08223-2502439 (9305W001) confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extracting, with the at least one processor, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

9. The method of claim 8, further comprising: receiving, with the at least one processor, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determining, with the at least one processor, whether the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verifying, with the at least one processor, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete, with the at least one processor, the at least one data element extracted from the merged text.

10. The method of claim 8, further comprising: before using the at least one LLM to extract the at least one data element: identifying, with the at least one processor, using at least one machine learning model, at least one section within the at least one page of the66E6779.DOCX Page 48 of 56Attorney Docket No. 08223-2502439 (9305W001) at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determining, with the at least one processor, whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintaining, with the at least one processor, the at least section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, removing, with the at least one processor, the at least one section from the at least one document.1 1 . The method of claim 8, further comprising: before using the at least one LLM to extract the at least one data element: anonymizing, with the at least one processor, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element: de-anonymizing, with the at least one processor, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

12. The method of claim 8, further comprising: automatically modifying, with the at least one processor, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text, the at least one data element, or any combination thereof.66E6779.DOCX Page 49 of 56Attorney Docket No. 08223-2502439 (9305W001)13. The method of claim 8, wherein extracting, with the at least one processor, using the at least one LLM, the at least one data element includes providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair, one or more expected formats for at least one value of the at least one key-value pair, or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair, the one or more expected formats for the at least one value of the at least one key-value pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

14. The method of claim 8, wherein generating, with the at least one processor, the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, includes: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first subbounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes being greater than the size of the66E6779.DOCX Page 50 of 56Attorney Docket No. 08223-2502439 (9305W001) second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score.

15. A computer program product comprising at least one non- transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to: obtain at least one image associated with at least one page of at least one document; generate, using at least one image segmentation technique, based on the at least one image, at least one bounding box associated with at least one of the following: at least one column of text on the at least one page in the at least one document; at least one table on the at least one page in the at least one document; or any combination thereof, wherein the at least one image segmentation technique applies at least one first optical character recognition (OCR) technique within the at least one bounding box to generate at least one first sub-bounding box within the at least one bounding box and a first text associated with the at least one first subbounding box, wherein the first text associated with the at least one first sub-bounding box is associated with a first confidence score; apply at least one second OCR technique different than the at least one first OCR technique within the at least one bounding box to generate at least one66E6779.DOCX Page 51 of 56Attorney Docket No. 08223-2502439 (9305W001) second sub-bounding box associated with the at least one bounding box and a second text associated with the at least one second sub-bounding box, wherein the second text associated with the at least one second sub-bounding box is associated with a second confidence score; generate, based on at least one of the following: (i) whether the at least one first sub-bounding box overlaps with the at least one second sub-bounding box; (ii) the first confidence score; (iii) the second confidence score; or any combination thereof, at least one merged bounding box including merged text generated from at least one of the first text, the second text, or any combination thereof; and extract, using at least one large language model (LLM), based on at least one key-value pair of interest, from the at least one document including the at least one page including the at least one merged bounding box including the merged text, at least one data element from the merged text.

16. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: receive, from the at least one LLM, at least one further bounding box associated with the at least one data element extracted from the merged text; determine whether at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text; in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, verify, using a fuzzy match, that the at least one data element extracted from the merged text corresponds to the at least one key-value pair of interest; and in response to determining that the at least one further bounding box associated with the at least one data element extracted from the merged text is included in the at least one document including the at least one page including the at least one merged bounding box including the merged text, delete the at least one data element extracted from the merged text.66E6779.DOCX Page 52 of 56Attorney Docket No. 08223-2502439 (9305W001)17. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: before using the at least one LLM to extract the at least one data element: identify, using at least one machine learning model, at least one section within the at least one page of the at least one document including the at least one merged bounding box including the merged text, wherein the at least one section includes one or more merged bounding boxes of the at least one merged bounding box, and wherein the at least one machine learning model generates at least one relevance score associated with the at least one section; determine whether the at least one relevance score satisfies at least one threshold relevance score; in response to determining that the at least one relevance score associated with the at least one section satisfies the at least one threshold relevance score, maintain the at least one section in the at least one document; and in response to determining that the at least one relevance score associated with the at least one section fails to satisfy the at least one threshold relevance score, remove the at least one section from the at least one document.

18. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: before using the at least one LLM to extract the at least one data element: anonymize, using a personal information (PI) tokenization technique, PI included in the merged text in the at least one merged bounding box; and after using the at least one LLM to extract the at least one data element:66E6779.DOCX Page 53 of 56Attorney Docket No. 08223-2502439 (9305W001) de-anonymize, using the PI tokenization technique, the personal information in the merged text included in the at least one data element extracted from the merged text.

19. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: automatically modify, using the at least one LLM, based on a context associated with at least one of the merged text, the at least one key-value pair, or any combination thereof, at least one of the following: the merged text; at least one data element; or any combination thereof, wherein the program instructions, when executed by the at least one processor, cause the at least one processor to extract, using the at least one LLM, the at least one data element by providing the LLM with at least one of the following: one or more synonyms for at least one key of the at least one key-value pair; one or more expected formats for at least one value of the at least one key-value pair; or any combination thereof, wherein at least one of the following: the one or more synonyms for the at least one key of the at least one key-value pair, the one or more expected formats for the at least one value of the at least one key-value pair, or any combination thereof, is determined based on at least one previous key-value pair used to extract, using the at least one LLM, at least one previous data element from at least one previous document, and wherein each of the at least one key-value pair and the at least one previous key-value pair is associated with a same client.

20. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, cause the at least one processor to generate the at least one merged bounding box including the merged text generated from the at least one of the first text, the second text, or any combination thereof, by: for each page of the at least one page of the at least one document, and for each bounding box of the at least one bounding box on that page: iteratively comparing, by moving from a same initial position of that bounding box toward a same opposite position of that bounding box, each first sub-bounding box of the at least one first sub-bounding box for that66E6779.DOCX Page 54 of 56Attorney Docket No. 08223-2502439 (9305W001) bounding box and each second sub-bounding box of the at least one second sub-bounding box for that bounding box; in response to a size of a first sub-bounding box being smaller than a size of a second sub-bounding box by a threshold amount, iteratively combining a next first sub-bounding box with that first sub-bounding box until a size of the combined first sub-bounding boxes corresponds to or is greater than the size of the second sub-bounding box, wherein the combined first subbounding boxes is associated with a combined first confidence score determined as a weighted average of first confidence scores associated with the first sub-bounding boxes combined therein; in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes being greater than the size of the second sub-bounding box by the threshold amount, iteratively combining a next second sub-bounding box with that second sub-bounding box until a size of the combined second sub-bounding boxes corresponds to or is greater than the size of the first sub-bounding box or the size of the combined first sub-bounding boxes, wherein the combined second sub-bounding boxes is associated with a combined second confidence score determined as a weighted average of second confidence scores associated with the second sub-bounding boxes combined therein; and in response to the size of the first sub-bounding box or the size of the combined first sub-bounding boxes corresponding to the size of the second sub-bounding box or the combined second sub-bounding boxes within a threshold tolerance or reaching the same opposite position, generating the at least one merged bounding box including the merged text based on whichever of that first sub-bounding box or combined first sub-bounding boxes and that second sub-bounding box or combined second sub-bounding boxes is associated with a higher confidence score. E6779.DOCX Page 55 of 56

Images