Document information processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a document information processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: recognizing a document image according to a handwritten character recognition mode, and performing row division on the basis of coordinate position information of each character string in a recognition result to obtain row information of fields included in each character string; performing information group division on each character string according to an information group mode of the receipt image to obtain each information group; dividing each information block according to column dimensions to obtain respective column information of each information block, and obtaining respective row information of each information block according to the row information of the fields included in each information block; and when it is determined that the receipt image does not meet the table character recognition condition, obtaining receipt information in the receipt image based on each information block and the respective row information and column information of each information block. By adopting the method, the accuracy of document information extraction can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a document information processing method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] The information distribution in various documents such as handwritten documents or machine-printed documents has great flexibility and uncertainty, especially in handwritten documents, whose character writing method, stroke order, font size and writing position are all very uncertain. This uncertainty makes it difficult to accurately judge the row and column relationship between characters in the process of document information recognition, increases the difficulty of information extraction, and makes it easy to make mistakes in character segmentation, feature extraction and classification recognition, resulting in character confusion, missed recognition and misrecognition in the recognition results, thereby reducing the accuracy of document information extraction. Summary of the invention

[0003] Based on this, it is necessary to provide a document information processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of document information extraction in response to the above technical problems.

[0004] In a first aspect, the present application provides a document information processing method, comprising:

[0005] Acquire a document image, and perform recognition on the document image according to a handwritten character recognition method to obtain a recognition result of the document image, wherein the recognition result includes each character string and the coordinate position information of each character string;

[0006] Divide each string into rows based on the coordinate position information of each string, and obtain row information of the fields included in each string;

[0007] According to the information group pattern of the document image, based on the coordinate position information of each character string and the row information of each character string, each character string is divided into information groups to obtain each information group that conforms to the information group pattern;

[0008] Divide each information group according to the column dimension to obtain the column information of each information group, and obtain the row information of each information group according to the row information of the fields included in each information group;

[0009] Determine the column information of the fields included in each character string according to the column information of each information group;

[0010] When it is determined that the document image does not meet the table character recognition condition based on the row information and column information of the fields included in each character string, the document information in the document image is obtained based on each information group and the row information and column information of each information group.

[0011] In a second aspect, the present application further provides a document information processing device, comprising:

[0012] A handwritten character recognition module is used to obtain a document image, recognize the document image according to a handwritten character recognition method, and obtain a recognition result of the document image, wherein the recognition result includes each character string and the coordinate position information of each character string;

[0013] A string row division module is used to divide each string into rows based on the coordinate position information of each string, and obtain row information of the fields included in each string;

[0014] An information group division module is used to divide each character string into information groups according to the information group mode of the document image, based on the coordinate position information of each character string and the row information of each character string, to obtain each information group that conforms to the information group mode;

[0015] A row and column information determination module is used to divide each information group according to the column dimension, obtain the column information of each information group, and obtain the row information of each information group according to the row information of the fields included in each information group;

[0016] A field column information determination module, used to determine the column information of the fields included in each character string according to the column information of each information group;

[0017] The document information acquisition module is used to obtain the document information in the document image based on each information group and the row information and column information of each information group when it is determined that the document image does not meet the table character recognition conditions based on the row information and column information of the fields included in each character string.

[0018] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above document information processing method when executing the computer program.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above document information processing method are implemented.

[0020] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above document information processing method when executed by a processor.

[0021] The document information processing method, device, computer equipment, computer-readable storage medium and computer program product identify the document image in a handwritten character recognition manner to obtain a recognition result including each character string and each coordinate position information of each character string, divide each character string into rows based on each coordinate position information of each character string to obtain row information of each field included in each character string, divide each character string into information groups according to the information group pattern of the document image based on each coordinate position information of each character string and each row information of each character string to obtain each information group that conforms to the information group pattern, divide each information group according to the column dimension to obtain each column information of each information group, obtain each row information of each information group based on the row information of each field included in each information group, determine the column information of each field included in each character string based on the column information of each information group, and when it is determined that the document image does not meet the table character recognition condition based on the row information and column information of each field included in each character string, obtain the document information in the document image based on each information group and each row information and column information of each information group. In the process of document information processing, the document image is recognized according to the handwritten character recognition method, and the row division is performed based on the coordinate position information in the recognition result to obtain the row information of the fields included in each character string. The information groups are divided based on the information group pattern and the row information of each character string, and the row information and column information of each information group are determined to obtain the document information in the document image. This can overcome the problem of difficulty in document information extraction caused by the lack of row and column information in the handwritten character recognition method, and improve the accuracy of document information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 An application environment diagram of a document information processing method in an embodiment;

[0024] Figure 2 A schematic diagram of a process of processing document information in one embodiment;

[0025] Figure 3 A schematic diagram of a process for obtaining document information in one embodiment;

[0026] Figure 4Schematic diagram of a document image in an embodiment;

[0027] Figure 5 Schematic diagram of the recognition result of a document image according to handwritten OCR recognition in an embodiment;

[0028] Figure 6 Schematic diagram of the recognition result of a document image according to table OCR recognition in an embodiment;

[0029] Figure 7 Schematic diagram of the recognition result of a document image according to table OCR recognition in another embodiment;

[0030] Figure 8 Schematic flowchart of a document information processing method in another embodiment;

[0031] Fig. 9 Schematic diagram of the recognition result of a handwritten document in an embodiment;

[0032] Fig.10 Schematic diagram of the recognition result of a handwritten document in another embodiment;

[0033] Fig.11 Schematic diagram of a document image belonging to the single information group type in an embodiment;

[0034] Fig.12 Schematic diagram of a document image belonging to the multi - information group type in an embodiment;

[0035] Fig.13 Schematic diagram of the glyph splitting of the character "poor" in an embodiment;

[0036] Fig.14 Schematic diagram of a document image for which document information needs to be extracted in another embodiment;

[0037] Fig.15 For Fig.14 Schematic diagram of the document information extraction result of the document image therein;

[0038] Fig.16 Schematic block diagram of a document information processing device in an embodiment;

[0039] Fig.17 Internal structure diagram of a computer device in an embodiment;

[0040] Fig.18 Internal structure diagram of a computer device in another embodiment. Detailed implementation manners

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The document information processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0043] The terminal 102 may photograph the document whose document information needs to be extracted to obtain a document image. The terminal 102 may send the photographed document image to the server 104 via the network so that the server 104 may process the document information. The server 104 can recognize the document image in a handwritten character recognition manner to obtain a recognition result including each character string and the coordinate position information of each character string. The server 104 divides each character string into rows based on the coordinate position information of each character string to obtain the row information of each field included in each character string. The server 104 divides each character string into information groups according to the information group mode of the document image based on the coordinate position information of each character string and the row information of each character string to obtain each information group that conforms to the information group mode. The server 104 divides each information group according to the column dimension to obtain the column information of each information group, and obtains the row information of each information group according to the row information of the field included in each information group. The server 104 determines the column information of each field included in each character string according to the column information of each information group. When it is determined that the document image does not meet the table character recognition condition based on the row information and column information of the field included in each character string, the server 104 obtains the document information in the document image based on each information group and the row information and column information of each information group. The server 104 may feed back the obtained document information to the terminal 102 so that the document information can be visually displayed in the terminal 102 .

[0044] In some embodiments, the document information processing method may also be implemented independently by the terminal 102 or the server 104. For example, after the terminal 102 captures the document image, the terminal 102 may directly perform document information processing based on the document image. For another example, the document image may be stored in the data storage system of the server 104. The server 104 may obtain the document image from the data storage system and perform document information processing on the obtained document image. The document information extracted by the server 104 may also be stored in the data storage system.

[0045] Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0046] In an exemplary embodiment, Figure 2 As shown, a document information processing method is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 212. Among them:

[0047] Step 202, obtaining a document image, and performing recognition on the document image according to a handwritten character recognition method to obtain a recognition result of the document image, wherein the recognition result includes each character string and the coordinate position information of each character string.

[0048] Among them, a document refers to a written voucher filled in a certain format to prove various matters such as the occurrence of economic transactions, the establishment of creditor-debtor relationships, or the receipt and payment of funds. The document can be obtained manually by handwriting or by printing based on a printer. The document image is an image collected from the document, which can be obtained by photographing the document with a camera or a camera. The handwritten character recognition method is a processing method suitable for character recognition for documents that do not have document form information to extract document information, such as character recognition for user handwritten documents through the handwritten character recognition method. The handwritten character recognition method can be implemented by calling the handwritten OCR (Optical Character Recognition) recognition interface to convert the text included in the handwritten document into a computer-editable text through methods such as image processing, pattern recognition and deep learning. The recognition result is the result obtained by recognition based on the document image, which can specifically include each character string recognized from the document image and the coordinate position information corresponding to each character string in the document image. Based on the coordinate position information, the distribution position of the corresponding character string in the document image can be determined, so as to divide the rows and columns for each character string in the document image, thereby extracting the document information. The document information can be the voucher information corresponding to various matters carried in the document corresponding to the document image.

[0049] Exemplarily, the server may obtain a document image that needs to be processed for document information extraction, and the server may perform recognition on the document image in accordance with a handwritten character recognition method. Specifically, the server may perform recognition on the document image by calling a handwritten OCR interface to obtain a recognition result of the document image. The recognition result of the document image may specifically include each character string recognized from the document image and the coordinate position information of each character string, and the coordinate position information may specifically include horizontal and vertical coordinate information. Among them, the character string may include at least one field, and each field may include at least one character, that is, at least one character may constitute a field, and at least one field may constitute a character string. In some embodiments, the recognition result of the document image may include a character string divided by a text box, and the coordinate position information of the character string may be obtained based on the position information of the text box, and the position information of the text box may be specifically used as the coordinate position information of the character string identified by the text box.

[0050] Step 204 , dividing each string into rows based on the coordinate position information of each string, and obtaining row information of the fields included in each string.

[0051] Among them, row division refers to dividing each string into different rows, and the row information may specifically include row identification information, such as the row serial number, row position information, and other information. Optionally, the server may perform row division for each string based on the coordinate position information of each string. Specifically, the server may determine the height information of the string based on the coordinate position information, and divide each string into rows based on the height information of the string. For example, strings within the same height position range may be divided into the same row, so that each string may be divided into different rows. After dividing each string into its respective rows, the server may obtain row information of the fields included in the string based on the row to which the string belongs. Specifically, the row to which the string belongs may be used as the row to which each field included in the string belongs, so that row information of the fields in the string may be obtained based on the row to which the string belongs, such as determining the row serial number, row position information, and the like.

[0052] Step 206 , according to the information group pattern of the document image, based on the coordinate position information of each character string and the row information of each character string, divide each character string into information groups to obtain each information group that conforms to the information group pattern.

[0053] The information group mode refers to the type of document information included in each information group when the document information in the document image is divided into information groups. The document information type included in the information group may include entity type and non-entity type. For example, the information group mode may include "commodity name-quantity-unit price-unit", that is, the document information in the document image is recorded according to the information group unit of "commodity name-quantity-unit price-unit", wherein "commodity name" and "unit" may be entity types, such as "apple", "watermelon", "jin", "gram", "box", etc.; and "quantity" and "unit price" may be non-entity types, such as "10", "12", etc. Different documents may have different information group modes, and the information group mode may be determined by identifying and judging the document image. Based on the information group mode of the document image, the document information type corresponding to the document information included in each information group in the document image may be determined. In the document image, each row may include at least one information group. When each row includes one information group, each row of the document image is regarded as an information group, and the document image belongs to a single information group image. When at least one row includes at least two information groups, that is, when the information group in at least one row is repeated at least once, there is a row in the document image including multiple information groups, and the document image belongs to a multiple information group image. The information groups obtained by division conform to the information group mode, that is, the information groups obtained by division are specifically single information groups, and the document information included in each information group belongs to the information group mode and there is no repeated document information type. In some embodiments, the information groups obtained by division may include document information of at least part of the document information types in the information group mode. For example, the information group mode includes document information types such as "commodity name-quantity-unit price-unit", and the information group A obtained by division may be "commodity name 1-quantity 1-unit price 1-unit 1", the information group B may be "commodity name 2-unit price 2-unit 2", and the information group C may be "commodity name 3", wherein the information group A, the information group B and the information group C are all single information groups.

[0054] Optionally, the server can determine the information group mode of the document image. Specifically, the server can make a determination based on the document image, such as through a pre-trained network model, or can make a determination based on the document image based on text feature rules. The information group mode of the document image can be determined, that is, all document information types included in the complete information group of the document image can be determined. In some embodiments, the information group mode of the document image can also be directly determined and configured by the user, that is, the user can analyze based on the document image and set the corresponding information group mode. The server can divide the information group based on the coordinate position information of each character string and the row information of each character string, so as to divide each information group from each character string, and each information group obtained by division conforms to the information group mode of the document image. The row information of each character string can be determined based on the row division result for each character string, that is, the row information of each character string is the same as the row information of the field included in each character string. Among them, each information group obtained by division is a single information group conforming to the information group mode, that is, there is no repetition of the information group mode in each information group obtained by division. In some embodiments, the server may identify the entity type to which each field in each string belongs, and divide the fields of each string into information groups based on the entity type in the information group pattern to obtain information groups that conform to the information group pattern.

[0055] Step 208, divide each information group according to the column dimension to obtain the column information of each information group, and obtain the row information of each information group according to the row information of the fields included in each information group.

[0056] Among them, the document information in the document image is presented in units of information groups. After being divided into each information group, each information group needs to be rearranged to divide rows and columns based on the information group, so as to determine the row and column information of each information group, specifically including row information and column information.

[0057] Exemplarily, the server may determine row information and column information for each information group, wherein the row information of the information group may be determined based on the row information of the fields included in the information group. Specifically, the row information of each field obtained by row division for each character string may be reused, and the row information of the fields included in the information group may be directly used as the row information of the entire information group. For column information, the server may perform column dimension division based on each information group, that is, perform column division for each information group, thereby determining the respective column information of each information group. In some embodiments, the server may determine the horizontal coordinate information of each information group, specifically, the horizontal coordinate information of the entire information group may be determined based on the position information of each field included in the information group, and the columns may be aligned based on the horizontal coordinate information, thereby realizing the column division for the information group, and obtaining the column information of each information group. The column information may specifically include, but is not limited to, column sequence number, column position information, etc.

[0058] Step 210: Determine the column information of the fields included in each character string according to the column information of each information group.

[0059] Exemplarily, after determining the column information of each information group, the server may determine the column information of the included fields based on the column information of the information group, and specifically may use the column information of the information group as the column information of the fields included in the information group.

[0060] Step 212, when it is determined that the document image does not meet the table character recognition condition based on the row information and column information of the fields included in each character string, the document information in the document image is obtained based on each information group and the row information and column information of each information group.

[0061] Among them, the table character recognition condition is used to determine whether it is necessary to perform character recognition on the document image through the table character recognition method. Specifically, the determination can be made based on the row information and column information of each field in the document image, such as determining whether the row information and column information of each field carry table information, thereby determining whether the document image is suitable for document information extraction using the table character recognition method.

[0062] Exemplarily, the server may obtain the table character recognition condition, and determine whether the table character recognition condition is met based on the row information and column information of each field. In some embodiments, after determining the row and column information of each field, the server may further determine the row and column information of the string to which each field belongs, and cluster each string based on the row and column information of each string, so as to determine whether the document image meets the table character recognition condition in combination with the clustering result. For example, the server may determine the image category of the document image based on the clustering result, and determine whether the document image belongs to the category that meets the table character recognition condition. When the document image does not belong to the category that meets the table character recognition condition, it can be determined that the document image does not meet the table character recognition condition, and then the document information in the document image can be directly obtained based on each information group and the respective row and column information. In some embodiments, the server may extract the original document information from the information group, and determine the corresponding reference information from the document management system, and the reference information may include the document template information preset in the document management system. The server may also generate the corresponding identification information based on the row and column information of the information group, and the server may obtain the document information of the document image based on the original document information, the reference information and the identification information.

[0063] In the above-mentioned document information processing method, the server recognizes the document image in accordance with the handwritten character recognition method to obtain a recognition result including each character string and the coordinate position information of each character string, divides each character string into rows based on the coordinate position information of each character string to obtain the row information of each field included in each character string, divides each character string into information groups according to the information group pattern of the document image based on the position information of each character string and the row information of each character string to obtain each information group that conforms to the information group pattern, divides each information group according to the column dimension to obtain the column information of each information group, and obtains the row information of each information group based on the row information of the field included in each information group, determines the column information of each field included in each character string based on the column information of each information group, and when it is determined that the document image does not meet the table character recognition conditions based on the row information and column information of the field included in each character string, the document information in the document image is obtained based on each information group and the row information and column information of each information group. In the process of document information processing, the document image is recognized according to the handwritten character recognition method, and the row division is performed based on the coordinate position information in the recognition result to obtain the row information of the fields included in each character string. The information groups are divided based on the information group pattern and the row information of each character string, and the row information and column information of each information group are determined to obtain the document information in the document image. This can overcome the problem of difficulty in document information extraction caused by the lack of row and column information in the handwritten character recognition method, and improve the accuracy of document information extraction.

[0064] In an exemplary embodiment, each string is divided into rows based on its respective coordinate position information to obtain row information of the fields included in each string, including: determining the first string of the first row from each string, taking the first row as the target row, and taking the first string in the target row as the target string; determining the character reference height information of the target row according to the coordinate position information of the target string; determining the next string adjacent to the target string from each string; when the coordinate position information of the next string meets the character reference height information, dividing the next string into the target row, updating the character reference height information according to the coordinate position information of the next string, updating the next string to the target string, and returning to execute the step of determining the next string adjacent to the target string from each string; when the coordinate position information of the next string does not meet the character reference height information, determining the next string as the first string in the next row of the target row, updating the next row of the target row to the target row, and returning to execute the step of taking the first string in the target row as the target string; when the row division for each string is completed, according to the row to which each string belongs after division, the row information of the fields included in each string is obtained.

[0065] The first row is the first row when dividing each string into rows, the first string is the first string in a row, the target row is the row currently being divided into rows, and the target string is the string used as a reference when dividing into rows. Character reference height information is reference information used when dividing a string into rows, and may include but is not limited to character height position information, such as the maximum longitudinal coordinate point, the minimum longitudinal coordinate point, and the average longitudinal coordinate point of the string.

[0066] Optionally, for each string obtained by identification, the server can determine the first string of the first row, and specifically can use the first string of all strings as the first string of the first row, such as the first string "watermelon" can be determined as the first string of the first row. In a specific application, the server can sort each string from left to right and then from top to bottom to construct an ordered string sequence, so as to determine the first string of all strings based on the string sequence, and the first string can be used as the first string of the first row. The server can use the first row as the target row, and use the first string of the target row as the target string to perform row division processing for other strings. The server can determine the coordinate position information of the target string, and obtain the character reference height information of the target row based on the coordinate position information. Specifically, the server can determine the maximum longitudinal coordinate point and the minimum coordinate point of the target string based on the coordinate position information of the target string, and obtain the character reference height information of the target row based on the maximum longitudinal coordinate point and the minimum longitudinal coordinate point, indicating that the character height position of the string belonging to the target row should belong to between the maximum longitudinal coordinate point and the minimum longitudinal coordinate point. For example, for the target character string "Watermelon", the character reference height information of the target row can be determined according to the coordinate position information of the character string, including the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of "Watermelon".

[0067] Furthermore, the server can obtain the next string adjacent to the target string from each string to perform row division for the adjacent next string, and specifically determine whether the next string belongs to the target row, that is, determine whether the next string belongs to the target row or belongs to the next row of the target row. The server can determine the coordinate position information of the next string, and match the coordinate position information with the character reference height information of the target row to determine whether the coordinate position information of the next string meets the character reference height information of the target row. In a specific application, the server can determine the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the next character based on the coordinate position information of the next string, and the server can determine whether the next string is within the longitudinal coordinate point range of the character reference height information based on the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the next character, thereby determining whether the next string needs to be divided into the target row. When it is determined that the coordinate position information of the next string meets the character reference height information, the server can divide the next string into the target row, thereby realizing row division for the next string.

[0068] Furthermore, the server can update the character reference height information of the target row based on the coordinate position information of the next character string, such as updating the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the target row in the character reference height information based on the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the next character, and updating the next character string to the target character string, so as to use the next character string as a reference for subsequent character string row division. After dividing the next character string into the target row and updating the next character string to the target character string, the server can return to the process of obtaining the next character string adjacent to the target character string, thereby performing the character string row division traversal process.

[0069] Furthermore, when it is determined that the coordinate position information of the next character string does not conform to the character reference height information, it indicates that the longitudinal coordinate point of the next character string deviates greatly from the longitudinal coordinate point range of the target row, and it can be considered that the next character string does not belong to the target row. The server can divide the next character string into the next row of the target row and use it as the first character string of the next row. The server can update the next row as the target row to perform row division processing for subsequent characters. The specific server can return the first character string in the target row as the target string, thereby realizing the update of the target row and the target string. After traversing each character string and completing the row division, the server can determine the row to which each character string belongs, thereby determining the navigation information of the fields included in the character string. The specific server can use the row information of the row to which the string belongs as the row information of each field included in the string.

[0070] In this embodiment, the server determines the character reference height information based on the coordinate position information of the character string included in each line, divides the lines based on the character reference height information and the coordinate position information of the character string, and adaptively updates the character reference height information of each line. After traversing each character string, the handwritten tilted characters can be divided into the same line, thereby ensuring the accuracy of the character string line division, which is conducive to improving the accuracy of document information extraction.

[0071] In an exemplary embodiment, determining character reference height information of a target row according to coordinate position information of a target string includes: determining height range information of the target string according to the coordinate position information of the target string; and obtaining character reference height information of the target row based on the height range information of the target string.

[0072] Wherein, the height range information may include the longitudinal coordinate point range information of the target string, and specifically may include the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the target string. Exemplarily, the server may determine the longitudinal coordinate point range of the target string based on the coordinate position information of the target string, and specifically may include the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the target string, thereby obtaining the height range information of the target string. The server may obtain the character reference height information of the target row based on the height range information of the target string. In specific applications, the server may directly use the height range information of the target string as the character reference height information of the target row. In some embodiments, the server may obtain the average longitudinal coordinate point information of all strings, and based on the average longitudinal coordinate point information of all strings and the height range information of the target string, comprehensively obtain the character reference height information of the target row. For example, the server may weightedly fuse the average longitudinal coordinate point information of all strings and the height range information of the target string to obtain the character reference height information of the target row.

[0073] Furthermore, when the coordinate position information of the next character string meets the character reference height information, the next character string is divided into a target row, and the character reference height information is updated according to the coordinate position information of the next character string, including: determining the height range information of the next character string according to the coordinate position information of the next character string; determining the height information deviation of the height range information of the next character string relative to the character reference height information; when the height information deviation is less than a deviation threshold, the next character string is divided into a target row, and the character reference height information is updated according to the height range information of the next character string, and the deviation threshold is obtained based on the height information statistics of the fields included in each character string.

[0074] The height information deviation is obtained by comparing the height range information of the next character string with the character reference height information of the target row, and may specifically include the deviation value of the longitudinal coordinate point, such as the deviation value of the maximum longitudinal coordinate point and the deviation value of the minimum longitudinal coordinate point. The deviation threshold is used to divide the row to which the next character string belongs, and is specifically used to determine whether the next character string is divided into the target row or the row next to the target row. The specific value of the deviation threshold can be obtained based on the height information statistics of the fields included in each character string, such as the average height of the fields included in each character string can be used as the deviation threshold.

[0075] Optionally, the server can determine the coordinate position information of the next string, and determine the height range information of the next string based on the coordinate position information, which can specifically include the longitudinal coordinate point range information of the next string, such as the maximum longitudinal coordinate point and the minimum longitudinal coordinate point of the next string. The server can compare the height range information of the next string with the character reference height information of the target row to determine the height information deviation of the height range information of the next string relative to the character reference height information, such as determining the deviation value of the maximum longitudinal coordinate point and the deviation value of the minimum longitudinal coordinate point between the next string and the character reference height information. In some embodiments, the server can directly obtain the height information deviation based on the deviation value of the maximum longitudinal coordinate point and the deviation value of the minimum longitudinal coordinate point, or can screen based on the deviation value of the maximum longitudinal coordinate point and the deviation value of the minimum longitudinal coordinate point, such as the smaller value of the deviation value of the maximum longitudinal coordinate point and the deviation value of the minimum longitudinal coordinate point can be used as the height information deviation. The server can compare the height information deviation with a preset deviation threshold, and when it is determined that the height information deviation is less than the deviation threshold, it can be considered that the next string belongs to the target row, that is, the next string can be divided into the target row.

[0076] The server can update the character reference height information of the target row based on the height range information of the next character string, such as using the maximum longitudinal coordinate point with a larger value in the height range information of the next character string and the character reference height information as the updated maximum longitudinal coordinate point, and using the minimum longitudinal coordinate point with a smaller value as the updated minimum longitudinal coordinate point. The server can use the updated character reference height information to perform row division processing on subsequent character strings, and the character reference height information can be adaptively updated according to the height range information of the character string included in the target row, so that characters that are tilted by handwriting can also be accurately divided into rows. In some embodiments, when it is determined that the height information deviation is not less than the deviation threshold, it can be considered that the coordinate position information of the next character string does not meet the character reference height information of the target row, and the server can divide the next character string into the next row of the target row, and specifically, the next character string can be used as the first character string in the next row, thereby achieving row division processing for the next character string.

[0077] In this embodiment, the server determines the character reference height information of the target row based on the coordinate position information of the target character string. When the height range information of the next character string deviates from the height information of the character reference height information less than a deviation threshold, the next character string is divided into the target row and the character reference height information of the target row is updated. The height range information of the character string included in the target row can be used for adaptive updating, so that characters that are tilted by handwriting can also be accurately divided into rows, which is beneficial to improving the accuracy of document information extraction.

[0078] In an exemplary embodiment, according to the information group pattern of the document image, each string is divided into information groups based on the coordinate position information of each string and the row information of each string to obtain each information group that conforms to the information group pattern, including: determining the information group pattern of the document image, the information group pattern including the entity types that constitute the information group; when it is determined according to the information group pattern that the document image belongs to multiple information group types, performing entity recognition on each field in each string according to the entity type to obtain the entity type to which each field in each string belongs; determining the order of information group division for each field in each string based on the coordinate position information of each string and the row information of each string; according to the order of information group division for each field in each string, dividing each field in each string into information groups based on the entity type to which each field in each string belongs to obtain each information group that conforms to the information group pattern.

[0079] Among them, the document information type in the information group mode of the document image may include entity type and non-entity type. The entity type may specifically include document information of named entities, that is, it can be directly identified and distinguished according to the named entity based on the document information; while the non-entity type cannot be directly identified and distinguished based on the document information. The entity type can be distinguished according to the name of the document information. For example, the document information type of the information group mode includes the entity type of "commodity name". In the document image, it can be identified as the document information type belonging to "commodity name" based on the name "apple", "watermelon", "tomato", etc.; while for the document information of non-entity type, it may be numerical data in the document image, such as numerical data such as "1", "10", "15" in the document image. It is impossible to directly determine which document information type it belongs to in the information group mode based on the numerical value, and specifically it is impossible to determine whether it belongs to the document information type such as "unit price" or "quantity". Therefore, the information group division can be performed for each string field in the document image based on the document information type belonging to the entity type in the information group mode.

[0080] The multi-information group type means that at least one line in the document image includes multiple single information groups, that is, there is a repetition of the information group pattern in at least one line, and the repetition of the information group pattern can be repeated once or multiple times. When the information group division is performed based on the identified character strings, the fields in a character string may belong to the same information group or to multiple different information groups, that is, the division of information groups is implemented for the fields included in each character string, so as to divide each field in the document image into each information group. The information group division order refers to the order in which all fields are divided into different information groups, which can be obtained by sorting the fields, such as sorting according to the order of the rows to which the fields belong and the left-right order within the row, so as to obtain the information group division order of each field.

[0081] Exemplarily, the server may determine the information group pattern corresponding to the document image, and each document image may have a different information group pattern, that is, the document information content and display method carried in the document image may be different. The server may perform pattern recognition on the document image, such as performing pattern recognition on the document image through a pre-trained network model, to determine the information group pattern of the document image. In some embodiments, the server may also determine the various fields included in the string recognized by the document image according to the text feature rule to determine the information group pattern of the document image. In the information group pattern of the document image, the entity type constituting the information group may be included, and the corresponding document information may be identified from each field of the document image based on the entity type, thereby dividing each field into information groups based on the document information belonging to the entity type. The server may determine whether the document image includes a repetition of the information group pattern based on the information group pattern, and specifically determine whether multiple repetitions of the information group pattern appear in the same row of the document image, thereby determining whether the document image belongs to a multi-information group type. For example, the information group pattern is "product name-product specifications-product unit price", where "product name" and "product specifications" belong to the entity type. The server can determine from the document image whether any row includes multiple fields belonging to "product name", or whether any row includes multiple fields belonging to "product specifications". If so, it can be considered that the document image belongs to the multi-information group type. In this case, it is necessary to first divide the multi-information group into single information groups.

[0082] In some embodiments, the server determines that the document image does not belong to an information group type, but to a single information group type, that is, for each line in the document image, the fields included in the character string in the line all belong to the information group corresponding to the line, and the information group pattern does not repeat in the same line. The server can directly divide each line into an information group, thereby quickly dividing the information groups that conform to the information group pattern.

[0083] Furthermore, the server may perform entity recognition on the fields included in each string of the document image according to the entity type in the information group pattern, so as to determine the entity type to which each field in each string belongs. In some embodiments, not all fields in each string belong to the entity type, that is, they may include fields that belong to non-entity types. The server may first identify the entity fields from the fields included in each string based on entity recognition, and classify the identified entity fields according to the entity type in the information group pattern, so as to divide the entity fields into the entity types included in the information group pattern. In each single information group, there is at most one entity type included in the information group pattern, and no duplication occurs. Based on this, the entity type corresponding to each entity field can be used to divide the information group. Specifically, two entity fields belonging to the same entity type can be divided into two different information groups.

[0084] The server can determine the information group division order of the fields included in each string, and specifically can determine the division order according to the coordinate position information of each string and the row information of each string. For example, the server can determine the first division order of the fields in each row in the order from the first row to the last row, and for multiple fields in the same row, the server can determine the second division order according to the position relationship from left to right, so as to obtain the information group division order by combining the first division order and the second division order. The server can divide the information groups for the fields included in each string in turn according to the information group division order, and can refer to the entity type to which the field corresponds when dividing the information groups, so as to divide each field into different information groups, so that the information groups obtained by division conform to the information group mode, and specifically the information groups obtained by division can be single information groups conforming to the information group mode.

[0085] In this embodiment, the server divides each field into information groups based on the entity type to which the fields included in each string belong and the order of information group division determined based on the coordinate position information and row information. Based on the document information type to which each field in the document image belongs, each field can be divided into the corresponding information group, thereby ensuring the accuracy of information group division and further improving the accuracy of document information extraction.

[0086] In an exemplary embodiment, information groups that conform to an information group pattern are obtained by dividing each field in each string into information groups in an order of dividing each field in each string based on the entity type to which each field in each string belongs, including: building an entity list based on each field in each string in accordance with the information group division order of each field in each string; and dividing each field in the entity list into information groups in turn according to the entity type included in the information group pattern to obtain each information group that conforms to the information group pattern.

[0087] Among them, the entity list can be obtained by sorting the entity fields in all fields of the document image, and can be obtained by sorting according to the order of information group division. Exemplarily, the server can build the entity list based on each field according to the order of information group division, and divide the information group based on the entity list. In some embodiments, the fields included in each string can include entity fields belonging to the entity type and non-entity fields that do not belong to the entity type. The server can filter out the entity fields belonging to the entity type from all fields, and build the entity list based on the order of information group division of each entity field. The server can divide the fields in the entity list into information groups in turn according to the entity types included in the information group mode, so as to obtain each information group that conforms to the information group mode. In some embodiments, when the entity list is constructed by the entity fields belonging to the entity type, and each field also includes non-entity fields that do not belong to the entity type, such as including various numerical fields such as "15", "7", "150", etc., the server can divide each non-entity field into a corresponding information group based on the document information type in the information group mode, the positional relationship between the non-entity field and the entity field.

[0088] For example, if the information group pattern is "Product Name - Product Specification - Unit Price - Quantity", for the field "Apple Box 80 10 Oranges 6 catties 7", among which fields such as "apple", "box", "orange", and "jin" belong to entity types. Specifically, "apple" and "orange" belong to the document information type of "product name", and "box" and "jin" belong to the document information type of "product specification", that is, "apple" and "box" can be divided into information group A, and "orange" and "jin" can be divided into information group B. For numerical fields such as "80", "10", "6", and "7", the server can divide the numerical fields between the corresponding fields of the product names of information group A and the product names of information group B into information group A, corresponding to the document information types of "unit price" and "quantity", respectively, so that "80" and "10" can be divided into information group A, and "6" and "7" can be divided into information group B, and finally information group A [(product name) apple-(product specification) box-(unit price) 80-(quantity) 10)] and information group B [(product name) orange-(product specification) jin-(unit price) 6-(quantity) 7)] are obtained.

[0089] In this embodiment, the server constructs an entity list according to the order of information group division of each field, and divides information into groups for the fields in the entity list in sequence according to the entity types included in the information group pattern. It can divide information into groups for each field in sequence based on the entity types of the information group pattern, thereby ensuring the accuracy of information group division and further improving the accuracy of document information extraction.

[0090] In an exemplary embodiment, each information group is divided according to a column dimension to obtain column information of each information group, and based on the row information of the fields included in each information group, the row information of each information group is obtained, including: determining the information group position information of each information group according to the coordinate position information of the fields included in each information group; aligning each information group according to the column dimension based on the information group position information of each information group to divide each information group into columns to obtain the column information of each information group; determining the row information of the fields included in each information group based on the row information of the fields included in each character string; using the row information of the fields included in each information group as the row information of each information group.

[0091] Among them, the information group position information refers to the coordinate position information of the information group, which can be obtained based on the coordinate position information of each field included in the information group. For example, an external area can be constructed based on the coordinate position information of each field included in the information group, and the information group position information of the information group can be obtained based on the external area.

[0092] Optionally, after dividing each information group, the server can determine the row information and column information of each information group respectively. Each information group includes at least one field, and the server can determine the coordinate position information of the field included in the information group, and determine the information group position information of the information group based on the coordinate position information. After the server determines the information group position information of each information group, it can align the information group position information of each information group in the column dimension, thereby dividing each information group in the column dimension to obtain the column information of each information group. In some embodiments, the server can determine the information group boundary and the information group center line of the corresponding information group based on the information group position information, and the server can align the information group boundary and the information group center line of each information group, thereby realizing the column division for each information group. For the row dimension, the server can determine the row information of the field included in the information group, and use the row information of the field included in the information group as the row information of the information group. After traversing each information group, the server can obtain the row information of each information group.

[0093] In this embodiment, for each information group, the server determines the row information and column information based on the fields included in the information group, so that the information group can be accurately divided into rows and columns, which is conducive to improving the accuracy of document information extraction.

[0094] In an exemplary embodiment, after dividing each information group according to the column dimension, obtaining the column information of each information group, and obtaining the row information of each information group according to the row information of the fields included in each information group, the document information processing method also includes: when the document image belongs to a multi-information group type, sorting according to a preset sorting method based on the row information and column information of each information group, and updating the row information and column information of each information group according to the sorting result.

[0095] Among them, the sorting method is used to re-sort each information group, so that when the document image belongs to the type of multiple information groups, the rows with multiple information groups are split, so that each row corresponds to a single information group, which is conducive to extracting document information from the information group. Exemplarily, for a document image belonging to the type of multiple information groups, the server can obtain a pre-set sorting method, which is an arrangement and display method for each information group in the document image. Specifically, for each column, it can be sorted in the order of row numbers from small to large. For example, for the information groups in the first column, they can be sorted in the order from the first row to the last row; and then for the information groups in the second column, they can be sorted in the order from the first row to the last row, and finally the sorting result is obtained. The server can update the row information and column information of each information group based on the sorting result. By sorting each information group through the sorting method, each information group can be arranged into a list of information groups belonging to the same column, and the server can determine the row information and column information of each information group based on the information group list.

[0096] For example, when the document image includes 3 columns of information groups, the first column includes 20 rows, the second column includes 20 rows, and the third column includes 10 rows, the total number of information groups is 20+20+10, which is 50 in total. The server can sort the 50 information groups according to a preset sorting method. Specifically, the first column can be sorted in the order from the first row to the 20th row to obtain an information group list including 20 rows of information groups. For the second column, the server can sort and add the information groups in the order from the first row to the 20th row to the information group list, thereby obtaining an information group list including 40 rows of information groups; further, for the third column, the server can add the information groups in the order from the first row to the tenth row to the information group list, and finally obtain an information group list including 50 rows of information groups. These 50 information groups form a 1*50 information group list, that is, a 1-column 50-row information group list is obtained, and each row corresponds to an information group. The server can update the corresponding row information and column information based on each information group in the information group list.

[0097] In this embodiment, for document images of multiple information group types, the server can re-sort each information group according to a preset sorting method, and update the row information and column information of the information group according to the sorting result, so that the information group can be divided according to the sorting method, so that the document information can be extracted based on the re-sorted information groups, which is conducive to ensuring the accuracy of the document information extraction.

[0098] In an exemplary embodiment, when it is determined that the document image does not meet the table character recognition conditions based on the row information and column information of the fields included in each character string, the document information in the document image is obtained based on each information group and the row information and column information of each information group, including: determining the row information and column information of each character string according to the row information and column information of each field included in each character string; clustering each character string according to the row information and column information of each character string to obtain a character string clustering result; classifying the document image based on the string clustering result and each character string through a pre-trained classification model to obtain an image category of the document image; when the image category belongs to a category that does not meet the table character recognition conditions, generating the document information in the document image based on each information group and the row information and column information of each information group.

[0099] The string clustering result is obtained by clustering the strings, and specifically may include the clustering result obtained by clustering according to the row dimension and the column dimension. The classification model is used to classify the image category of the image. Specifically, the classification model can classify the document image according to the input string and the string clustering result to determine the image category of the document image. The classification model can be pre-trained according to the historical image category data, and the classification model can be constructed based on various neural network algorithms.

[0100] Exemplarily, the server may determine the row and column information of each field in the document image, and determine the row and column information of the string to which the field belongs based on the row and column information of the field, such as the server may use the row and column information of the field included in the string as the row and column information of the string. The server may cluster each string based on the row and column information of each string, and specifically may cluster according to the row dimension based on the row information, and cluster according to the column dimension based on the column information, so as to obtain the string clustering result. The string clustering result may include the clustering distance between different string categories and the statistical value for the clustering distance. The server may obtain a pre-trained classification model, specifically may include a text classification model, such as a fasttext model, and the server may classify the document image based on the string clustering result and each string through the classification model, so as to obtain the image category of the document image. In some embodiments, the server may input the string clustering result and each string into the classification model, so that the classification model may classify the image category of the document image, so as to determine the image category of the document image. The server can determine whether the document image meets the table character recognition conditions based on the image category of the document image, and specifically determine whether the image category of the document image is a category that meets the table character recognition conditions. When the document image does not meet the table character recognition conditions, it indicates that the document image is not suitable for document information extraction through table character recognition. The server can then generate the document information in the document image based on each information group and the corresponding row and column information.

[0101] In this embodiment, the server clusters each string and classifies the document image based on the string clustering results and the strings through a pre-trained classification model. When it is determined based on the image category of the document image that the document image does not meet the table character recognition conditions, the document information of the document image is directly generated based on each information group and their respective row and column information. The document information can be accurately extracted through handwritten character recognition, which can improve the processing efficiency of document information extraction while ensuring the accuracy of the document information.

[0102] In an exemplary embodiment, after determining the column information of the fields included in each character string based on the column information of each information group, the document information processing method also includes: when it is determined that the document image meets the table character recognition conditions based on the row information and column information of the fields included in each character string, the document image is recognized according to the table character recognition method to obtain the table character recognition result; and the document information in the document image is constructed based on the table character recognition result.

[0103] Among them, the table character recognition method is suitable for identifying the table text information in the document image to extract the document information. The table character recognition method can be implemented by calling the table OCR recognition interface. The table OCR recognition interface can accurately identify the table text information in the document image, including the text content, row and column structure, etc. in the table. It is suitable for identifying the table content with black borders and complete horizontal and vertical lines. In the scenario where a large amount of table data needs to be processed, it can greatly improve work efficiency.

[0104] Optionally, when it is determined that the document image meets the table character recognition conditions, that is, the document image is suitable for document information extraction directly through table character recognition, it indicates that the document image may carry table information, such as table borders, table row and column structures, etc., then the server can directly extract the document information according to the table character recognition method, and specifically can call the table OCR interface to perform recognition to obtain the table character recognition result, which is the character recognition result obtained by extracting document information from the document image according to the table character recognition method. The server can construct the document information of the document image based on the table character recognition result, and specifically the server can use the table character recognition result as the original document information, and generate the document information of the document image in combination with the corresponding reference information determined from the document management system.

[0105] In this embodiment, when the document image meets the table character recognition conditions, the server directly identifies the document image according to the table character recognition method, and constructs a corresponding document image according to the table character recognition result, so that the document information with table information can be extracted through the table character recognition method, which can ensure the accuracy of the document information while improving the processing efficiency of the document information extraction.

[0106] In an exemplary embodiment, Figure 3 As shown, the process of acquiring the document information, that is, obtaining the document information in the document image based on each information group and the row information and column information of each information group, includes steps 302 to 306.

[0107] in:

[0108] Step 302, performing glyph matching on the fields included in each information group and the standard fields of the document to obtain glyph matching results.

[0109] Among them, the document standard field is a standard field pre-set in the document management system, and the fields included in the document image are mapped to the document standard field, so that the document information can be generated through the standard field to ensure the uniformity of the document information. The glyph matching result is the result of matching the fields in the information group with the document standard field according to the character glyphs. Optionally, the server can obtain the preset document standard field, and perform glyph matching on the fields included in the information group with each document standard field respectively, and specifically calculate the character glyph edit distance between the fields in the information group and the document standard field, so as to obtain the glyph matching result.

[0110] Step 304: when the glyph matching result meets the correction determination condition, the fields included in each information group are corrected to obtain each corrected information group.

[0111] The correction judgment condition is used to determine whether it is necessary to correct the fields included in the information group. For example, when the fields in the information group do not match the standard fields in the document, it can be considered that there may be writing errors in the fields in the information group, and correction is required, that is, the correction judgment condition is met. Exemplarily, when the glyph matching result meets the correction judgment condition, such as when the character glyph editing distance between the fields in the information group and the standard fields in the document is less than a preset distance threshold, the server can correct the fields included in the information group, and specifically, the fields in the information group can be corrected based on the standard fields in the document, thereby obtaining an information group after the fields are corrected.

[0112] Step 306: Obtain the document information in the document image based on the corrected information groups and the corrected row information and column information of each information group.

[0113] Optionally, the server may extract the document information based on the corrected information group. Specifically, the server may extract the document information of the document image based on each corrected information group and each row and column information of each corrected information group.

[0114] In this embodiment, when the server determines that the correction judgment condition is met based on the glyph matching result between the document standard field and the field in the information group, it corrects the field included in the information group and obtains the document information in the document image based on the information group after the field correction, which can reduce the impact of field glyph errors and thus improve the accuracy of document information extraction.

[0115] The present application also provides an application scenario, which applies the above-mentioned document information processing method. Specifically, the application of the document information processing method in the application scenario is as follows:

[0116] For the document information extraction processing of document images, it is generally to identify the text information in the document image and perform document entry processing. The document information extraction system usually consists of three parts: OCR module, information extraction module, and post-processing module. Among them, the OCR module is used to convert image information into text information, the information extraction module is used to structure the text information and map it to the document management system, specifically to the business system field; the post-processing module is used for identification correction, fuzzy matching, field completion and other processing to ensure the accuracy of document information.

[0117] At present, in the document information extraction system, the user manually selects the mode and performs character recognition on the document image by calling the OCR interface corresponding to the selected mode. The user then manually selects the field mapping relationship to map the recognition result to the business system field, thereby extracting the document information. On the other hand, for fixed-plate document images, character recognition can be performed by calling the OCR customized recognition interface to extract the document information. For the extracted document information, field correction can be performed through ES (Elasticsearch) fuzzy matching, and the final document information can be fed back to the front end for intuitive display to the user. However, the document information extraction system is not flexible enough when processing various types of images, the effect is average, the user operation cost is high, and the user experience is not good.

[0118] With the emergence of Large Language Model (LLM) technology, some document information extraction systems directly call the OCR interface for recognition, and the large model outputs structured document information, and then corrects the fields through ES fuzzy matching, and finally feeds back the confirmed document information to the front end. However, the processing of directly calling the OCR interface for recognition still needs to solve the trade-off or selection problem of three interfaces, namely handwritten OCR recognition, table OCR recognition and OCR customized recognition. Its main disadvantages are: when there is a lot of text information in the picture, the output error rate of the large model is high, which cannot meet the needs of high-concurrency batch processing, and the output takes too long. The cost becomes more expensive as the number of characters in the picture increases, and the cost is relatively uncontrollable. It is difficult to construct a suitable text input form for the large model for complex table information, and it is not suitable for the rapid input of complex picture information.

[0119] Specifically, the OCR recognition required by the OCR module of the document information extraction system can be divided into three types: handwriting OCR recognition, table OCR recognition, and OCR customized recognition. Among them, table text recognition supports the recognition of regular simple tables with complete borders, can recognize tables without table borders but with clear row and column positions, and supports complex table text recognition with merged cells; OCR customized recognition: autonomously create recognition templates and classifiers for fixed-format cards and bills to recognize fixed-format images. However, for handwriting OCR recognition, its main disadvantage is that if there is table information in the image, there is no row and column information in the returned information, which causes difficulties in the subsequent information extraction stage; the main disadvantage of table OCR recognition is that if the background of the image is messy with cells, the text in the image will be affected by the background cells, etc., resulting in unnecessary segmentation problems between characters; the main disadvantage of OCR customized recognition is that it can only recognize fixed-format images, and it is necessary to list and collect all possible formats in advance. The costs of the above three types of recognition are, from low to high, handwriting OCR recognition, table OCR recognition, and OCR customized recognition.

[0120] In some application scenarios, there may be various problems in the document image, such as abbreviation replacement, header information interference, digital mark interference, information smearing, writing adhesion, irregular writing, text coverage, typos, footer information interference, various writing modes, etc. The various handwritten and handwritten mixed unit prices encountered in the recording scene are in various forms, making it impossible to accurately extract structured document information from the document image. Figure 4 As shown in , there are cluttered cells in the background of the document image; Figure 5 As shown, for Figure 4 When the document image is subjected to character recognition according to handwriting OCR recognition, the background cells do not affect the division of the text strings, that is, each string can be identified from the document image, specifically: Figure 5 The fields selected by the rectangular boxes in Figure 6 As shown, for Figure 4 When the document image is used for character recognition according to the table OCR recognition, the background cells seriously affect the division of the text string. Figure 7 As shown, when character recognition is performed on another document image through table OCR recognition, even if the document image is tilted, its row and column information is well recognized and divided, so that the document information can be accurately extracted.

[0121] Therefore, no OCR interface can accurately recognize every type of image (handwritten stand-alone tables, handwritten tables, printed text), and the images contain irrelevant information such as smudges, marks, digital modifications, notes, etc., which interfere with information extraction and easily recognize irrelevant information such as table headers and footers; when users do not write in accordance with the table frame specifications, that is, when there is irregular writing and disordered format, it is easy to cause the missing of recognized information, which affects the structure and extraction. In addition, the documents written by users are not completely consistent with the system products, and there are problems such as product borrowing and abbreviations. Even if the recognition is accurate, it is difficult to map the product information, resulting in the inability to distinguish between the recognized information and the supplier, which reduces the accuracy of the document information.

[0122] The document information processing method provided in this embodiment is as follows: Figure 8 As shown, the present embodiment can realize one-click order generation from multiple images to multiple orders, that is, one-click order generation can be performed on multiple document images to obtain the document information corresponding to each document image. Specifically, the document image can be recognized by calling the handwriting OCR interface, and the intelligent classification model can be used to make a judgment based on the recognition result to automatically select the corresponding OCR interface; further, information extraction and intelligent mapping can be performed on the recognition result to obtain the original document information; for the extracted original document information, text information intelligent error correction can be performed, and error correction can be performed through the user's personalized error correction table, and finally the final document information can be obtained after ES fuzzy matching processing, and the final document information can be sent to the front end for display.

[0123] Specifically, for the OCR module, various OCR recognition methods have their own advantages and disadvantages. In solving the problem of how to better select the OCR interface, this embodiment adopts the fasttext model that incorporates coordinate information to classify the category to which the picture belongs, reduces invalid repeated interface calls, improves performance and reduces costs. For the information extraction module, this embodiment solves the problem of information extraction difficulties caused by the lack of row and column information in handwritten OCR based on the row and column division algorithm of adaptive hyperparameters, so that the recognized string determines its row and column, and further determines its meaning information; further, this embodiment solves the problem of user handwritten pictures, writing freedom is very high, and often the same line has an indefinite number of "commodity name-quantity-unit" and other patterns repeated, and also supports the image recognition of the single information group pattern. In addition, this embodiment also combines natural language processing technology, and trains its own NER (Named Entity Recognition) model based on the commodity name, specification model, and unit of measurement data in the business field to solve the accuracy and performance of information extraction. After verification, the time to process a line of information is about 200 milliseconds, which is much more efficient than extracting using a large model. Specific as Fig. 9As shown, there is no row and column information in the handwritten document, which increases the difficulty of extracting document information; for example Fig.10 As shown, after OCR recognition, the fields with different meanings in the document image, namely "Product Name - Unit Price / Unit - Quantity", may be glued together into a single string. Specifically, "Head & Shoulders 15 / piece 10" is recognized as a single string, which includes three fields: "Head & Shoulders", "15 / piece", and "10". For the post - processing module, this embodiment is based on intelligent error correction processing of text information, and specifically adopts a systematic design such as confidence prediction + neural network + character similarity + preset word list to ensure the accuracy of document information.

[0124] The document information processing method provided in this embodiment can quickly and at low cost select a suitable interface for OCR recognition regardless of whether the user inputs a handwritten picture, a machine - printed picture, a mixed handwritten and machine - printed picture, various types of pictures with framed tables, various pictures with unframed tables, multi - information - group pictures, or single - information - group pictures. It can also solve some drawbacks of the corresponding OCR recognition interfaces, overcome problems such as user typos, abbreviations, and incomplete information, and through intelligent algorithms and designs, while improving the user's picture entry experience, it achieves a cheaper and faster effect.

[0125] The document information processing method provided in this embodiment involves an adaptive hyperparameter row - column division algorithm. Specifically, as Fig. 9 shown, the string recognized by handwritten OCR is selected by a rectangular frame. For example, in the second row, there are three strings ["Apple", "5", "9"]. For handwritten OCR recognition, only the four - point information of the rectangle around the string and the rectangular coordinate information of each character are available. It is necessary to determine that the three strings "Apple", "5", and "9" are in the same row, and "5", "90", and "21" are in the same column. Fig. 9 For the document image in, table recognition can be called. In the case where the above - mentioned document image is written in a relatively standard way, this document image may be determined by table recognition as an unframed table, and thus a table information with string row - column information is returned. However, this solution has a high cost, while the document information processing method provided in this embodiment has a lower cost and stable effects on unframed tables. Specifically, it adopts "handwritten OCR" + "adaptive hyperparameter row - column division algorithm" to achieve the effect of table OCR recognition, and at the same time overcomes the problem of fragmented character segmentation of the background picture with cell information in handwritten table recognition. Among them, "adaptive" means that regardless of the size of handwritten characters, row and column information can be well divided.

[0126] Specifically, in the first step, the length and width of each character are globally counted, and the average horizontal length and vertical width of each character in all character strings are calculated. This average length and width characterize the minimum pixel distance of the average column and the minimum pixel distance of the average row of the characters in the document image recognized this time, which can be calculated as Qx and Qy respectively. Next, according to the OCR recognition results, the rows are first divided. According to the handwritten OCR recognition results, the coordinates of the four points of the "upper left corner", "upper right corner", "lower left corner" and "lower right corner" of the string "apple" are obtained. If this is the first string of this row, then the maximum vertical coordinate point My1 and the minimum vertical coordinate point Ny1 are calculated at this time. The maximum vertical coordinate point My=My1 and the minimum vertical coordinate point Ny=Ny1 of this row. Then the coordinates of the four points of the "upper left corner", "upper right corner", "lower left corner" and "lower right corner" of the next string "5" are judged in turn, and the maximum vertical coordinate point My2 and the minimum vertical coordinate point Ny2 for the current "5" are calculated.

[0127] If min(abs(My-My2), abs(Ny-Ny2)) is determined, if the result of this equation is less than the self-adaptive hyperparameter Qy, then it is determined to be the same row, otherwise it is determined to be a new row. If it is the same row, the maximum vertical coordinate point that needs to be updated at this time is My=max(My1,My2), and the minimum vertical coordinate point is Ny=min(Ny1,Ny2). The row judgment of the third string is carried out in sequence until it does not belong to this row. At this time, the maximum and minimum vertical coordinate points of this row are cleared and reinitialized.

[0128] The reason for the instant update is that the user's handwriting is often tilted, but the floating range of the tilt between two consecutive character strings is relatively controllable. With the continuous update of the parameters of the maximum and minimum vertical coordinate points in the same line, this kind of very long handwriting tilt can be judged as the same line.

[0129] Based on the row division of each character string in the document image, the row information of each character string can be obtained, and the row information of the fields included in the character string can be further determined. According to the row information of the fields included in the character string, information group division can be performed for each field in the document image, that is, each information group is divided from the document image.

[0130] For information groups in the document image, such as Fig.11 As shown, the document image includes a single information group, that is, each row corresponds to an information group, and there is no repetition of the information group pattern in any row; Fig.12As shown, the receipt image belongs to the multi-information group type, and there are indefinite repetitions of the information group pattern in the receipt image. In addition, the handwriting of the handwritten image is variable, and it is often recognized as a character string as the user's writing interval becomes smaller.

[0131] The uncertain number of repetitions of the pattern makes it difficult to map the identified information to the system’s keywords. Fig.10 The "Aiwusheng cold perm 30 / piece 20" is a single information group, but Fig.12 The "7 bottles of nuts and 6 pounds of papaya" is a multiple information group, which is the "product name-quantity-unit" pattern repeated twice. At the same time, there are rows repeated 3 times in the figure. This number is indefinite and is called a "multiple information group". Multiple information groups mean that the recognition results of handwritten OCR are often stuck or split. Although the character strings in the same row can be divided into rows and columns, how to split the information groups is a problem that must be solved for information entry. At the same time, because there are multiple information groups in the same row, it is easy to check the information because the alignment of the fields is not good, and the user needs to remember whether the row between two rows has been checked. In order to optimize the convenience of user checking, the greedy algorithm of the multiple information groups in this embodiment can realize the information groups of the priority column, from top to bottom, then the second column, from top to bottom, and so on.

[0132] Specifically, the document image provided by the user is very flexible. The pattern information of the document image can be obtained by using the fasttext classification model / text feature rule judgment / user selection, for example, the current document image is obtained as a multi-information group of the "product name-quantity-unit" pattern. Then, the row and column partitioning algorithm is used to divide the input text into corresponding rows, and a method is constructed to record the position information of each character, so that the position information and row information of each string can be determined.

[0133] Furthermore, when performing multi-information group segmentation processing, the pre-trained NER model can be used to identify the two types of information, "product name" and "product specification", and obtain the entity list entity_list. The specific processing logic can be as follows:

[0134] 1. Initialization: empty product name list name_list, empty product specification list spec_list;

[0135] 2. Traverse the entity list entity_list, that is, obtain the fields from the entity list entity_list in turn for processing;

[0136] Branch 1: When the current field is of product name type and the current field is the first field, directly add the field to name_list.

[0137] Otherwise, determine whether the information group has a product name.

[0138] If the information group does not have a product name, when the product specification list is not empty, the task current product name is the start position of the new information group, parse the previous information group, add it to the information group list, and then add the product name to the product

[0139] List of names;

[0140] When the product specification is empty, only add the product name to the product name list;

[0141] When a product name already exists in an information group, search for the content between the product name and the previous product name. If there is a space or specification information, it is considered that a new information group has appeared. The previous information group is parsed and added to the information group list. Then the product name is added to the product name list. Otherwise, the current product name and the previous product name are considered to belong to the same product and are concatenated.

[0142] Branch 2: The current field is the product specification type. When the information group is empty, directly add the field to spec_list; judge the current name_list. If it is empty, it means that the first field is the product specification, then the product name recognition is missed, add an empty string as the product name, and record the product name coordinates, using the starting and ending coordinates of the product specification at this time.

[0143] After dividing each single information group, you can divide it into columns. Specifically, after dividing the information groups, the interval coordinates of the information groups are known, and the above logical downstream information is known. You can rearrange multiple information groups, that is, align them through the position information of each information group to divide the column information of the information groups. Specifically, traverse all information groups, first initialize the first information group as the first row, judge the next information group, and each time judge the element whose midline position is the most similar column among all information groups (the judgment method is to traverse other information groups to see whether the midline coordinates of the current information group are within the left and right boundaries of other information groups. If so, return this element; secondly, return the midline of the current information group that falls within one of the left and right boundaries of other information groups; if neither of the above exists, return None, and the currently judged information group is a new column). Get the element of the most similar column. If it exists: judge the row where the element of the current judgment information group is located, take out all the elements of this row, and calculate the maximum column value of the elements in this row. If the original column of the most similar column is equal to the maximum column value of the elements in this row, then the column number of the currently judged column is the column of the most similar column element + 1, otherwise it is the column value of the most similar element. After completing the division of rows and columns for the information group, the table information will be organized into a two-dimensional string list to facilitate subsequent information extraction.

[0144] For the single information group obtained by division, first pre-process the characters and convert all Chinese numbers into Arabic numbers. At this time, it is a single information group, and the product name field in this information group has been determined. First, remove the product name field to obtain a string, so what needs to be done is to determine the content and corresponding meaning of other fields; specifically find the mode information of the current image, use Arabic numbers to cut (cut the quantity, unit, unit price, amount field), and then try multiple combinations to cut the field, and then find the optimal solution from the results of these combinations. The optimal result evaluation criteria may include: the largest number of fields and the value of each field that best meets the requirements.

[0145] Finally, according to the analysis results and positions of the information groups, the rows and columns are re-divided and re-sorted. For the multi-information group scenario, the information group of the first column can be output first, from top to bottom, then the second column, from top to bottom, and so on.

[0146] Furthermore, the document information processing method provided in this embodiment adopts the fasttext model that incorporates coordinate information to perform automatic category determination for the image OCR interface call. Specifically, the fasttext model is often used for text classification. This embodiment adopts its model in the order recording system. After the handwritten OCR recognizes the text information, special word segmentation information is designed, and features with position information are constructed to enter the fasttext model for image classification, so as to determine whether it is necessary to call the table OCR interface, thereby reducing the technical cost of the order recording system without introducing additional user operations. This solution is also applied to pattern recognition and classification corresponding to multiple information groups of images.

[0147] Specifically, the commonly used fasttext system is used for text classification, and often uses a word segmenter for word segmentation. However, the OCR result of the document image recognition in this embodiment has many strings. The reason why these strings are segmented is that they carry position information. Therefore, before word segmentation, the spacing information of these strings that imply position information must be retained. The word segmentation in this embodiment forces different strings to be in different words, rather than simply concatenating the strings in advance and then segmenting them. This design is caused by the fact that this embodiment is used for image recognition. Unlike traditional text classification, its word segmentation information is often included in punctuation and words, and the string spacing of image OCR is very important for word segmentation.

[0148] Specifically, for the string row and column information obtained by the previous adaptive row and column division algorithm, cluster the vertical and horizontal strings of the string, then sort the classes with the number of clusters greater than the threshold by the center point coordinates, then calculate the pixel distance between the classes, and normalize it with the adaptive parameter. The distances between the clustering points obtained with the adaptive parameter above are statistical values of the position information of the picture and contain position statistical information. In order to enable the fasttext classification model to distinguish the digital information in the picture and the constructed statistical digital information, the clustering points can be sorted according to the row and column information first, the distance between adjacent clustering points is counted, and the corresponding prefix is constructed before the statistical value. If it is a row, it is "clustering distance, row + statistical value", and if it is a column, it is "clustering distance, column + statistical value". In this way, the unique position information of the picture recognition system can be further incorporated into the classification model, so as to perform image classification on the document image to determine whether it meets the table character recognition conditions.

[0149] Furthermore, this embodiment performs intelligent error correction based on confidence prediction + neural network + character similarity + preset word list, etc. Specifically, it adopts the idea of the open-source OCR-Correct project. IDS (Ideographic Description Sequence) refers to representing ideographic characters with sequences, similar to representing English words with letter sequences, and Chinese characters can be represented with strokes and structural symbols. In this embodiment, it may include a character IDS file for the Chinese character set, which is used to calculate the character glyph edit distance between two characters. As Fig.13 shown, in the description of splitting the character "贫" based on IDS, the character "贫" can be split according to its glyph for glyph matching, which can be specifically used to calculate the character glyph edit distance.

[0150] Common order entry systems often use ES fuzzy matching for phonetic similarity, character similarity, or introduce semantic similarity using the latest deep learning after the OCR result. On this basis, this embodiment also newly introduces the concept of glyph similarity into the system. For the OCR result of this embodiment, calculate the glyph edit distance between the characters with a confidence level lower than the threshold and the IDS of the candidate words in the system word list. When it meets the edit distance threshold, select the character with the smallest edit distance for trial correction, and use ES fuzzy matching to test its error correction effect. For example, "理鱼" can be corrected to "鲤鱼", and "尤鱼" can be corrected to "鱿鱼", etc. The document information processing method of this embodiment is based on the user personalized error correction table module. Before ES fuzzy matching, the common misspelled words of this user can be automatically corrected according to the user's historical correction results to improve the matching effect.

[0151] In a specific application, such as Fig.14As shown in FIG. 1 , it is a document image for which document information extraction is required, and the document image includes various handwritten information such as commodity names, quantities, unit prices, and amounts; Fig.15 As shown, part of the document information extracted by the document information processing method based on this embodiment, specifically includes the document information of various commodities such as "Branch Pumpkin - Tea Can Packing" and "Branch Pumpkin - Washed Edition".

[0152] The document information processing method of this embodiment, based on the row and column division algorithm of adaptive hyperparameters, can solve the problem of difficulty in information extraction caused by the lack of row and column information in handwritten OCR recognition; further, the greedy algorithm combined with the multi-information group of NER can solve the problem of difficulty in extraction caused by the flexible and changeable writing method and mode of users. Moreover, this embodiment adopts a fasttext model that incorporates special processing coordinate information features, and designs a fasttext feature structure suitable for the image recording system to classify the interface category and image mode applicable to the image OCR, which can reduce the user confirmation steps, and has fast speed, low cost and good user experience. In addition, this embodiment adopts the method of confidence prejudgment + neural network + character similarity + pre-set vocabulary to perform error correction. On the basis of pre-OCR recognition, it not only incorporates the confidence information of OCR recognition to assist error correction, but also incorporates the glyph information of Chinese characters, and at the same time incorporates the pre-configured vocabulary to control the correction range to calculate the edit distance to perform glyph error correction, which can ensure the accuracy of document information extraction.

[0153] The document information processing method of this embodiment makes it possible to generate documents with one click. There is no need for the user to perform other manual operations. A business document can be generated with just one click. What is achieved is that no matter whether the user inputs handwritten pictures, machine-printed pictures, mixed pictures of handwritten and machine-printed pictures, various types of framed table pictures, various frameless table pictures, multi-information group pictures, or single information group pictures, the appropriate interface can be selected for OCR recognition quickly and at low cost, and some shortcomings of the corresponding OCR recognition interface can be solved, and problems such as typos, abbreviations, incomplete information, etc. of users can be overcome. Through intelligent algorithms and designs, the user's picture recording experience can be improved while achieving cheaper and faster results.

[0154] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0155] Based on the same inventive concept, the embodiment of the present application also provides a document information processing device for implementing the document information processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more document information processing device embodiments provided below can refer to the limitations of the document information processing method above, and will not be repeated here.

[0156] In an exemplary embodiment, Fig.16 As shown, a document information processing device 1600 is provided, comprising: a handwritten character recognition module 1602, a string row division module 1604, an information group division module 1606, a row and column information determination module 1608, a field column information determination module 1610 and a document information acquisition module 1612, wherein:

[0157] The handwritten character recognition module 1602 is used to obtain a document image, and perform recognition on the document image according to the handwritten character recognition method to obtain a recognition result of the document image, wherein the recognition result includes each character string and the coordinate position information of each character string;

[0158] The string row division module 1604 is used to divide each string into rows based on the coordinate position information of each string, and obtain row information of the fields included in each string;

[0159] An information group division module 1606 is used to divide each character string into information groups according to the information group pattern of the document image based on the coordinate position information of each character string and the row information of each character string, so as to obtain each information group that conforms to the information group pattern;

[0160] The row and column information determination module 1608 is used to divide each information group according to the column dimension, obtain the column information of each information group, and obtain the row information of each information group according to the row information of the fields included in each information group;

[0161] The field column information determination module 1610 is used to determine the column information of the fields included in each character string according to the column information of each information group;

[0162] The document information acquisition module 1612 is used to obtain the document information in the document image based on each information group and the row information and column information of each information group when it is determined that the document image does not meet the table character recognition conditions based on the row information and column information of the fields included in each character string.

[0163] In one embodiment, the string row division module 1604 is also used to determine the first string of the first row from each string, take the first row as the target row, and take the first string in the target row as the target string; determine the character reference height information of the target row according to the coordinate position information of the target string; determine the next string adjacent to the target string from each string; when the coordinate position information of the next string meets the character reference height information, divide the next string into the target row, update the character reference height information according to the coordinate position information of the next string, update the next string to the target string, and return to execute the step of determining the next string adjacent to the target string from each string; when the coordinate position information of the next string does not meet the character reference height information, determine the next string as the first string in the next row of the target row, update the next row of the target row to the target row, and return to execute the step of taking the first string in the target row as the target string; when the row division for each string is completed, according to the row to which each string belongs after division, the row information of the fields included in each string is obtained.

[0164] In one embodiment, the string row division module 1604 is also used to determine the height range information of the target string based on the coordinate position information of the target string; obtain the character reference height information of the target row based on the height range information of the target string; determine the height range information of the next string based on the coordinate position information of the next string; determine the height information deviation of the height range information of the next string relative to the character reference height information; when the height information deviation is less than the deviation threshold, divide the next string into the target row, and update the character reference height information based on the height range information of the next string, and the deviation threshold is obtained based on the height information statistics of the fields included in each string.

[0165] In one embodiment, the information group division module 1606 is also used to determine the information group pattern of the document image, where the information group pattern includes entity types that constitute the information groups; when it is determined that the document image belongs to multiple information group types according to the information group pattern, entity recognition is performed on each field in each string according to the entity type to obtain the entity type to which each field in each string belongs; based on the coordinate position information of each string and the row information of each string, the information group division order of each field in each string is determined; according to the information group division order of each field in each string, information group division is performed on each field in each string based on the entity type to which each field in each string belongs to obtain each information group that conforms to the information group pattern.

[0166] In one embodiment, the information group division module 1606 is also used to build an entity list based on each field in each string according to the information group division order of each field in each string; and to divide the information into groups for each field in the entity list in turn according to the entity type included in the information group pattern to obtain each information group that conforms to the information group pattern.

[0167] In one embodiment, the row and column information determination module 1608 is further used to determine the information group position information of each information group based on the coordinate position information of the fields included in each information group; align each information group according to the column dimension based on the information group position information of each information group to divide each information group into columns to obtain the column information of each information group; determine the row information of the fields included in each information group based on the row information of the fields included in each character string; and use the row information of the fields included in each information group as the row information of each information group.

[0168] In one embodiment, it also includes an information group rearrangement module, which is used to sort the document image according to a preset sorting method based on the row information and column information of each information group when the document image belongs to a multi-information group type, and update the row information and column information of each information group according to the sorting result.

[0169] In one embodiment, the document information acquisition module 1612 is further used to determine the row information and column information of each string according to the row information and column information of the fields included in each string; cluster each string according to the row information and column information of each string to obtain the string clustering result; classify the document image based on the string clustering result and each string through a pre-trained classification model to obtain the image category of the document image; when the image category belongs to a category that does not meet the table character recognition conditions, generate the document information in the document image based on each information group and the row information and column information of each information group.

[0170] In one embodiment, the document information acquisition module 1612 is also used to determine that the document image meets the table character recognition conditions based on the row information and column information of the fields included in each character string, recognize the document image according to the table character recognition method, and obtain the table character recognition result; based on the table character recognition result, construct the document information in the document image.

[0171] In one embodiment, the document information acquisition module 1612 is also used to perform glyph matching on the fields included in each information group with the document standard fields to obtain glyph matching results; when the glyph matching results meet the correction judgment conditions, the fields included in each information group are corrected to obtain corrected information groups; based on the corrected information groups and the row information and column information of each corrected information group, the document information in the document image is obtained.

[0172] Each module in the document information processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0173] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.17 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various data involved in the document information processing method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a document information processing method is implemented.

[0174] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.18As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a document information processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0175] Those skilled in the art will understand that Fig.17 The structure shown in or 18 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0178] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0180] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A document information processing method, characterized in that: The method comprises: Acquire a document image, and perform recognition on the document image according to a handwritten character recognition method to obtain a recognition result of the document image, wherein the recognition result includes each character string and coordinate position information of each character string; Dividing each string into rows based on the coordinate position information of each string, and obtaining row information of fields included in each string; According to the information group pattern of the document image, based on the coordinate position information of each character string and the row information of each character string, the character strings are divided into information groups to obtain information groups that conform to the information group pattern; Divide the information groups according to the column dimension to obtain the column information of the information groups, and obtain the row information of the information groups according to the row information of the fields included in the information groups; determine the column information of the fields included in the character strings according to the column information of the information groups; When it is determined that the document image does not meet the table character recognition condition based on the row information and column information of the fields included in each of the character strings, the document information in the document image is obtained based on the information groups and the row information and column information of each of the information groups.

2. The method according to claim 1, characterized in that The dividing each string into rows based on the coordinate position information of each string to obtain row information of each field included in each string includes: Determine the first character string in the first row from the various character strings, take the first row as the target row, and take the first character string in the target row as the target character string; Determine character reference height information of the target row according to the coordinate position information of the target character string; Determine the next character string adjacent to the target character string from the respective character strings; When the coordinate position information of the next character string meets the character reference height information, the next character string is divided into the target row, the character reference height information is updated according to the coordinate position information of the next character string, the next character string is updated as the target character string, and the step of determining the next character string adjacent to the target character string from the various character strings is returned to be executed; When the coordinate position information of the next character string does not conform to the character reference height information, the next character string is determined as the first character string in the next row of the target row, the next row of the target row is updated as the target row, and the step of taking the first character string in the target row as the target character string is returned to be executed; When the row division for each of the character strings is completed, the row information of the fields included in each of the character strings is obtained according to the row to which each of the character strings belongs after the division.

3. The method according to claim 2, characterized in that The step of determining the character reference height information of the target row according to the coordinate position information of the target character string includes: Determining height range information of the target character string according to the coordinate position information of the target character string; Obtaining character reference height information of the target row based on the height range information of the target character string; When the coordinate position information of the next character string meets the character reference height information, dividing the next character string into the target row, and updating the character reference height information according to the coordinate position information of the next character string, comprises: Determine the height range information of the next character string according to the coordinate position information of the next character string; Determine a height information deviation of the height range information of the next character string relative to the character reference height information; When the height information deviation is less than a deviation threshold, the next character string is divided into the target row, and the character reference height information is updated according to the height range information of the next character string, and the deviation threshold is obtained based on the height information statistics of the fields included in each character string.

4. The method according to claim 1, characterized in that: The step of dividing each character string into information groups according to the information group pattern of the document image and based on the coordinate position information of each character string and the row information of each character string to obtain each information group conforming to the information group pattern includes: determining an information group pattern of the document image, wherein the information group pattern includes entity types constituting an information group; When it is determined according to the information group mode that the document image belongs to a multi-information group type, entity recognition is performed on each field in each string according to the entity type to obtain the entity type to which each field in each string belongs; Determine the order of dividing the information groups of the fields in the respective strings based on the coordinate position information of the respective strings and the row information of the respective strings; According to the information group division order of each field in each string, information group division is performed for each field in each string based on the entity type to which each field in each string belongs, so as to obtain each information group conforming to the information group pattern.

5. The method according to claim 4, characterized in that The information group division is performed for each field in each string according to the information group division order of each field in each string and based on the entity type to which each field in each string belongs, to obtain each information group conforming to the information group mode, including: According to the order of dividing the information groups of the fields in the strings, constructing an entity list based on the fields in the strings; According to the entity types included in the information group pattern, information groups are divided for each field in the entity list in turn to obtain each information group that complies with the information group pattern.

6. The method according to claim 1, characterized in that The dividing the information groups according to the column dimension to obtain the column information of the information groups, and obtaining the row information of the information groups according to the row information of the fields included in the information groups, includes: Determine the information group position information of each of the information groups according to the coordinate position information of the fields included in each of the information groups; Based on the information group position information of each information group, the information groups are aligned according to the column dimension to divide the information groups into columns, thereby obtaining the column information of each information group; Determine the row information of the fields included in each of the information groups based on the row information of the fields included in each of the character strings; The row information of the fields included in each of the information groups is used as the row information of each of the information groups.

7. The method according to claim 1, characterized in that After dividing the information groups according to the column dimension to obtain the column information of the information groups, and obtaining the row information of the information groups according to the row information of the fields included in the information groups, the method further includes: When the document image belongs to a multi-information group type, the information groups are sorted according to a preset sorting method based on their respective row information and column information, and the row information and column information of each information group are updated according to the sorting result.

8. The method according to claim 1, characterized in that When it is determined that the document image does not meet the table character recognition condition based on the row information and column information of the fields respectively included in the respective character strings, obtaining the document information in the document image based on the respective information groups and the row information and column information of the respective information groups includes: Determine the row information and column information of each character string according to the row information and column information of the fields included in each character string; Clustering the strings according to their respective row information and column information to obtain a string clustering result; Using a pre-trained classification model, classifying the document image based on the string clustering result and the individual strings, to obtain an image category of the document image; When the image category belongs to a category that does not satisfy the table character recognition condition, the document information in the document image is generated based on the respective information groups and the row information and column information of the respective information groups.

9. The method according to claim 1, characterized in that: After determining the column information of the fields included in each of the character strings according to the column information of each of the information groups, the method further includes: When it is determined that the document image meets the table character recognition condition based on the row information and column information of the fields included in each of the character strings, the document image is recognized according to the table character recognition method to obtain a table character recognition result; The document information in the document image is constructed based on the table character recognition result.

10. The method according to any one of claims 1 to 9, characterized in that: The obtaining the document information in the document image based on the respective information groups and the respective row information and column information of the respective information groups comprises: Performing font matching on the fields included in each of the information groups and the standard fields of the document to obtain font matching results; When the glyph matching result meets the correction determination condition, correcting the fields included in each of the information groups to obtain each corrected information group; Based on the corrected information groups and the row information and column information of the corrected information groups, the document information in the document image is obtained.

11. A document information processing device, characterized in that: The device comprises: A handwritten character recognition module, used to obtain a document image, recognize the document image according to a handwritten character recognition method, and obtain a recognition result of the document image, wherein the recognition result includes each character string and the coordinate position information of each character string; A string row division module, used for dividing each string into rows based on the coordinate position information of each string, and obtaining row information of fields included in each string; An information group division module, configured to divide each character string into information groups according to the information group mode of the document image, based on the coordinate position information of each character string and the row information of each character string, to obtain each information group conforming to the information group mode; A row and column information determination module, used to divide the information groups according to the column dimension, obtain the column information of the information groups, and obtain the row information of the information groups according to the row information of the fields included in the information groups; A field column information determination module, used to determine the column information of the fields included in each of the character strings according to the column information of each of the information groups; A document information acquisition module is used to obtain the document information in the document image based on the various information groups and the respective row information and column information of the various information groups when it is determined that the document image does not meet the table character recognition conditions based on the row information and column information of the fields respectively included in the various character strings.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images