Invoice generation method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of computer vision, in particular to a bill generation method and device, electronic equipment and storage medium, which are used for saving bill input time and improving bill input efficiency. The method comprises the following steps: performing posture correction on an obtained bill image; detecting the coordinate information of a plurality of text blocks in the bill image, wherein each text block comprises at least one text line; performing character recognition on any text block to obtain a character sequence in the text block; detecting whether the bill structure in the bill image comprises a table, and if yes, obtaining the coordinate information of a plurality of cells in the detected table; and generating a bill corresponding to the bill image based on the coordinate information of the plurality of text blocks, the character sequence corresponding to the plurality of text blocks, the plurality of cells and the coordinate information of the plurality of cells.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a bill generation method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In some banking businesses, banks need to issue bills and audit the bills. For example, in international settlement business, banks need to issue letters of credit and audit the bills corresponding to the letters of credit. After the bills are submitted to the bill center, the bill information needs to be first entered, and then the bills are audited by manual according to international conventions and practical rules, with the principle of "single consistency" and "document consistency".

[0003] The traditional bill information entry is completed by manual, however, the types and formats of bills are various, and the information entry process is very time-consuming and inefficient. SUMMARY

[0004] Embodiments of the present application provide a bill generation method and device, electronic equipment and a storage medium, which are used to save the bill entry time and improve the bill entry efficiency.

[0005] In a first aspect, the embodiments of the present application provide a bill generation method, which comprises:

[0006] performing posture correction on the obtained bill image;

[0007] detecting coordinate information of each text block in the corrected bill image, each text block containing at least one text line;

[0008] performing character recognition on any text block to obtain a character sequence in the text block;

[0009] detecting whether the bill structure in the bill image contains a table, and if the table is contained, obtaining coordinate information of each cell in the detected table;

[0010] generating a bill corresponding to the bill image based on the coordinate information of each text block, the character sequence corresponding to each text block, the table and the coordinate information of each cell.

[0011] In a possible embodiment, the posture correction on the obtained bill image comprises:

[0012] detecting heights and widths of a plurality of text lines in the image of the document, and if the heights and widths of the plurality of text lines satisfy a first preset condition, rotating the image of the document by a first preset angle; and detecting inclinations of a plurality of text blocks in the image of the document, and based on the inclinations of the plurality of text blocks, performing inclination correction on the image of the document.

[0013] detecting a plurality of text lines in the image of the document, and for a plurality of target text lines in the plurality of text lines, respectively performing character recognition to obtain a character confidence of each of the plurality of target text lines, and if it is determined that characters in the image of the document are in an upside-down state based on the character confidence of each of the plurality of target text lines, rotating the image of the document by a second preset angle.

[0014] In a possible embodiment, the rotating the image of the document by the first preset angle if the heights and widths of the plurality of text lines satisfy the first preset condition comprises:

[0015] determining a first number of text lines with a height greater than a width and a second number of text lines with a height less than a width;

[0016] if the first number is greater than the second number, rotating the image of the document by the first preset angle.

[0017] In a possible embodiment, the performing inclination correction on the image of the document based on the inclinations of the plurality of text blocks comprises:

[0018] based on the inclinations of the plurality of text blocks, determining an average inclination of the plurality of text blocks;

[0019] rotating the image of the document by a corresponding angle according to the average inclination.

[0020] In a possible embodiment, the for a plurality of target text lines in the plurality of text lines, respectively performing character recognition to obtain a character confidence of each of the plurality of target text lines comprises:

[0021] for any one of the target text lines, performing character recognition on the target text line to obtain a first character confidence of the target text line, and after rotating the target text line by the second preset angle, performing character recognition on the rotated target text line to obtain a second character confidence of the rotated target text line;

[0022] taking the first character confidence and the second character confidence of any one of the target text lines as the character confidence of the target text line.

[0023] In a possible embodiment, the determining that the characters in the document image are in an upside-down state based on the character confidence of each of the target text lines comprises:

[0024] determining a third number of target text lines whose first character confidence reaches a preset value, and determining a fourth number of target text lines whose second character confidence reaches the preset value;

[0025] if the third number and the fourth number satisfy a preset number condition, determining that the characters in the document image are in an upside-down state.

[0026] In a possible embodiment, the detecting the coordinate information of each of the text blocks in the document image comprises:

[0027] detecting the coordinate information of each of the text blocks in the document image based on a text detection model;

[0028] the character recognition on the text blocks to obtain the character sequence in the text blocks comprises:

[0029] recognizing the characters in the text blocks based on a character recognition model to obtain the character sequence in the text blocks;

[0030] the detecting whether the document structure in the document image contains a table comprises:

[0031] detecting whether the document structure in the document image contains a table based on a target detection model.

[0032] In a possible embodiment, the text detection model comprises a text line detection module and a text block positioning module;

[0033] then the detecting the coordinate information of each of the text blocks in the document image based on the text detection model comprises:

[0034] detecting the coordinate information of each of the text lines in the document image by the text line detection module;

[0035] based on the coordinate information of each of the text lines, dividing the text lines into the text blocks by the text block positioning module, and for any text block, obtaining the coordinate information of the text block based on the coordinate information of at least one text line included in the text block.

[0036] In a possible embodiment, the generating the document corresponding to the document image based on the coordinate information of each of the text blocks, the character sequence corresponding to each of the text blocks, the plurality of cells, and the coordinate information of each of the plurality of cells comprises:

[0037] generate a bill form based on the plurality of cells and the coordinate information of each of the plurality of cells by using a hypertext markup language (HTML);

[0038] add the character sequence corresponding to each of the text blocks to a corresponding position in the bill form based on the coordinate information of each of the text blocks, and generate a bill corresponding to the bill image.

[0039] In a possible embodiment, the method further includes:

[0040] if the bill structure in the bill image does not include a form, generating a bill corresponding to the bill image based on the coordinate information of each of the plurality of text blocks and the character sequence corresponding to each of the plurality of text blocks.

[0041] In a second aspect, the present application provides a bill generation device, which includes:

[0042] a correction module configured to perform pose correction on the obtained bill image;

[0043] a text detection module configured to detect, for the corrected bill image, coordinate information of a plurality of text blocks in the bill image, each of the text blocks including at least one text line;

[0044] a character recognition module configured to perform character recognition on any of the text blocks to obtain a character sequence in the text block;

[0045] a form detection module configured to detect whether a bill structure in the bill image includes a form, and if the bill structure includes a form, obtain coordinate information of a plurality of cells in the detected form;

[0046] a first generation module configured to generate a bill corresponding to the bill image based on the coordinate information of each of the plurality of text blocks, the character sequence corresponding to each of the plurality of text blocks, the plurality of cells, and the coordinate information of each of the plurality of cells.

[0047] In a possible embodiment, the correction module is further configured to:

[0048] detect a height and a width of each of a plurality of text lines in the bill image, and if the height and the width of each of the plurality of text lines satisfy a first preset condition, rotate the bill image by a first preset angle, and detect an inclination angle of each of a plurality of text blocks in the bill image, and perform inclination correction on the bill image based on the inclination angle of each of the plurality of text blocks.

[0049] detect a plurality of text lines in the document image, and perform character recognition on a plurality of target text lines in the plurality of text lines respectively to obtain a character confidence of each of the plurality of target text lines, and if it is determined that the characters in the document image are in an upside-down state based on the character confidence of each of the plurality of target text lines, the document image is rotated by a second preset angle.

[0050] In a possible embodiment, if the height and width of each of the plurality of text lines satisfy a first preset condition, the correction module is further configured to:

[0051] determine a first number of text lines with a height greater than a width and a second number of text lines with a height less than a width;

[0052] if the first number is greater than the second number, the document image is rotated by the first preset angle.

[0053] In a possible embodiment, when the correction module corrects the document image based on the inclination angle of each of the plurality of text blocks, the correction module is further configured to:

[0054] determine an average inclination angle of the plurality of text blocks based on the inclination angle of each of the plurality of text blocks;

[0055] rotate the document image by a corresponding angle according to the average inclination angle.

[0056] In a possible embodiment, when the correction module performs character recognition on a plurality of target text lines in the plurality of text lines respectively to obtain a character confidence of each of the plurality of target text lines, the correction module is further configured to:

[0057] for any target text line, perform character recognition on the target text line to obtain a first character confidence of the target text line, and perform character recognition on the target text line after the target text line is rotated by the second preset angle to obtain a second character confidence of the rotated target text line;

[0058] use the first character confidence and the second character confidence of any target text line as the character confidence of the target text line.

[0059] In a possible embodiment, when the correction module determines that the characters in the document image are in an upside-down state based on the character confidence of each of the plurality of target text lines, the correction module is further configured to:

[0060] determine a third number of the target text lines whose first character confidence reaches the preset value, and determine a fourth number of the target text lines whose second character confidence reaches the preset value;

[0061] if the third number and the fourth number satisfy a preset number condition, determine that the characters in the document image are in an upside-down state.

[0062] In a possible embodiment, the text detection module is further configured to:

[0063] detect, based on a text detection model, coordinate information of each of the text blocks in the document image;

[0064] The character recognition module is further configured to:

[0065] perform character recognition on the text blocks based on a character recognition model to obtain character sequences in the text blocks;

[0066] The table detection module is further configured to:

[0067] detect, based on a target detection model, a plurality of cells contained in the table in the document image and coordinate information of each of the plurality of cells.

[0068] In a possible embodiment, the text detection model includes a text line detection module and a text block positioning module.

[0069] The text detection module is further configured to:

[0070] detect, by the text line detection module, coordinate information of each of the text lines in the document image;

[0071] based on the coordinate information of each of the text lines, divide the plurality of text lines into the plurality of text blocks by the text block positioning module, and for any text block, obtain coordinate information of the text block based on the coordinate information of at least one text line contained in the text block.

[0072] In a possible embodiment, the first generation module is further configured to:

[0073] generate a document table based on the plurality of cells and the coordinate information of each of the plurality of cells by using a hypertext markup language (HTML);

[0074] add, based on the coordinate information of each of the text blocks, a character sequence corresponding to each of the text blocks to a corresponding position in the document table to generate a document corresponding to the document image.

[0075] In a possible embodiment, the apparatus further includes:

[0076] The second generating module is configured to, if the document structure in the document image does not contain a table, generate a document corresponding to the document image based on the coordinate information of each of the text blocks, and the character sequence corresponding to each of the text blocks.

[0077] In a third aspect, the present application provides an electronic device, comprising:

[0078] a memory configured to store program instructions;

[0079] a processor configured to invoke the program instructions stored in the memory, and execute the steps included in the method according to any one of the first aspect by using the obtained program instructions.

[0080] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program comprises program instructions, which, when executed by a computer, cause the computer to execute the method according to any one of the first aspect.

[0081] In a fifth aspect, the present application provides a computer program product, which comprises computer program code, and when the computer program code is executed on a computer, causes the computer to execute the method according to any one of the first aspect.

[0082] The document generation method provided by the embodiments of the present application has at least the following beneficial effects:

[0083] In the embodiments of the present application, after obtaining the document image, the document image is first corrected to ensure that the input is a forward document image, and improve the reliability of subsequent text detection and character recognition. Then, the coordinate information of the plurality of text blocks in the corrected document image is detected, and the character sequence in each text block is recognized. Finally, it is detected whether the document structure in the document image contains a table. If it contains, the coordinate information of the plurality of cells of the detected table is obtained, and then the document corresponding to the document image is generated based on the coordinate information of each of the plurality of text blocks, the character sequence corresponding to each of the plurality of text blocks, the plurality of cells, and the coordinate information of each of the plurality of cells. In this way, by processing the document image, the document corresponding to the document image can be automatically generated. Compared with the traditional manual input of the document, the time for inputting the document is saved, and the efficiency of inputting the document is improved.

[0084] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the following description, or can be learned by practice of the present application. The objects and other advantages of the present application can be realized and achieved by the structure particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0086] Figure 1 A flow chart of a bill generation method provided by an embodiment of the present application;

[0087] Figure 2 A schematic diagram of a bill provided by an embodiment of the present application;

[0088] Figure 3 A flow chart of another bill generation method provided by an embodiment of the present application;

[0089] Figure 4 A structural diagram of a bill generation device provided by an embodiment of the present application;

[0090] Figure 5 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of protection of the present application. In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other at will. And, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in different order from here.

[0092] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. "Multiple" in the present application can mean at least two, for example, can be two, three or more, and the embodiments of the present application are not limited.

[0093] The collection, transmission, use, etc. of data in the technical solutions of the application comply with relevant national laws and regulations.

[0094] Before introducing the document generation method provided by the embodiments of the application, in order to facilitate understanding, first, the technical background of the embodiments of the application is introduced in detail.

[0095] In some banking businesses, banks need to issue documents and audit the documents. The document is a written proof of the actual situation of a transaction or matter obtained or prepared when the economic business occurs, and is the original data and important basis for accounting.

[0096] For example, in international settlement business, banks need to issue a letter of credit and audit the document corresponding to the letter of credit. The letter of credit is a written document issued by the bank to the seller (importer) at the request of the buyer (importer) to guarantee the payment of the payment responsibility. The document not only needs to meet the UCP600, ISBP and other specifications, but also needs to meet the individual requirements of the trade when issuing the letter of credit. The information of the documents of different trades is very different from each other. If the document does not meet the requirements, it will easily affect the entire transaction process.

[0097] After the bank submits the document to the document center, it first needs to enter the document information, and then manually audits the document according to the international convention and practical rules, with the principle of "single consistency" and "document consistency". The traditional document information entry is completed by manual work. However, the types and formats of the documents are various, and the information entry process is very time-consuming and inefficient.

[0098] With the rapid replacement of computer hardware and the accumulation of massive data, computer vision technology has developed rapidly, and its application fields involve all aspects, such as retail industry, manufacturing industry, medical care, automatic driving, etc. Intelligent character recognition (ICR) is an important direction in computer vision technology. ICR integrates artificial intelligence technology on the basis of optical character recognition (OCR), endows the system with learning ability, and further improves the system accuracy and character recognition level.

[0099] The source of the document information is the document image, that is, the document image. In the context of rapid growth of the number of document images, in order to speed up the business processing speed of document information entry and reduce labor costs, using computer vision technology to process the document image, extracting the document information from the document image, and restoring the document to become a better choice.

[0100] The bill audit has high consistency requirements, and therefore how to accurately extract bill information in a bill image and restore the bill information to an accurate position is the main goal of the bill processing process. The bill image often has non-forward conditions, and if the bill image is directly subjected to text detection and recognition, the recognition accuracy is often very low, and the output of the text detection and recognition is scattered text blocks, which cannot correspond to the corresponding position in the bill. Therefore, directly using a single text detection and recognition method cannot restore the complete structured information of the bill, which brings challenges to the extraction of bill information.

[0101] Therefore, the bill generation method, device, electronic equipment and storage medium provided in the embodiments of the present application use ICR technology to analyze a bill image, convert the image content into a structured text language recognizable by a computer, and then perform subsequent analysis, processing and other bill auditing steps according to the conversion result, thereby saving the time for bill entry and improving the efficiency of bill entry.

[0102] The bill generation method of the embodiments of the present application will be described below in combination with the accompanying drawings and specific embodiments.

[0103] Referring to Figure 1 The bill generation method provided in the embodiments of the present application can be executed by a terminal device or a server, and specifically includes the following steps S101-S105.

[0104] Step S101, posture correction is performed on the obtained bill image.

[0105] In this step, the posture of the bill image is corrected so that the text in the corrected bill image is forward, thereby improving the reliability of subsequent text detection and text recognition.

[0106] Specifically, the bill image is corrected multiple times, including first set angle correction, tilt correction and second set angle correction. The first set angle can be 90 degrees, and the second set angle can be 180 degrees. The execution order of the first set angle correction and the tilt correction is not limited, and the second set angle correction is executed after the first set angle correction and the tilt correction to ensure that the text in the finally obtained bill image is forward. The three correction methods will be further described in detail in the following embodiments of the present application.

[0107] Step S102, for the corrected bill image, the coordinate information of each text block in the bill image is detected, and each text block contains at least one text line.

[0108] In the detection of the text block, the text lines can be detected first, and then the similar text lines can be divided into a text block. Specifically, the similar text lines can be determined based on the coordinate information of the detected text lines. For example, if the difference between the vertical coordinates of two text lines is not greater than a first set value, and the difference between the horizontal coordinates is not greater than a second set value, the two text lines can be regarded as similar text lines and can be attributed to the same text block. As shown in the document in FIG. 8, the two text lines in the dashed box can be a text block, and the document further includes other text blocks, each of which is composed of one or more text lines. Figure 2

[0109] In some embodiments, step S102 can include the following implementation.

[0110] Based on the text detection model, the coordinate information of each of the text blocks in the document image is detected.

[0111] Optionally, the text detection model can include a text line detection module and a text block positioning module. The coordinate information of the text blocks in the document image is detected, and the detection can include the following steps A1-A2.

[0112] A1. The coordinate information of each of the text lines in the document image is detected by the text line detection module.

[0113] A2. The text lines are divided into text blocks by the text block positioning module based on the coordinate information of each of the text lines. For any text block, the coordinate information of the text block is obtained based on the coordinate information of at least one text line included in the text block.

[0114] The text detection model can use a text detection method based on image segmentation. First, the structure of the document image is segmented by the text line detection module to achieve pixel-level semantic segmentation. Then, the text lines are constructed according to the segmentation result to generate a segmentation mapping (i.e., the document image is converted into text lines after segmentation), and finally the coordinate information of each text line is obtained. Then, the text block positioning module is used to obtain the multiple text blocks and their coordinate information from the document image based on the coordinate information of each text line.

[0115] ​For example, the above text detection model can adopt an existing text detection network, for example, PSENET (Shape Robust Text Detection with Progressive Scale Expansion Network, shape robust text detection network with progressive scale expansion). The PSENET network is a pixel segmentation-based method, which adopts the idea of progressive expansion, starting from the smallest scale kernel (which can be understood as a pixel), separating the closely adjacent words through the smallest scale kernel, gradually expanding to a larger scale kernel (which can be understood as a phrase), until expanding to the largest scale kernel, forming the final text line. In this way, the text lines that are very close can be distinguished to ensure the accurate position of the text line. Further, based on the predicted coordinate information of each text line in the invoice image, each character block and its coordinate information are obtained.

[0116] It should be noted that the text detection model of the embodiments of the present application can also adopt other detection networks in addition to the above PSENET, which is not limited here.

[0117] Step S103, for any text block, performing character recognition on the text block to obtain the character sequence in the text block.

[0118] In some embodiments, step S102 can include the following implementation:

[0119] Based on the character recognition model, performing character recognition on the text block to obtain the character sequence in the text block.

[0120] Wherein, each text block is taken as a candidate box containing characters, the candidate box containing characters is input into the character recognition model, and the character sequence in the candidate box is output.

[0121] The character recognition model can adopt an existing character recognition network, for example, a CRNN (Convolutional Recurrent Neural Network). The CRNN is an end-to-end recognition network for recognizing text sequences of indefinite length, including feature extraction, sequence analysis, and sequence decoding. The feature extraction part adopts a deep CNN (Convolutional Neural Network) to extract image features of the candidate frame. The sequence analysis part adopts a bidirectional LSTM (Long Short-Term Memory) to extract sequence features in the image features. The sequence decoding part adopts a CTC (Connectionist temporal classification) method to process the alignment problem of the indefinite length sequence in the image and restore the character sequence in the sequence features.

[0122] In addition, the character recognition model of the embodiment of the present application can recognize English characters and Chinese characters at the same time to improve the character recognition accuracy.

[0123] In step S104, it is detected whether the document structure in the document image contains a table. If the table is contained, the coordinate information of each cell in the detected table is obtained.

[0124] In some embodiments, step S104 can include the following implementation:

[0125] Based on the target detection model, it is detected whether the document structure in the document image contains a table. If the table is contained, the coordinate information of each cell in the detected table is obtained.

[0126] The target detection model can adopt an existing target detection network. The target detection model detects whether the document image contains a table. If the table is detected, all cells in the table are detected, and the coordinate information of each cell is obtained. For example, the target detection network can be a PANET (Path Aggregation Network). The PANET fully utilizes feature fusion, introduces a bottom-up path augmentation structure, uses a shallow feature of the network for a segmentation network, and shallow features are very important for target detection because target detection is pixel-level classification. The shallow features are mainly edge shapes and the like. On this basis, adaptive feature pooling is introduced to make the extracted ROI (region of interest) features more rich. Finally, a fully-connected fusion layer is used to obtain more accurate results by fusing the output of a foreground-background binary classification branch.

[0127] It should be noted that the target detection model of the embodiments of the present application can also adopt other target detection networks in addition to the above-mentioned PANET, which is not limited herein.

[0128] In the related art, in order to restore the document table corresponding to the document image, different document table templates are established according to different document images in advance, a document template library is established based on the document table templates, and when the document table corresponding to the document image is restored, the document table template corresponding to the document image is matched by searching the document template library, and the matching is specifically performed by the similarity of the image. However, as the number of documents increases, the document table templates need to be continuously increased, which increases the size of the document template library and the matching time. In order to solve the above problem, the embodiments of the present application detect the document table by the target detection model, obtain the coordinate information of the plurality of cells, and quickly restore the document structure corresponding to the document image according to the coordinate information of the plurality of cells.

[0129] In step S105, the document corresponding to the document image is generated based on the coordinate information of each of the plurality of text blocks, the text sequence corresponding to each of the plurality of text blocks, the plurality of cells, and the coordinate information of each of the plurality of cells.

[0130] In some embodiments, step S105 can include the following steps B1-B2:

[0131] B1, a document table is generated based on the plurality of cells and the coordinate information of each of the plurality of cells by using the HTML (Hypertext Markup Language).

[0132] HTML (Hypertext Markup Language) is a markup language that uses a series of tags to connect disparate resources into a logical whole. In this embodiment, HTML can be used to quickly and conveniently generate document tables based on multiple cells and their respective coordinate information.

[0133] B2. Based on the coordinate information of each text block, add the text sequence corresponding to each text block to the corresponding position in the document table to generate the document corresponding to the document image.

[0134] Specifically, based on the coordinate information of each text block and the coordinate information of the aforementioned document table, the corresponding position of each text block in the document table can be determined, so that the text sequence corresponding to each text block can be added to the corresponding position in the document table.

[0135] In some embodiments, considering that some documents may not contain tables, when the document structure in the detected document image does not contain a table, the document corresponding to the document image can be generated directly based on the coordinate information of each of the multiple text blocks and the text sequence corresponding to each of the multiple text blocks.

[0136] In this embodiment, by performing text detection, character recognition, and table detection on the corrected document image, the document corresponding to the document image can be automatically generated. Compared with the traditional manual document entry, this saves document entry time and improves document entry efficiency.

[0137] Furthermore, this application embodiment uses different models to perform text detection, character recognition, and document structure restoration tasks in document information extraction. That is, the entire document generation task is divided into three modules for processing. Different modules can be optimized and improved to accurately generate documents corresponding to document images, which plays an important role in document review.

[0138] The following describes the posture correction method for the document image in step S101 above.

[0139] In some embodiments, such as Figure 3 As shown, step S101 above performs posture correction on the acquired document image, which may include the following steps S1011-S1013:

[0140] Step S1011: Detect the height and width of each of the multiple text lines in the document image. If the height and width of each of the multiple text lines meet the first preset condition, rotate the document image by the first preset angle.

[0141] The first set angle can be 90 degrees, each text line corresponds to a text box, and includes height and width. Since the width of the text line in the document is greater than the height, by detecting the height and width of each text line, it can be determined whether the document image needs to be rotated by 90 degrees.

[0142] Specifically, a text detection model can be used to detect the multiple text lines in the document image to obtain the height and width of each text line. The text detection model can be the same as or different from the text detection model used in step S102, and no limitation is made thereto. Optionally, in order to improve the text detection speed, a DBNET (differentiable Binarization net) can be used to detect the height and width of each of the multiple text lines in the document image. The DBNET network not only has fast detection speed, but also can detect text boxes of any shape.

[0143] If the height and width of each of the multiple text lines satisfy the first preset condition in step S1011, the document image is rotated by a first set angle, which can include the following steps C1-C2:

[0144] C1, determine a first number of text lines with a height greater than a width and a second number of text lines with a height less than a width.

[0145] C2, if the first number is greater than the second number, the document image is rotated by the first set angle.

[0146] For example, after detecting the text lines in the document image, the number n1 of text lines with a height greater than a width and the number n2 of text lines with a height less than a width are counted. When n1 is greater than n2, it indicates that the document image needs to be rotated by 90 degrees, otherwise, no rotation is performed.

[0147] In step S1012, the inclination angles of the multiple text blocks in the document image are detected, and the document image is corrected based on the inclination angles of the multiple text blocks.

[0148] The text detection model is used to detect the coordinate information of the multiple text lines in the document image, and then the multiple text blocks and their coordinate information are obtained, and the inclination angles of the multiple text blocks are obtained. The text detection model can be the same as or different from the text detection model used in step S102, and no limitation is made thereto. Optionally, an EAST (Efficient and Accuracy Scene Text) network is used to detect the inclination angles of the text blocks in the document image. The EAST network is a full convolutional network that can flexibly generate character-level or text line predictions. The shape of the predicted text line can be a rotated rectangle or an irregular quadrilateral.

[0149] In some optional embodiments, the step S1012 of performing the tilt correction on the document image based on the respective tilt angles of the plurality of text blocks can include the following steps:

[0150] D1, determining an average tilt angle of the plurality of text blocks based on the respective tilt angles of the plurality of text blocks;

[0151] D2, rotating the document image by a corresponding angle according to the average tilt angle.

[0152] In this embodiment, the tilt correction on the document image based on the average tilt angle of the plurality of text blocks can reduce the dependence on text detection and avoid the impact of detection errors of some text blocks on the tilt correction result.

[0153] It should be noted that the order of the steps S1011 and S1012 is not limited in the embodiments of the present application, and the step S1012 can also be performed before the step S1011.

[0154] In step S1013, a plurality of text lines in the document image are detected, and for a plurality of target text lines in the plurality of text lines, character recognition is performed respectively to obtain a character confidence of each of the plurality of target text lines. If it is determined that the characters in the document image are in an upside-down state based on the character confidence of each of the plurality of target text lines, the document image is rotated by a second set angle.

[0155] The second set angle can be 180 degrees. Since the document image after the above-mentioned 90-degree rotation correction and tilt correction can be in a forward direction or a reverse direction, it is necessary to determine whether 180-degree rotation correction is needed by using the character confidence.

[0156] In step S1013, the text detection model is used to detect the plurality of text lines in the document image. The text detection model can be the same as or different from the text detection model used in step S102, and the order is not limited. Optionally, the PSENET network is used to detect the coordinate information of the plurality of text lines in the document image, and then the plurality of text lines are sorted according to the coordinate information, and a plurality of target text lines are selected from the plurality of text lines. For example, the text lines arranged in the middle are selected as the target text lines, or the text lines are selected as the target text lines, and the order is not limited.

[0157] Further, the character recognition model is used to perform character recognition on the target text lines respectively to obtain character confidence of each target text line. The character confidence of each target text line can be an average of the confidence of the characters in the target text line. The confidence of each character represents a probability of detecting the character. If the character in the target text line is in a normal direction, the corresponding character confidence is high. If the character in the target text line is in an inverted direction, the corresponding character confidence is low. Then, whether the characters in the document image are in an inverted state (i.e., in an inverted direction) is determined based on the character confidence of the target text lines.

[0158] For example, assuming that the total number of target text lines is m, the number of target text lines with a character confidence less than a preset value is m1, and the number of target text lines with a character confidence greater than or equal to the preset value is m2. If m1 exceeds a number threshold, it is determined that the characters in the document image are in an inverted state, and the document image needs to be rotated by 180 degrees. If m2 exceeds the number threshold, it is determined that the characters in the document image are in a normal direction, and the document image does not need to be rotated by 180 degrees. The preset value and the number threshold can be set as needed, for example, the preset value is 0.9 and the number threshold is 0.8m, which are not limited.

[0159] Optionally, to more accurately determine whether the characters in the document image are in a normal direction or an inverted direction, in the step S1013, the character recognition is performed on the target text lines to obtain the character confidence of each target text line, which can include the following steps a1-b2.

[0160] a1. For any target text line, the character recognition is performed on the target text line to obtain a first character confidence of the target text line, and the character recognition is performed on the target text line after being rotated by a second preset angle to obtain a second character confidence of the rotated target text line.

[0161] a2. The first character confidence and the second character confidence of any target text line are taken as the character confidence of the target text line.

[0162] Further, based on the character confidence of each target text line, whether the characters in the document image are in an inverted state can be determined, which can include the following steps b1-b2.

[0163] b1. A third number of target text lines with a first character confidence reaching a preset value is determined, and a fourth number of target text lines with a second character confidence reaching the preset value is determined.

[0164] The preset value can be set as needed, for example, 0.9, which is not limited.

[0165] b2、if the third quantity and the fourth quantity satisfy a preset quantity condition, it is determined that the text in the invoice image is in an upside-down state.

[0166] On the contrary, if the third quantity and the fourth quantity do not satisfy the preset quantity condition, it is determined that the text in the invoice image is in a normal state and does not need to be rotated by 180 degrees.

[0167] The preset quantity condition can be set as needed, for example, the fourth quantity is more than half of the total number of target text lines and greater than the third quantity.

[0168] For example, the PSENET is used to predict the coordinate information of the plurality of text lines in the invoice image, the plurality of text lines are sorted according to the coordinate information, the m' target text lines in the middle are selected, the 180-degree rotation is performed, the text recognition model CRNN is used to perform text recognition on the target text lines before and after rotation, and the text confidence before and after rotation is output. The number of text confidences before rotation that exceed 0.9 is recorded as m1', and the number of text confidences after rotation that exceed 0.9 is recorded as m2'. If m1' is more than half of the total number of target text lines m' and greater than m2', it is determined that the invoice image is in a normal state and does not need to be rotated by 180 degrees. If m2' is more than half of the total number of target text lines m' and greater than m1', it is determined that the invoice image is in an upside-down state and needs to be rotated by 180 degrees. If neither m1' nor m2' exceeds half of m', it indicates that the invoice image is blurred and cannot be judged whether to be rotated by 180 degrees according to the text confidence, and no rotation is performed.

[0169] In the embodiments of the present application, through the above-mentioned 90-degree rotation correction, inclination correction and 180-degree rotation correction on the invoice image, it can be ensured that the subsequent text detection and text recognition are performed on the normal invoice image, and the reliability of text detection and text recognition is improved.

[0170] Based on the same inventive concept, the embodiments of the present application provide an invoice generation device, please refer to Figure 4 The device comprises:

[0171] The correction module 41 is configured to perform posture correction on the obtained invoice image.

[0172] The text detection module 42 is configured to detect the coordinate information of each text block in the corrected invoice image, and each text block contains at least one text line.

[0173] The text recognition module 43 is configured to perform text recognition on any text block to obtain a text sequence in the text block.

[0174] The table detection module 44 is configured to detect whether the document structure in the document image contains a table, and if the document structure contains a table, obtain the coordinate information of each cell in the detected table.

[0175] The first generation module 45 is configured to generate a document corresponding to the document image based on the coordinate information of each text block, the text sequence corresponding to each text block, the plurality of cells, and the coordinate information of each cell.

[0176] In the embodiments of the present application, by processing the document image, the document corresponding to the document image can be automatically generated. Compared with the traditional manual entry of the document, the time for entering the document is saved, and the efficiency of entering the document is improved.

[0177] In a possible embodiment, the correction module 41 is further configured to:

[0178] detect the height and width of each text line in the document image, and if the height and width of each text line satisfy a first preset condition, rotate the document image by a first preset angle; and detect the inclination angle of each text block in the document image, and correct the inclination of the document image based on the inclination angle of each text block.

[0179] detect the plurality of text lines in the document image, and for a plurality of target text lines in the plurality of text lines, respectively perform character recognition to obtain the character confidence of each target text line, and if it is determined that the characters in the document image are in an inverted state based on the character confidence of each target text line, rotate the document image by a second preset angle.

[0180] In a possible embodiment, if the height and width of each text line satisfy the first preset condition, when the correction module 41 rotates the document image by the first preset angle, the correction module 41 is further configured to:

[0181] determine a first number of text lines with a height greater than a width and a second number of text lines with a height less than a width;

[0182] if the first number is greater than the second number, rotate the document image by the first preset angle.

[0183] In a possible embodiment, when the correction module 41 corrects the inclination of the document image based on the inclination angle of each text block, the correction module 41 is further configured to:

[0184] determine an average inclination angle of the plurality of text blocks based on the inclination angle of each text block;

[0185] rotate the document image by a corresponding angle according to the average inclination angle.

[0186] In a possible implementation, the correction module 41 is further configured to:

[0187] for any target text line, performing character recognition on the target text line to obtain a first character confidence of the target text line, and performing character recognition on the target text line after a second preset angle rotation to obtain a second character confidence of the target text line after the rotation;

[0188] taking the first character confidence and the second character confidence of any target text line as the character confidence of the target text line.

[0189] In a possible implementation, when it is determined that the characters in the document image are in an upside-down state based on the character confidence of each of the target text lines, the correction module 41 is further configured to:

[0190] determining a third number of target text lines whose first character confidence reaches a preset value, and determining a fourth number of target text lines whose second character confidence reaches the preset value;

[0191] if the third number and the fourth number satisfy a preset number condition, determining that the characters in the document image are in the upside-down state.

[0192] In a possible implementation, the text detection module 42 is further configured to:

[0193] detecting, based on a text detection model, coordinate information of each of a plurality of text blocks in the document image;

[0194] The character recognition module 43 is further configured to:

[0195] performing character recognition on the text block based on a character recognition model to obtain a character sequence in the text block;

[0196] The table detection module 44 is further configured to:

[0197] detecting, based on a target detection model, whether a document structure in the document image contains a table.

[0198] In a possible implementation, the text detection model includes a text line detection module and a text block positioning module.

[0199] The text detection module 42 is further configured to:

[0200] detecting, by the text line detection module, coordinate information of each of a plurality of text lines in the document image;

[0201] The text block positioning module divides the plurality of text lines into a plurality of text blocks based on the coordinate information of the plurality of text lines, and obtains coordinate information of any text block based on the coordinate information of at least one text line included in the text block.

[0202] In a possible implementation, the first generating module 35 is further configured to:

[0203] The first generating module 35 generates the document table based on the plurality of cells and the coordinate information of the plurality of cells by using the HyperText Markup Language (HTML).

[0204] The first generating module 35 adds the character sequence corresponding to each text block to a corresponding position in the document table based on the coordinate information of each text block, to generate the document corresponding to the document image.

[0205] In a possible implementation, the apparatus further includes:

[0206] The second generating module is configured to, if the document structure in the document image does not include a table, generate the document corresponding to the document image based on the coordinate information of the plurality of text blocks and the character sequence corresponding to each of the plurality of text blocks.

[0207] Based on the same inventive concept, an electronic device is provided in the embodiments of the present application, which can implement the functions of the document generation apparatus discussed above. Please refer to Figure 5 The device includes a processor 501 and a memory 502, the memory 502 is configured to store program instructions, and the processor 501 is configured to call the program instructions stored in the memory 502 and perform the steps included in any of the above document generation methods according to the obtained program instructions.

[0208] In the embodiments of the present application, the specific connection medium between the memory 502 and the processor 501 is not limited. For example, the memory 502 and the processor 501 are connected through a bus, and the bus can be divided into an address bus, a data bus, a control bus, etc.

[0209] The memory 502 can include a Read-Only Memory (ROM) and a Random Access Memory (RAM), and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0210] The processor 501 can be a general processor, including a central processing unit, a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, etc.

[0211] Based on the same inventive concept, the embodiment of the present application provides a computer readable storage medium, and a computer program product includes computer program codes, which, when running on a computer, causes the computer to execute the invoice generation method as any of the preceding embodiments. Since the principle of the computer readable storage medium to solve the problem is similar to the invoice generation method, the implementation of the computer readable storage medium can be referred to the implementation of the method, and the repeated parts will not be described here.

[0212] In the specific implementation process, the computer storage medium can include a universal serial bus flash drive (USB, Universal Serial Bus Flash Drive), a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage media that can store program codes.

[0213] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes computer program codes, which, when running on a computer, causes the computer to execute the invoice generation method as any of the preceding embodiments. Since the principle of the computer program product to solve the problem is similar to the invoice generation method, the implementation of the computer program product can be referred to the implementation of the method, and the repeated parts will not be described here.

[0214] The computer program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0215] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0216] The present application is described in reference to the flow diagrams and / or block diagrams of the methods, apparatus (systems) and computer program products according to this application. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0217] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0219] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their legal equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method of generating a document, characterized by, include: After performing tilt correction and 90-degree correction on the document image to be processed, multiple target text lines in the document image are detected. For each target text line, character recognition is performed to obtain the first character confidence score of the target text line. After rotating the target text line by 180 degrees, character recognition is performed on the rotated target text line to obtain the second character confidence score of the rotated target text line. The third number of target text lines whose first text confidence reaches a preset value is determined, and the fourth number of target text lines whose second text confidence reaches the preset value is determined. When the fourth number exceeds half of the total number of target text lines and is greater than the third number, the text in the document image is determined to be inverted, and the document image is corrected by 180 degrees. For the corrected document image, a pixel-based text detection model is used to perform pixel-level semantic segmentation on the document image, and text lines are constructed based on the segmentation results to obtain the coordinate information of each of the multiple text lines in the document image. The multiple text lines are then divided into multiple text blocks. Specifically, for any two text lines, if the difference between the vertical coordinates of the two text lines is not greater than a first preset value and the difference between the horizontal coordinates is not greater than a second preset value, then the two text lines are assigned to the same text block. For each text block, based on the coordinate information of at least one text line contained in the text block, the coordinate information of the text block is obtained. For any text block, character recognition is performed on the text block to obtain the character sequence in the text block. Detect whether the document structure in the document image contains a table; if it contains a table, obtain the coordinate information of each of the multiple cells in the detected table. Based on the coordinate information of each of the multiple text blocks, the text sequence corresponding to each of the multiple text blocks, the multiple cells, and the coordinate information of each of the multiple cells, a document corresponding to the document image is generated.

2. The method of claim 1, wherein, The process of performing tilt correction and 90-degree correction on the document image to be processed includes: The height and width of each of the multiple text lines in the document image are detected. If the height and width of each of the multiple text lines meet a first preset condition, the document image is rotated by a first preset angle. The tilt angle of each of the multiple text blocks in the document image is detected, and the tilt correction of the document image is performed based on the tilt angle of each of the multiple text blocks.

3. The method of claim 2, wherein, If the height and width of each of the multiple text lines satisfy a first preset condition, then the document image is rotated by a first preset angle, including: Determine the first number of text lines whose height is greater than their width and the second number of text lines whose height is less than their width; If the first quantity is greater than the second quantity, then the document image is rotated by the first set angle.

4. The method according to claim 2, characterized in that, The tilt correction of the document image based on the tilt angle of each of the multiple text blocks includes: Based on the tilt angle of each of the multiple text blocks, determine the average tilt angle of the multiple text blocks; The document image is rotated by a corresponding angle according to the average tilt angle.

5. The method according to claim 1, characterized in that, The step of performing character recognition on the text block to obtain the character sequence in the text block includes: Based on the character recognition model, the text block is subjected to character recognition to obtain the character sequence in the text block; The step of detecting whether the document structure in the document image contains a table includes: Based on the object detection model, it is determined whether the document structure in the document image contains a table.

6. The method according to claim 1, characterized in that, The step of generating a document corresponding to the document image based on the coordinate information of each of the multiple text blocks, the text sequence corresponding to each of the multiple text blocks, the multiple cells, and the coordinate information of each of the multiple cells includes: A document table is generated using Hypertext Markup Language (HTML) based on multiple cells and their respective coordinate information. Based on the coordinate information of each text block, the text sequence corresponding to each text block is added to the corresponding position in the document table to generate the document corresponding to the document image.

7. The method according to claim 1, characterized in that, The method further includes: If the document structure in the document image does not contain a table, then a document corresponding to the document image is generated based on the coordinate information of each of the multiple text blocks and the text sequence corresponding to each of the multiple text blocks.

8. A document generation device, characterized in that, include: The correction module is used to perform tilt correction and 90-degree correction on the document image to be processed. Then, for the obtained document image, it detects multiple target text lines in the document image and performs character recognition on each target text line to obtain a first character confidence score. After rotating the target text line by 180 degrees, it performs character recognition on the rotated target text line to obtain a second character confidence score. It determines a third number of target text lines whose first character confidence score reaches a preset value, and a fourth number of target text lines whose second character confidence score reaches the preset value. When the fourth number exceeds half the total number of target text lines and is greater than the third number, it determines that the text in the document image is inverted and performs 180-degree correction on the document image. The text detection module is used to perform pixel-level semantic segmentation on the corrected document image using a pixel-based text detection model, construct text lines based on the segmentation results, obtain the coordinate information of each text line in the document image, and divide the multiple text lines into multiple text blocks; wherein, for any two text lines, if the difference between the ordinates of the two text lines is not greater than a first preset value and the difference between the abscissas is not greater than a second preset value, then the two text lines are assigned to the same text block; for each text block, the coordinate information of the text block is obtained based on the coordinate information of at least one text line contained in the text block; The text recognition module is used to perform text recognition on any of the text blocks to obtain the text sequence in the text block; The table detection module is used to detect whether the document structure in the document image contains a table. If a table is contained, the coordinate information of each of the multiple cells in the detected table is obtained. The first generation module is used to generate a document corresponding to the document image based on the coordinate information of each of the multiple text blocks, the text sequence corresponding to each of the multiple text blocks, the multiple cells, and the coordinate information of each of the multiple cells.

9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-7.

Images