A method and system for identifying content of a document image

Abstract

The application discloses a kind of bill image content identification method and system, method includes: using nonlinear diffusion filter to the source bill image obtained with standard bill image is constructed to obtain nonlinear diffusion scale space;Using image block strategy removes the first layer scale space in nonlinear diffusion scale space, then carries out grid unit division to nonlinear diffusion scale space and obtains stable uniform point feature for feature point extraction;According to point feature, gradient position direction histogram algorithm is used to construct feature point descriptor;Using bilateral fast approximation nearest neighbor algorithm, the feature points expressed by feature point descriptor are matched, and the point pairs of false matching are removed by using random sample consensus algorithm to obtain matching point pair, and the bill image of successful matching is obtained;The bill image is preprocessed, target text is detected, and specific field detection is carried out to obtain the text image labeled with specific field and input into the text recognition model to recognize the text content and obtain the field information.

Description

Technical Field

[0001] This invention relates to the field of invoice image content recognition, and specifically to a method and system for invoice image content recognition. Background Technology

[0002] As the company's various business lines have grown and expanded, the number of invoices issued and received has increased daily, leading to a surge in pressure on manual review of invoices. With a large volume and diverse formats of invoices, the efficiency and accuracy of manual processing cannot be guaranteed. For common scenarios with simple layouts and clear printing, such as ID card, bank card, license plate, and business card recognition, there are already mature commercial applications and high recognition rates. However, invoices, as written documents recording matters in economic activities, are particularly difficult to match, locate, and recognize due to their diverse types and varied styles. The positions of characteristic words differ among different invoices, and the fonts and formats (such as uppercase and lowercase numbers) vary greatly. During auditing, if the number of invoices is large, the workload for manual auditing becomes enormous, resulting in reduced efficiency and a higher risk of errors.

[0003] In recent years, with the digital transformation of enterprises, new vitality has been injected into enterprises. More and more enterprises have begun to adopt electronic forms and use computers to replace manual labor for some large-volume and process-oriented tasks, in order to improve efficiency and reduce errors caused by manual completion. Summary of the Invention

[0004] The technical problem this invention aims to solve is that invoices, as written documents recording matters in economic activities, are extremely difficult to match, locate, and identify due to their diverse types and ever-changing styles. Different invoices have different feature word positions, and the fonts, capitalization, and other formats vary greatly. During auditing, if the number of invoices is large, the workload for manual auditing becomes enormous, leading to reduced efficiency and increased errors. This invention provides a method for recognizing invoice image content, and also provides an invoice image content recognition system. Based on the needs of a company's finance department for recognizing and storing invoice content, this system combines feature matching and machine learning algorithms to match and identify field content in invoice images. It can quickly and accurately extract and store important field information contained in invoice images according to business rules, solving the pain points of slow speed and low accuracy in manual invoice matching and content extraction. This simplifies the workflow of company finance personnel, improves the efficiency of financial auditing, and facilitates the orderly operation of the company's finance department, thus addressing the shortcomings of existing technologies.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, a method for recognizing the content of a bill image includes the following steps:

[0007] Step 1: Use nonlinear diffusion filtering to construct a nonlinear diffusion scale space from the source ticket image and the standard ticket image;

[0008] Step 2: After removing the first layer of scale space in the nonlinear diffusion scale space using an image segmentation strategy, the nonlinear diffusion scale space is divided into grid cells. After the nonlinear diffusion scale space is divided into grid cells, feature points are extracted in the grid cells to obtain stable and uniform point features.

[0009] Step 3: Construct feature point descriptors using the gradient position orientation histogram algorithm based on the point features;

[0010] Step 4: Use the bilateral fast approximate nearest neighbor algorithm to perform feature matching on the feature points expressed by the feature point descriptor, and use the random sampling consensus algorithm to remove mismatched point pairs to obtain matching point pairs and obtain the successfully matched ticket image;

[0011] Step 5: Preprocess the ticket image to obtain a preprocessed image;

[0012] Step 6: Perform target text detection on the preprocessed image to obtain the target text image;

[0013] Step 7: Perform specific field detection on the target text image to obtain a text image labeled with specific fields;

[0014] Step 8: Input the text image into the text recognition model to obtain field information by recognizing the text content.

[0015] The aforementioned method for recognizing content from a ticket image involves using the nonlinear diffusion filter to construct a nonlinear diffusion equation L in the nonlinear diffusion scale space from the acquired source ticket image and the standard ticket image. t for:

[0016]

[0017] Where div is the divergence operator and p(m,n,t) is the diffusion function. and Δ are the gradient and Laplace operator, respectively, and the above formula is the anisotropic diffusion equation;

[0018] The diffusion function p(m,n,t) is expressed as:

[0019]

[0020] Here, g(·) is the edge function; when g(·) is between 0 and 1, it can not only preserve edges but also sharpen brightness edges. ‖‖ is the modulo operation. It is the gradient after Gaussian filtering.

[0021] When p(m,n,t) is a constant, it can be simplified to an isotropic diffusion equation:

[0022]

[0023] The above-mentioned method for recognizing content from a ticket image includes the following specific steps in step 2:

[0024] The first scale space in the nonlinear diffusion scale space is removed using an image segmentation strategy;

[0025] The nonlinear diffusion scale space of each layer is divided into grid cells;

[0026] The number of feature points is obtained, the number of feature points in the nonlinear diffusion scale space after each layer of grid cell division is determined, and feature points are extracted.

[0027] The feature points are extracted in blocks, corresponding to the corresponding positions in the original nonlinear diffusion scale space. The position of each block is determined according to the size and number of blocks, thus determining the position of the feature points in the original linear diffusion scale space.

[0028] By statistically analyzing all feature points, a stable and uniform point feature is obtained.

[0029] The above-mentioned method for recognizing content from a ticket image includes the following specific steps for step 3:

[0030] The feature points are corrected based on their location;

[0031] The gradient position orientation histogram algorithm is used to construct feature point descriptors from the corrected feature points.

[0032] The above-mentioned method for recognizing content from a ticket image includes the following specific steps for step 4:

[0033] Calculate the Euclidean distance between points q and m and point p, denoted as d. pq and d pm The Euclidean distances between points r and s and point n are denoted as d. nr and d ns The calculation formula is as follows:

[0034]

[0035] Where i is the lower bound of the Σ accumulation function, and points q and m are the nearest neighbor feature points q and m, respectively, of point p in the source document image denoted as I1, and the second nearest neighbor feature point m, respectively, in the standard document image denoted as I2. The corresponding feature vector is F. q F m F p Let r and s be the nearest and second nearest neighbor feature points of point n in I2 on I1, respectively, and their corresponding feature vectors be F. r F s F n ;

[0036] Calculate d pq With d pm d nr With d ns The distance ratios are denoted as R1 and R2, and their calculation formulas are as follows:

[0037]

[0038] Determine the relationship between R1, R2 and the threshold T respectively:

[0039] If R1 < T, then point p and point q are successfully matched; otherwise, the matching fails.

[0040] If R2 < T, then point n and point r are successfully matched; otherwise, the matching fails.

[0041] Keep pairs of points that are identical in two matching results;

[0042] Four pairs of points are randomly selected. The distance between the remaining point pairs and the pre-matched point pairs after coordinate transformation is calculated using the affine transformation model parameters. Incorrectly matched point pairs are eliminated to obtain one-to-one matching point pairs between the source ticket image and the standard ticket image, and the successfully matched ticket image is obtained.

[0043] The above-mentioned method for recognizing content from a ticket image includes a step 5 method for preprocessing the ticket image to obtain a preprocessed image, which is described below:

[0044] After scaling each ticket image to 224×224, a residual network model is used to perform orientation recognition on the resized ticket images. Ticket images with a rotation angle of not 0 are rotated to obtain a positive image and output as the preprocessed image, or ticket images with a rotation angle of 0 are directly output as the preprocessed image.

[0045] The residual network model is trained using the slim framework in TensorFlow with the ticket data at 0°, 90°, 180°, 270° and rotated ticket data with angle labels as input data.

[0046] The above-mentioned method for recognizing content from a ticket image includes a step 6 method for detecting target text in the preprocessed image to obtain the target text image, which is described below:

[0047] After the fields to be detected in the preprocessed image are annotated using an open-source image annotation tool, they are input into the target text detection model to detect the target text image, which includes the invoice code, invoice number, invoice date, and pre-tax amount.

[0048] The above-mentioned method for recognizing content from a document image includes a specific method for obtaining a text image labeled with a specific field in step 7:

[0049] The OpenCV minAreaRect() function is used to fit a minimum bounding rectangle to the detected target text image, resulting in the coordinates of the minimum bounding rectangle and the corresponding rotation angle relative to the X-axis.

[0050] The `boundingRect()` function of OpenCV is used to obtain a bounding box without rotation angle for the target text image. Based on this bounding box, the text data image can be extracted from the original ticket image.

[0051] After rotating the text data image according to the rotation angle along the X-axis, a projection cutting method is used to remove residual boundaries to obtain a useful text data image.

[0052] The useful text data image is scaled to 400×100 and then input into a specific field recognition model for detection to obtain the text image labeled with specific fields;

[0053] The method for constructing the text recognition model described in step 8 is as follows:

[0054] The simulated ticket text image data was used as input data to construct the text recognition model using DenseNet+BiLSTM+CTC.

[0055] In a second aspect, a bill image content recognition system includes: a nonlinear diffusion scale space construction module, a point feature extraction module, a feature point descriptor construction module, a bill image processing module, a preprocessing module, a target text detection module, a specific field detection module, and a field information extraction module.

[0056] The nonlinear diffusion scale space construction module is used to construct a nonlinear diffusion scale space from the acquired source ticket image and the standard ticket image using nonlinear diffusion filtering.

[0057] The point feature extraction module is used to remove the first layer of scale space in the nonlinear diffusion scale space using an image block strategy, and then divide the nonlinear diffusion scale space into grid cells. It is also used to extract feature points in the grid cells of the nonlinear diffusion scale space after grid cell division to obtain stable and uniform point features.

[0058] The feature point descriptor construction module is used to construct feature point descriptors based on the point features using the gradient position orientation histogram algorithm;

[0059] The ticket image processing module is used to perform feature matching on the feature points expressed by the feature point descriptor using a bilateral fast approximate nearest neighbor algorithm and to remove mismatched point pairs using a random sampling consensus algorithm to obtain matched point pairs and acquire successfully matched ticket images.

[0060] The preprocessing module is used to preprocess the ticket image to obtain a preprocessed image;

[0061] The target text detection module is used to perform target text detection on the preprocessed image to obtain a target text image;

[0062] The specific field detection module is used to perform specific field detection on the target text image to obtain a text image labeled with specific fields;

[0063] The field information extraction module is used to input the text image into the text recognition model to obtain field information through text content recognition.

[0064] Thirdly, a processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any of the first aspects.

[0065] The technical solution provided by the present invention, which relates to a method and system for recognizing the content of invoice images, has the following technical effects:

[0066] This invention addresses the needs of a company's finance department for identifying and storing invoice content. By combining feature matching and machine learning algorithms to match invoice images and identify field content, it can quickly and accurately extract and store important field information contained in invoice images according to business rules. This solves the pain points of slow speed and low accuracy in manual invoice matching and content extraction, simplifies the workflow of company finance personnel, improves the efficiency of financial auditing, and helps the orderly operation of the company's finance department. Attached Figure Description

[0067] Figure 1 This is a flowchart of a method for recognizing the content of a ticket image according to the present invention;

[0068] Figure 2 This is a schematic diagram of the structure of a ticket image content recognition system according to the present invention.

[0069] The accompanying figure is labeled as follows:

[0070] The system includes a nonlinear diffusion scale space construction module 100, a point feature extraction module 200, a feature point descriptor construction module 300, a ticket image processing module 400, a preprocessing module 500, a target text detection module 600, a specific field detection module 700, and a field information extraction module 800. Detailed Implementation

[0071] In order to make the technical means, inventive features, objectives and effects of the invention easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to specific illustrations. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0072] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0074] Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are merely for clarity of description and are not intended to limit the scope of the invention. Any changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0075] like Figure 1 As shown, in a first aspect, a first embodiment, a method for recognizing the content of a ticket image includes the following steps:

[0076] Step 1: Use nonlinear diffusion filtering to construct a nonlinear diffusion scale space from the source ticket image and the standard ticket image;

[0077] Step 2: After removing the first scale space in the nonlinear diffusion scale space using an image segmentation strategy, the nonlinear diffusion scale space is divided into grid cells. After the nonlinear diffusion scale space is divided into grid cells, feature points are extracted in the grid cells to obtain stable and uniform point features.

[0078] Step 3: Construct feature point descriptors using the gradient position orientation histogram algorithm based on the point features;

[0079] Step 4: Use the bilateral fast approximate nearest neighbor algorithm to perform feature matching on the feature points expressed by the feature point descriptor, and use the random sampling consensus algorithm to remove mismatched point pairs to obtain matching point pairs and obtain the successfully matched ticket image;

[0080] Step 5: Preprocess the ticket image to obtain a preprocessed image;

[0081] Step 6: Perform target text detection on the preprocessed image to obtain the target text image;

[0082] Step 7: Perform specific field detection on the target text image to obtain a text image labeled with specific fields;

[0083] Step 8: Input the text image into the text recognition model to obtain field information by recognizing the text content.

[0084] The aforementioned method for recognizing content from invoice images involves using nonlinear diffusion filtering to construct a nonlinear diffusion equation L in the nonlinear diffusion scale space from the acquired source invoice image and the standard invoice image. t for:

[0085]

[0086] Where div is the divergence operator and p(m,n,t) is the diffusion function. and Δ are the gradient and Laplace operator, respectively, and the above formula is the anisotropic diffusion equation;

[0087] The diffusion function p(m,n,t) is expressed as:

[0088]

[0089] Here, g(·) is the edge function; when g(·) is between 0 and 1, it can not only preserve edges but also sharpen brightness edges. ‖‖ is the modulo operation. It is the gradient after Gaussian filtering;

[0090] When p(m,n,t) is a constant, it can be simplified to an isotropic diffusion equation:

[0091]

[0092] The original SIFT algorithm uses a Gaussian filter to construct the scale space, but the Gaussian filter has poor edge preservation, resulting in blurred natural boundaries of objects. As the scale increases, more edge details are lost in the Gaussian scale space. The nonlinear diffusion scale space can effectively preserve edge detail features and has low blur between images, so the feature extraction effect is better, which is beneficial to the subsequent uniform gradient calculation.

[0093] The above-mentioned method for recognizing content from a ticket image includes the following specific steps in step 2:

[0094] The first scale space in the nonlinear diffusion scale space is removed using an image segmentation strategy.

[0095] Each nonlinear diffusion scale space is divided into grid cells;

[0096] The number of feature points is obtained, the number of feature points in the nonlinear diffusion scale space after each layer of grid cells is determined, and feature points are extracted.

[0097] The feature points are extracted in blocks, corresponding to the corresponding positions in the original nonlinear diffusion scale space. The position of each block is determined according to the size and number of blocks, thus determining the position of the feature points in the original linear diffusion scale space.

[0098] By statistically analyzing all feature points, stable and uniform point features are obtained;

[0099] The first layer of the scale space image is the original resolution image. Most of the feature points extracted in this layer are speckle noise. The randomness of speckle noise can affect the actual feature points in the vicinity, leading to image mismatch. Therefore, the feature extraction operation in the first layer of scale space is cancelled, and the image in this layer is not divided into blocks, which speeds up the image registration speed to a certain extent. The size of the segmentation grid for each layer is determined according to the relationship between the scale space coefficients and can be adjusted according to the actual application requirements, which speeds up the algorithm to a certain extent.

[0100] Uneven distribution of feature points can lead to an over-concentration of successfully matched feature points, which is detrimental to subsequent geometric transformation models. To achieve a more accurate transformation model and higher registration precision by distributing feature points evenly across the entire image, a segmentation strategy is introduced in the scale space. The main purpose of the feature extraction stage is to obtain a sufficient number of relevant point features that are evenly distributed in the image and scale space. The block strategy obtains evenly distributed point features in the entire space by extracting feature points from each regular grid. Based on the idea of ​​the block strategy, the above-mentioned block strategy for scale space was designed.

[0101] The above-mentioned method for recognizing content from a ticket image includes the following specific steps for step 3:

[0102] The feature points are corrected based on their location;

[0103] The gradient position orientation histogram algorithm is used to construct feature point descriptors from the corrected feature points;

[0104] The Gradient Location and Orientation Histogram (GLOH) algorithm divides the circular neighborhood of a feature point into three concentric circles. The two outer concentric circles are further divided into eight equal parts, while the central circle remains unchanged, resulting in 17 sub-regions. The gradient direction of each sub-region (0°–180°) is then divided into eight directions, and a histogram of the gradient directions of the feature point is obtained. The peak value of the histogram represents the dominant direction of the feature point. This structure is isotropic and can increase the robustness of the descriptor.

[0105] The above-mentioned method for recognizing content from a ticket image includes the following specific steps for step 4:

[0106] Calculate the Euclidean distance between points q and m and point p, denoted as d. pq and d pm The Euclidean distances between points r and s and point n are denoted as d. nr and d ns The calculation formula is as follows:

[0107]

[0108] Where i is the lower bound of the Σ accumulation function, and points q and m are the nearest neighbor feature points q and m, respectively, of point p in the source document image denoted as I1, and the second nearest neighbor feature point m, respectively, in the standard document image denoted as I2. The corresponding feature vector is F. q F m F p Let r and s be the nearest and second nearest neighbor feature points of point n in I2 on I1, respectively, and their corresponding feature vectors be F. r F s F n ;

[0109] Calculate d pq With d pm d nr With d ns The distance ratios are denoted as R1 and R2, and their calculation formulas are as follows:

[0110]

[0111] Determine the relationship between R1, R2 and the threshold T respectively:

[0112] If R1 < T, then point p and point q are successfully matched; otherwise, the matching fails.

[0113] If R2 < T, then point n and point r are successfully matched; otherwise, the matching fails.

[0114] Keep pairs of points that are identical in two matching results;

[0115] Four pairs of points are randomly selected. The distance between the remaining pairs of points and the pre-matched pairs after coordinate transformation is calculated using the parameters of the affine transformation model. Incorrectly matched pairs of points are eliminated to obtain one-to-one matching pairs between the source ticket image and the standard ticket image. Successfully matched ticket images are obtained. The Random Sampling Consensus Algorithm (RANSAC) has high robustness and is often used to eliminate mismatched pairs of points to achieve fine matching.

[0116] The above-mentioned method for recognizing content from a ticket image includes a step 5 method for preprocessing the ticket image to obtain a preprocessed image, which is described below:

[0117] After scaling each ticket image to 224×224, a ResNet model is used to identify the orientation of the resized ticket images. Ticket images with a non-zero rotation angle are rotated to obtain a positive image as the preprocessed image, or ticket images with a zero rotation angle are directly output as the preprocessed image.

[0118] The residual network model is trained using the slim framework in TensorFlow with input data of ticket data at 0°, 90°, 180°, 270° and rotated ticket data with angle labels. When the entire ticket recognition system starts running, when a ticket image is input, the image orientation recognition model is first called to identify the image at four angles: 0°, 90°, 180°, and 270°. The image can then be adjusted to be close to 0° through rotation transformation before subsequent detection and recognition.

[0119] The above-mentioned method for recognizing content from a ticket image includes a step 6 method for detecting target text in the preprocessed image to obtain the target text image, which is described below:

[0120] The target text image was obtained by annotating the fields to be detected in the preprocessed image using the open-source image annotation tool (LabelMe) and then inputting it into the target text detection model. The target text image was obtained by detecting the invoice code, invoice number, invoice date and pre-tax amount under PSENet (a semantic segmentation-based text detection network).

[0121] To improve annotation efficiency, semi-supervised learning is used. First, a small subset of ticket images is annotated. Then, a text detection model with relatively low accuracy is trained. This pre-trained model is used to detect the remaining unannotated ticket images. The resulting detections should show that most are correct, requiring only manual correction of the few incorrect detections. This significantly improves annotation efficiency and reduces workload. This process is repeated to obtain all the annotation results. These annotations, after verification, are used as input to train the text detection network.

[0122] The above-mentioned method for recognizing content from invoice images includes a step 7 method for performing specific field detection on the target text image to obtain a text image labeled with specific fields, which is described below:

[0123] The OpenCV minAreaRect() function is used to fit a minimum bounding rectangle to the bounding box of the detected target text image, and the coordinates of the minimum bounding rectangle and the corresponding rotation angle relative to the X-axis are obtained.

[0124] The `boundingRect()` function of OpenCV is used to obtain a bounding box without rotation angle for the target text image. Based on this bounding box, the text data image can be extracted from the original ticket image.

[0125] After rotating the text data image according to the rotation angle along the X-axis, a projection cutting method is used to remove residual boundaries to obtain a usable text data image.

[0126] After scaling the useful text data image to 400×100, it is input into a specific field recognition model for detection to obtain text images labeled with specific fields. The text images contain five types: invoice code, invoice number, pre-tax amount, invoice time, and "other". The labels 0, 1, 2, 3, and 4 represent these five types respectively.

[0127] The method for constructing the text recognition model in step 8 is as follows:

[0128] The simulated invoice text image data was used as input data to construct a text recognition model using DenseNet+BiLSTM+CTC.

[0129] In text recognition, DenseNet is only used for feature extraction, therefore its final GlobalPool and SoftMax layers need to be removed. For the logits extracted by DenseNet, it is first necessary to use...

[0130] `tf.transpose(logits,[0,1,3,2])` is used to swap the width and channel dimensions to merge the height and channel dimensions. Next, keeping the number dimension (first dimension) and width dimension unchanged, the height and channel dimensions are merged to obtain a new `logits` with only three dimensions. `tf.transpose(logits,[0,2,1])` is used again to swap the new height and width dimensions to input the features into the BiLSTM network. Then, a bidirectional LSTM network is used to further extract features from the features extracted by DenseNet. The output of the BiLSTM network has already connected the forward and backward outputs, but this data does not meet the input requirements of CTC. Therefore, a fully connected layer needs to be added after the BiLSTM network. The number of elements is the number of characters in the current character set + 2 (one represents a space in CTC, and the other represents a blank space in CTC). After processing by the fully connected layer, the third dimension of the output of the BiLSTM network is fixed to a specified size, the first dimension remains unchanged, and the size of the second dimension represents the "frame number" in CTC. At this point, the features of the text image have been extracted and meet the input format requirements of CTC. Then, this result is input into CTC and decoded using the maximum probability path method to obtain the predicted text sequence. The predicted logits are then used to calculate CTCLoss with the ground truth corresponding to the text image to obtain the mean loss. The AdamOptimizer is used to minimize and optimize this loss with an initial learning rate of 0.01, which enables backpropagation of the loss and updates the network parameters.

[0131] During training, the batch size is 64, the dataset is read in shuffle mode by default, the maximum number of steps is 1e6, loss information is output every 100 steps, and the model is saved every 2000 steps, with a maximum of 4 most recently saved models retained. To more intuitively observe the difference between the training results and the ground truth, the CTC decoding results and the ground truth's EditDistance are output simultaneously during training. Finally, the identified invoice code, invoice number, invoice date, pre-tax amount, and other information are stored in a MySQL database for subsequent page data display and statistical analysis.

[0132] like Figure 2As shown, in a second aspect, a second embodiment, a ticket image content recognition system includes: a nonlinear diffusion scale space construction module 100, a point feature extraction module 200, a feature point descriptor construction module 300, a ticket image processing module 400, a preprocessing module 500, a target text detection module 600, a specific field detection module 700, and a field information extraction module 800.

[0133] The nonlinear diffusion scale space construction module 100 is used to construct a nonlinear diffusion scale space from the acquired source ticket image and the standard ticket image using nonlinear diffusion filtering.

[0134] The point feature extraction module 200 is used to divide the nonlinear diffusion scale space into grid cells after removing the first scale space in the nonlinear diffusion scale space using an image block strategy. It is also used to extract feature points in the grid cells of the nonlinear diffusion scale space after grid cell division to obtain stable and uniform point features.

[0135] The feature point descriptor construction module 300 is used to construct feature point descriptors based on point features using the gradient position orientation histogram algorithm;

[0136] The ticket image processing module 400 is used to perform feature matching on the feature points expressed by the feature point descriptor using a bilateral fast approximate nearest neighbor algorithm and to remove mismatched point pairs using a random sampling consensus algorithm to obtain matched point pairs and acquire successfully matched ticket images.

[0137] The preprocessing module 500 is used to preprocess the ticket image to obtain a preprocessed image;

[0138] The target text detection module 600 is used to perform target text detection on the preprocessed image to obtain the target text image;

[0139] The specific field detection module 700 is used to perform specific field detection on the target text image to obtain a text image labeled with the specific field;

[0140] The field information extraction module 800 is used to input text images into a text recognition model to obtain field information by recognizing the text content.

[0141] In a third aspect, a third embodiment, a processor is used to call and run a computer program from memory, causing a device with a chip mounted to perform any of the methods in the first aspect.

[0142] For example, memory can include random access memory, flash memory, read-only memory, programmable read-only memory, non-volatile memory, or registers, etc.

[0143] The processor can be a central processing unit (CPU) or a graphics processing unit (GPU). The memory can store executable instructions.

[0144] The processor can execute instructions stored in memory to implement the various processes described in this paper.

[0145] It is understood that the memory in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0146] The non-volatile memory can be ROM (Read-Only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically Erasable EPROM), or flash memory.

[0147] Volatile memory can be RAM (Random Access Memory), which is used as an external cache;

[0148] By way of example, but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDR SDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0149] In some implementations, the memory stores elements such as upgrade packages, executable units, or data structures, or subsets thereof, or extended sets thereof: operating systems and applications;

[0150] The operating system, which includes various system programs such as the framework layer, core library layer, and driver layer, is used to implement various basic business functions and handle hardware-based tasks.

[0151] Applications, encompassing various applications, are used to implement various application services. Programs implementing the methods of the embodiments of this invention can be included within these applications.

[0152] Those skilled in the art will understand that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware.

[0153] Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0154] Those skilled in the art can implement the described functions in different ways for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] In the embodiments of this application, the disclosed systems, apparatus and methods can be implemented in other ways;

[0156] For example, the division of units or modules is only a logical functional division, and there may be other division methods in actual implementation;

[0157] For example, multiple units, modules, or components can be combined or integrated into another system;

[0158] In addition, each functional unit or module in the embodiments of this application can be integrated into a processing unit or module, or it can exist as a separate physical entity, etc.

[0159] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0160] If the function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a machine-readable storage medium.

[0161] Therefore, the technical solution of this application can be embodied in the form of a software product, which can be stored in a machine-readable storage medium and can contain several instructions to cause an electronic device to execute all or part of the process of the technical solution described in the embodiments of this application.

[0162] The aforementioned storage media can include various media capable of storing program code, such as ROM, RAM, removable disk, hard disk, magnetic disk, or optical disk.

[0163] In summary, the document image content recognition method and system of the present invention, starting from the needs of the company's finance department for recognizing and storing invoice content, combines feature matching and machine learning algorithms to match document images and recognize field content. It can quickly and accurately extract and store important field information contained in document images according to business rules, solving the pain points of slow speed and low accuracy of manual document matching and content information extraction. It simplifies the workflow of the company's finance personnel, improves the efficiency of financial auditing, and helps the orderly operation of the company's finance department.

[0164] The specific embodiments of the invention have been described above. It should be understood that the invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood to be implemented in a manner common to the art; those skilled in the art can make various modifications or alterations within the scope of the claims, and make several simple deductions, variations or substitutions, which do not affect the substantive content of the invention.

Claims

1. A method for recognizing the content of a receipt image, characterized in that, Includes the following steps: Step 1: Use nonlinear diffusion filtering to construct a nonlinear diffusion scale space from the source ticket image and the standard ticket image; Step 2: After removing the first layer of scale space in the nonlinear diffusion scale space using an image segmentation strategy, the nonlinear diffusion scale space is divided into grid cells. After the nonlinear diffusion scale space is divided into grid cells, feature points are extracted in the grid cells to obtain stable and uniform point features. Step 3: Construct feature point descriptors using the gradient position orientation histogram algorithm based on the point features; Step 4: Use the bilateral fast approximate nearest neighbor algorithm to perform feature matching on the feature points expressed by the feature point descriptor, and use the random sampling consensus algorithm to remove mismatched point pairs to obtain matching point pairs and obtain the successfully matched ticket image; Step 5: Preprocess the ticket image to obtain a preprocessed image; Step 6: Perform target text detection on the preprocessed image to obtain the target text image; Step 7: Perform specific field detection on the target text image to obtain a text image labeled with specific fields; Step 8: Input the text image into the text recognition model to obtain field information through text content recognition. Specifically, the nonlinear diffusion equation L in the nonlinear diffusion scale space is obtained by constructing the source ticket image and the standard ticket image using the nonlinear diffusion filter. t for: ； Where div is the divergence operator and p(m,n,t) is the diffusion function. and Δ are the gradient and Laplace operator, respectively, and the above formula is the anisotropic diffusion equation; The diffusion function p(m,n,t) is expressed as: ； Where g(·) is the edge function, when g(·) is between 0 and 1, it can not only preserve the edge, but also sharpen the brightness edge, ‖‖ is the modulus operation, and ∇L(m, n, t) is the gradient after Gaussian filtering; When p(m,n,t) is a constant, it can be simplified to an isotropic diffusion equation: 。 2. The method for recognizing content in a ticket image as described in claim 1, characterized in that, The specific method in step 2 is as follows: The first scale space in the nonlinear diffusion scale space is removed using an image segmentation strategy; The nonlinear diffusion scale space of each layer is divided into grid cells; The number of feature points is obtained, the number of feature points in the nonlinear diffusion scale space after each layer of grid cell division is determined, and feature points are extracted. The feature points are extracted in blocks, corresponding to the corresponding positions in the original nonlinear diffusion scale space. The position of each block is determined according to the size and number of blocks, thus determining the position of the feature points in the original linear diffusion scale space. By statistically analyzing all feature points, a stable and uniform point feature is obtained.

3. The method for recognizing the content of a ticket image as described in claim 2, characterized in that, The specific method for step 3 is as follows: The feature points are corrected based on their location; The gradient position orientation histogram algorithm is used to construct feature point descriptors from the corrected feature points.

4. The method for recognizing the content of a ticket image as described in claim 3, characterized in that, The specific method for step 4 is as follows: Calculate the Euclidean distance between points q and m and point p, denoted as d. pq and d pm The Euclidean distances between points r and s and point n are denoted as d. nr and d ns The calculation formula is as follows: ； Where i is the lower bound of the Σ accumulation function, and points q and m are the nearest neighbor feature points q and m, respectively, of point p in the source document image denoted as I1, and the second nearest neighbor feature point m, respectively, in the standard document image denoted as I2. The corresponding feature vector is F. q F m F p Let r and s be the nearest and second nearest neighbor feature points of point n in I2 on I1, respectively, and their corresponding feature vectors be F. r F s F n ; Calculate d pq With d pm d nr With d ns The distance ratios are denoted as R1 and R2, and their calculation formulas are as follows: ； Determine the relationship between R1, R2 and the threshold T respectively: If R1 < T, then point p and point q are successfully matched; otherwise, the matching fails. If R2 < T, then point n and point r are successfully matched; otherwise, the matching fails. Keep pairs of points that are identical in two matching results; Four pairs of points are randomly selected. The distance between the remaining point pairs and the pre-matched point pairs after coordinate transformation is calculated using the affine transformation model parameters. Incorrectly matched point pairs are eliminated to obtain one-to-one matching point pairs between the source ticket image and the standard ticket image, and the successfully matched ticket image is obtained.

5. The method for recognizing the content of a ticket image as described in claim 4, characterized in that, The specific method for preprocessing the ticket image in step 5 to obtain the preprocessed image is as follows: After scaling each ticket image to 224×224, a residual network model is used to perform orientation recognition on the resized ticket images. Ticket images with a rotation angle of not 0 are rotated to obtain a positive image and output as the preprocessed image, or ticket images with a rotation angle of 0 are directly output as the preprocessed image. The residual network model is trained using the slim framework in TensorFlow with the ticket data at 0°, 90°, 180°, 270° and rotated ticket data with angle labels as input data.

6. The method for recognizing the content of a ticket image as described in claim 5, characterized in that, The specific method for obtaining the target text image by performing target text detection on the preprocessed image in step 6 is as follows: After the fields to be detected in the preprocessed image are annotated using an open-source image annotation tool, they are input into the target text detection model to detect the target text image, which includes the invoice code, invoice number, invoice date, and pre-tax amount.

7. The method for recognizing the content of a ticket image as described in claim 6, characterized in that, The specific method for performing specific field detection on the target text image in step 7 to obtain a text image labeled with specific fields is as follows: The minAreaRect() function of OpenCV is used to fit a minimum bounding rectangle to the detected target text image, and the coordinates of the minimum bounding rectangle and the corresponding rotation angle relative to the X axis are obtained. The `boundingRect()` function of OpenCV is used to obtain a bounding box without rotation angle for the target text image. Based on this bounding box, the text data image can be extracted from the original ticket image. After rotating the text data image according to the rotation angle along the X-axis, a projection cutting method is used to remove residual boundaries to obtain a useful text data image. The useful text data image is scaled to 400×100 and then input into a specific field recognition model for detection to obtain the text image labeled with specific fields; The method for constructing the text recognition model described in step 8 is as follows: The simulated ticket text image data was used as input data to construct the text recognition model using DenseNet+BiLSTM+CTC.

8. A document image content recognition system, characterized in that, It includes: a nonlinear diffusion scale space construction module, a point feature extraction module, a feature point descriptor construction module, a ticket image processing module, a preprocessing module, a target text detection module, a specific field detection module, and a field information extraction module; The nonlinear diffusion scale space construction module is used to construct a nonlinear diffusion scale space from the acquired source ticket image and the standard ticket image using nonlinear diffusion filtering. The point feature extraction module is used to remove the first layer of scale space in the nonlinear diffusion scale space using an image block strategy, and then divide the nonlinear diffusion scale space into grid cells. It is also used to extract feature points in the grid cells of the nonlinear diffusion scale space after grid cell division to obtain stable and uniform point features. The feature point descriptor construction module is used to construct feature point descriptors based on the point features using the gradient position orientation histogram algorithm; The ticket image processing module is used to perform feature matching on the feature points expressed by the feature point descriptor using a bilateral fast approximate nearest neighbor algorithm and to remove mismatched point pairs using a random sampling consensus algorithm to obtain matched point pairs and acquire successfully matched ticket images. The preprocessing module is used to preprocess the ticket image to obtain a preprocessed image; The target text detection module is used to perform target text detection on the preprocessed image to obtain a target text image; The specific field detection module is used to perform specific field detection on the target text image to obtain a text image labeled with specific fields; The field information extraction module is used to input the text image into a text recognition model to obtain field information through text content recognition. Specifically, the nonlinear diffusion equation L in the nonlinear diffusion scale space is obtained by constructing the source ticket image and the standard ticket image using the nonlinear diffusion filter. t for: ； Where div is the divergence operator and p(m,n,t) is the diffusion function. and Δ are the gradient and Laplace operator, respectively, and the above formula is the anisotropic diffusion equation; The diffusion function p(m,n,t) is expressed as: ； Where g(·) is the edge function, when g(·) is between 0 and 1, it can not only preserve the edge, but also sharpen the brightness edge, ‖‖ is the modulus operation, and ∇L(m, n, t) is the gradient after Gaussian filtering; When p(m,n,t) is a constant, it can be simplified to an isotropic diffusion equation: 。 9. A processor for retrieving and running a computer program from memory, causing a device with a chip mounted thereon to perform the method as described in any one of claims 1-7.

Images