Bill classification method and device, computing equipment and storage medium

Abstract

The invention discloses a bill classification method and device, computing equipment and a storage medium, and the method comprises the steps: carrying out the character line positioning and high-frequency feature elimination of an input image, carrying out the image classification through a deep learning network model, and outputting a plurality of candidate categories and corresponding posterior probabilities; keyword matching calculation, namely content classification, is carried out in the candidate categories according to the descending order of posterior probabilities; when the content classification does not have a satisfactory result, calculating a relative position relationship between every two keywords in keyword information according to the extracted keyword information of the to-be-classified bills to obtain a relationship matrix between the keywords of the keyword information of the to-be-classified bills; and performing matching calculation with a known keyword relationship matrix corresponding to the bill types of the candidate categories, namely performing relationship classification, and taking the candidate category with the maximum matching relationship as a classification result of the to-be-classified bills.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a bill classification method, device, computing equipment and storage medium. Background Art

[0002] In financial application scenarios, there are a large number of bill recognition and information structuring requirements. There are many types of tax bills, more than 30, and the information structuring method for actual application mainly relies on the heuristic extraction method of the tax bill format. Therefore, the classification of tax bills is crucial to information structuring. Once the classification is wrong, it may lead to the failure of information structuring to extract any important fields. The classification of tax bills is an image classification problem. The method that is easy to think of at present is to use deep learning tools, such as convolutional network models such as CNN or transformer for convolution classification.

[0003] The existing deep learning-based methods have greatly improved the accuracy compared to traditional image classification methods, but there are still the following problems. One is that the utilization of image content information is insufficient, such as the ineffective use of text content in the image and the correlation between text content. Second, when new styles of tax invoices with similar layouts (background color, shading) appear, the deep learning-based model needs to be retrained to solve the problem. Retraining is very costly in terms of economic and time costs in practical applications. At present, there are many types of bills, and many types of bills often have new models, such as fixed-rate, medical, general machine-printed, and highway tolls.

[0004] Therefore, it is urgent to study a bill classification method to improve the utilization of identification information and improve the applicability of the method so that there is no need to retrain the deep learning model when new styles of bills appear. Summary of the invention

[0005] This specification provides a bill classification method, apparatus, computing device and storage medium to overcome at least one technical problem existing in the related art.

[0006] According to a first aspect of an embodiment of this specification, a bill classification method is provided, comprising:

[0007] Obtain an input image of the bill to be classified, locate the text line of the input image, obtain the outer frame of the text line area of ​​the input image, perform Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtain image data after removing high-frequency features;

[0008] The image data after high-frequency features are removed is input into a trained deep learning network model, which calculates the matching degree between the image data and the existing image categories.

[0009] When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs the image as the corresponding image category as the classification result of the bill;

[0010] When the matching degree between the image data and one of the image categories is not greater than a preset first threshold, retaining a first number of candidate categories and corresponding posterior probabilities, and arranging the candidate categories from large to small according to the posterior probabilities;

[0011] According to the text line area identified in the bill to be classified, keyword information is extracted from the bill to be classified, and keyword matching calculation is performed on the keyword information and the existing keyword list of the candidate category established according to the bill type.

[0012] When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill;

[0013] When the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, and a relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category to obtain the candidate category with the largest matching relationship, and the candidate category is used as the classification result of the bill to be classified.

[0014] Preferably, the step of extracting keyword information from the bill to be classified based on the text line area identified on the bill to be classified, performing keyword matching calculation on the keyword information and the existing keyword list of the candidate category established according to the bill type, and taking the corresponding bill type as the classification result of the bill when the matching score of the keyword matching calculation is greater than a second threshold, includes:

[0015] According to the text line area identified in the bill to be classified, keyword information is extracted from the bill to be classified. The keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified is used as the keyword of the bill name. The keyword of the bill name is matched with the keyword of the candidate category. The matching calculation is performed one by one with the categories in the matching result in the order of the matching results to obtain the matching score of the matching calculation. The calculation method is:

[0016] Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1}, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then

[0017] ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1}

[0018] Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit; when the number of characters in the keyword information is greater than 3, the distance between the characters is less than or equal to 1 and hit;

[0019] When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

[0020] Preferably, after the step of extracting keyword information from the bill to be classified according to the text line area identified on the bill to be classified, and using the keyword extracted from the text line area corresponding to the upper middle position of the bill to be classified as the keyword of the bill name, the method further includes:

[0021] If no keyword is extracted from the text line area corresponding to the middle and upper part of the bill to be classified, that is, no keyword of the bill name is obtained, then

[0022] The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result,

[0023] If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold;

[0024] If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends;

[0025] If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

[0026] Preferably, when the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, a relationship matrix between the keywords of the keyword information of the bill to be classified is obtained, the relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category, a candidate category with the greatest matching relationship is obtained, and the step of using the candidate category as the classification result of the bill to be classified includes:

[0027] When the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, and the relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The calculation method is:

[0028] For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is:

[0029] v = ((x1-x2) / w, (y1-y2) / h);

[0030] For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows:

[0031]

[0032] in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors;

[0033] The relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows:

[0034] Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows:

[0035]

[0036] Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is:

[0037]

[0038] in is the number of keyword pairs in the lth category, and the final relationship classification result is w*:

[0039]

[0040] Among them, w* represents the average relationship matching score;

[0041] The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

[0042] According to a second aspect of the embodiments of this specification, a bill classification device is provided, comprising an image acquisition and preprocessing module, an image classification module, a content classification module and a relationship classification module, wherein

[0043] The image acquisition and preprocessing module is configured to acquire an input image of a bill to be classified, locate text lines on the input image, obtain an external frame of the text line area of ​​the input image, perform Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtain image data after removing high-frequency features;

[0044] The image classification module is configured to input the image data after removing high-frequency features into a trained deep learning network model, and the model calculates the matching degree between the image data and the existing image categories. When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs that the image is in the corresponding image category as the classification result of the bill;

[0045] The content classification module is configured to retain a first number of candidate categories and corresponding posterior probabilities when the matching degree between the image data and one of the image categories is not greater than a preset first threshold, and arrange the candidate categories from large to small according to the posterior probabilities; extract keyword information from the bill to be classified according to the text line area identified in the bill to be classified, perform keyword matching calculation on the keyword information and the existing keyword list of the candidate categories established according to the bill type, and when the matching score of the keyword matching calculation is greater than a second threshold, use the corresponding bill type as the classification result of the bill;

[0046] The relationship classification module is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, obtain a relationship matrix between the keywords of the keyword information of the bill to be classified, match the relationship matrix of the bill to be classified with the known keyword relationship matrix corresponding to the bill type of the candidate category, obtain the candidate category with the largest matching relationship, and use the candidate category as the classification result of the bill to be classified.

[0047] Preferably, the content classification module includes a first content classification submodule, wherein

[0048] The first submodule of content classification is configured to extract keyword information from the bill to be classified according to the text line area identified in the bill to be classified, use the keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified as the keyword of the bill name, match the keyword of the bill name with the keyword of the candidate category, and perform matching calculations with the categories in the matching results one by one in the order of the matching results to obtain a matching score of the matching calculation, which is calculated as follows:

[0049] Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1}, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then

[0050] ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1}

[0051] Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit; when the number of characters in the keyword information is greater than 3, the distance between the characters is less than or equal to 1 and hit;

[0052] When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

[0053] Preferably, the content classification module further includes a second content classification submodule, wherein

[0054] The second submodule of content classification is configured to: if no keyword is extracted from the text line area corresponding to the middle and upper position of the bill to be classified, that is, no keyword of the bill name is obtained, then

[0055] The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result,

[0056] If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold;

[0057] If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends;

[0058] If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

[0059] Preferably, the relationship classification module includes a relationship matrix calculation submodule and a relationship matrix matching submodule, wherein:

[0060] The relationship matrix calculation submodule is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, and obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the calculation method is:

[0061] For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is:

[0062] v = ((x1-x2) / w, (y1-y2) / h);

[0063] For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows:

[0064]

[0065] in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors;

[0066] The relationship matrix matching submodule is configured to perform matching calculations on the relationship matrix of the bill to be classified and the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows:

[0067] Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows:

[0068]

[0069] Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is:

[0070]

[0071] in is the number of keyword pairs in the lth category, and the final relationship classification result is w*:

[0072]

[0073] Among them, w* represents the average relationship matching score;

[0074] The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

[0075] According to a third aspect of an embodiment of this specification, a computing device is provided, including a storage device and a processor, wherein the storage device is used to store a computer program, and the processor runs the computer program to enable the computing device to execute the steps of a bill classification method.

[0076] According to a fourth aspect of the embodiments of this specification, a storage medium is provided, which stores a computer program used in the computing device, and when the computer program is executed by a processor, the steps of the bill classification method are implemented.

[0077] The beneficial effects of the embodiments of this specification are as follows:

[0078] The embodiments of this specification provide a bill classification method, apparatus, computing device and storage medium, which process the bill image to be identified through a three-level algorithm of image classification, content classification and relationship classification to achieve accurate classification of bill types. The method makes full use of bill identification information, improves the utilization rate of identification information, and thus improves the accuracy of identification. Furthermore, the method has good applicability, and through the cascading of content classification algorithm and relationship classification algorithm, the classification of new style bills is achieved, thereby avoiding or reducing the number of deep learning model training times.

[0079] The innovative features of the embodiments of this specification include:

[0080] 1. In this specification, a bill classification method with a three-level algorithm of image classification, content classification and relationship classification is proposed. The classification of bills is finally realized by combining image classification, content classification and relationship classification, so as to achieve a deeper utilization of information and improve classification accuracy. This is one of the innovative points of the embodiments of this specification.

[0081] 2. In this specification, a relationship classification method is proposed, including the calculation of the relationship matrix and the matching algorithm. By cascading the content classification algorithm and the relationship classification algorithm, the classification of new style bills is realized, avoiding or reducing the number of deep learning model training times, which is one of the innovative points of the embodiments of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of this specification or related technologies, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0083] Figure 1 A flowchart of a bill classification method provided in one embodiment of this specification;

[0084] Figure 2 A schematic diagram of the execution logic of a bill classification method provided in one embodiment of this specification;

[0085] Figure 3 A schematic diagram of the structure of a bill classification device provided in one embodiment of this specification;

[0086] Figure 4 A schematic diagram of the structure of a computing device provided in one embodiment of this specification;

[0087] Figure 5 A schematic diagram of the structure of a storage medium provided in one embodiment of this specification. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of this specification and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0090] The embodiments of this specification disclose a bill classification method, apparatus, computing device, and storage medium, which are described in detail below.

[0091] Figure 1 The following is a flow chart of a bill classification method provided in one embodiment of this specification. Figure 1 As shown, a bill classification method includes:

[0092] S110, obtaining an input image of the bill to be classified, locating the text line of the input image, obtaining an outer bounding box of the text line area of ​​the input image, performing Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtaining image data after removing the high-frequency features.

[0093] S120. Input the image data after removing high-frequency features into a trained deep learning network model. The model calculates the matching degree between the image data and the existing image categories. When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs that the image is in the corresponding image category as the classification result of the bill.

[0094] S130. When the degree of match between the image data and one of the image categories is not greater than a preset first threshold, retain a first number of candidate categories and corresponding posterior probabilities, and arrange the candidate categories in descending order according to the posterior probabilities; extract keyword information from the bill to be classified based on the text line area identified in the bill to be classified, perform keyword matching calculation on the keyword information and the existing keyword list of the candidate categories established according to the bill type; when the matching score of the keyword matching calculation is greater than a second threshold, use the corresponding bill type as the classification result of the bill.

[0095] The step of extracting keyword information from the bill to be classified based on the text line area identified by the bill to be classified, performing keyword matching calculation on the keyword information and the existing keyword list of the candidate category established according to the bill type, and taking the corresponding bill type as the classification result of the bill when the matching score of the keyword matching calculation is greater than a second threshold, includes:

[0096] According to the text line area identified in the bill to be classified, keyword information is extracted from the bill to be classified. The keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified is used as the keyword of the bill name. The keyword of the bill name is matched with the keyword of the candidate category. The matching calculation is performed one by one with the categories in the matching result in the order of the matching results to obtain the matching score of the matching calculation. The calculation method is:

[0097] Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1}, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then

[0098] ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1}

[0099] Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit. When the number of characters in the keyword information is greater than 3, the distance between characters is less than or equal to 1 and hit.

[0100] When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

[0101] If no keyword is extracted from the text line area corresponding to the middle and upper part of the bill to be classified, that is, no keyword of the bill name is obtained, then

[0102] The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result,

[0103] If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold;

[0104] If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends;

[0105] If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

[0106] S140. When the matching score calculated by keyword matching is not greater than the second threshold value, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, and a relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category to obtain the candidate category with the largest matching relationship, and the candidate category is used as the classification result of the bill to be classified.

[0107] Specifically, when the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated according to the keyword information extracted from the bill to be classified, and the relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The calculation method is:

[0108] For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is:

[0109] v = ((x1-x2) / w, (y1-y2) / h);

[0110] For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows:

[0111]

[0112] in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors;

[0113] Furthermore, the relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows:

[0114] Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows:

[0115]

[0116] Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is:

[0117]

[0118] in is the number of keyword pairs in the lth category, and the final relationship classification result is w*:

[0119]

[0120] Among them, w* represents the average relationship matching score;

[0121] The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

[0122] Figure 2 A schematic diagram of the execution logic of a bill classification method provided in an embodiment of this specification, such as Figure 2 As shown, in a specific embodiment, in the step of inputting an image, the input is a cropped bill image, which may be rotated at a small angle, and can be a photographed image or a scanned image.

[0123] In the image preprocessing 110 step, in the image-based classification model, the specific stroke information of the text is a low-level feature, which is available in different ticket types and is a common feature, not a distinguishing feature. If it exists in the classification model, it is equivalent to noise and will reduce the classification performance. The length and position of the text block as a whole have a distinguishing effect on the classification. Therefore, during the image preprocessing, the text line of the image is first located to obtain the outer frame of the text area, and then the text line area is Gaussian blurred to eliminate high-frequency features.

[0124] Text line detection and recognition: This step is not the focus of the present invention, and the existing text line detection model and recognition model can be used.

[0125] In the step of image classification 120, the image input in this embodiment is an image after the high-frequency information of the text line is removed, which is different from the existing method. The rest can be carried out with reference to the existing deep learning network image classification. When the model is output, Q1 recognition candidates and the corresponding posterior probabilities are retained in this embodiment, and the categories are arranged from large to small according to the size of the posterior probability. When a new style bill image appears, as long as its true category is in Q1, even if it is not the first candidate, it can be correctly identified through subsequent content classification and relationship classification.

[0126] In the step of content classification 130, first, a list of ticket type keywords is created, such as expressway toll {ikey1, ikey2, ...ikeym}, where i represents the i-th ticket type, and m represents the m key fields that need to be extracted in the information structuring process of the i-th ticket type. ikey1 represents a keyword string.

[0127] Secondly, the classification is performed when the bill title (the bill name in the upper part of the bill) is recognized. Under the condition that the text line is positioned and recognized correctly, the type of bill can be determined from the recognition result. Under this condition, the bill format classification is based on the following:

[0128] If the ms matching score is greater than 50%, it is identified as the title bill type. If the matching score is less than 50%, it enters the relationship classification.

[0129] Calculation of ms: Assume that Q1 bill keyword lists are matched with the text line recognition results in the current bill image, and the number of hit keyword entries is {k1, k2, ..., kQ1}. The number of keyword entries corresponding to Q1 bills is {K1, K2, ..., KQ1}, then

[0130] ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1}

[0131] Keyword matching hit is defined in the present invention as: when the number of keyword characters is less than or equal to 3, the string is completely matched and it is a hit; when the number of keyword characters is greater than 3, the edit distance is less than or equal to 1 and it is a hit.

[0132] Next, for the classification when the title is not recognized.

[0133] When ms is greater than 80%, it is recognized as the corresponding bill type. When ms is less than 50%, it is directly output as rejection, and the recognition ends. When ms is less than or equal to 80% and greater than or equal to 50%, it enters the relationship recognition.

[0134] For the step of relationship classification 140, relationship identification in this embodiment is defined as the relative position distance of keywords in the layout.

[0135] First, let’s talk about the calculation of the bill relationship matrix.

[0136] Traverse the samples of each bill, and calculate the relative position relationship between all the keywords such as {ikey1, ikey2, ...ikeym}. That is, after the calculation is completed, an mxm relationship matrix can be obtained.

[0137] Assuming that the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively, the relative position relationship of the two keywords on the sample image is vector v = ((x1-x2) / w, (y1-y2) / h).

[0138] For all training samples, the position relationship vectors of two keywords are averaged as the corresponding elements in the relationship matrix.

[0139]

[0140] This process is an offline calculation and can be regarded as a training process. If a new ticket type is added, just calculate the relationship matrix according to the above process.

[0141] Secondly, the relationship recognition results need to be calculated.

[0142] If the current bill image to be identified only matches one keyword, the identification is rejected and the identification process is exited.

[0143] Otherwise, the Q1 bill keyword list is matched with the text line recognition result in the current bill image, and the number of hit keyword entries is {k1, k2, ..., kQ1}. Traverse the kQ1 candidates to calculate the position relationship vector of each keyword on the current bill image and match the corresponding keyword relationship vector in the relationship matrix of the Q1 candidate bills. Matching mainly refers to the angle between the relationship vectors:

[0144]

[0145] Where l is the lth candidate bill type.

[0146] The average relationship matching of the lth class is:

[0147]

[0148] Where is the number of keyword pairs in the lth category. The final relationship classification result is:

[0149]

[0150] That is, the bill category with the largest average matching relationship is output as the final recognition result.

[0151] Figure 3 This is a schematic diagram of the structure of a bill classification device provided in one embodiment of this specification. Figure 3 As shown, a bill classification device 300 includes an image acquisition and preprocessing module 310, an image classification module 320, a content classification module 330 and a relationship classification module 340, wherein

[0152] The image acquisition and preprocessing module 310 is configured to acquire an input image of a bill to be classified, locate text lines on the input image, obtain an outer bounding box of the text line area of ​​the input image, perform Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtain image data after removing high-frequency features.

[0153] The image classification module 320 is configured to input the image data after removing high-frequency features into a trained deep learning network model, which calculates the matching degree of the image data with the existing image categories. When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs the image as the corresponding image category as the classification result of the bill.

[0154] The content classification module 330 is configured to retain a first number of candidate categories and corresponding posterior probabilities, and arrange the candidate categories in descending order according to the posterior probabilities, when the degree of match between the image data and one of the image categories is not greater than a preset first threshold value; extract keyword information from the bill to be classified based on the text line area identified in the bill to be classified, perform keyword matching calculation on the keyword information and the existing keyword list of the candidate categories established according to the bill type, and when the matching score of the keyword matching calculation is greater than a second threshold value, use the corresponding bill type as the classification result of the bill.

[0155] The content classification module includes a first content classification submodule, wherein

[0156] The first submodule of content classification is configured to extract keyword information from the bill to be classified according to the text line area identified in the bill to be classified, use the keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified as the keyword of the bill name, match the keyword of the bill name with the keyword of the candidate category, and perform matching calculations with the categories in the matching results one by one in the order of the matching results to obtain a matching score of the matching calculation, which is calculated as follows:

[0157] Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1}, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then

[0158] ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1}

[0159] Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit; when the number of characters in the keyword information is greater than 3, the distance between the characters is less than or equal to 1 and hit;

[0160] When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

[0161] The content classification module also includes a second content classification submodule, wherein

[0162] The second submodule of content classification is configured to: if no keyword is extracted from the text line area corresponding to the middle and upper position of the bill to be classified, that is, no keyword of the bill name is obtained, then

[0163] The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result,

[0164] If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold;

[0165] If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends;

[0166] If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

[0167] The relationship classification module 340 is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, obtain a relationship matrix between the keywords of the keyword information of the bill to be classified, match the relationship matrix of the bill to be classified with the known keyword relationship matrix corresponding to the bill type of the candidate category, obtain the candidate category with the largest matching relationship, and use the candidate category as the classification result of the bill to be classified.

[0168] The relationship classification module includes a relationship matrix calculation submodule and a relationship matrix matching submodule, wherein:

[0169] The relationship matrix calculation submodule is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, and obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the calculation method is:

[0170] For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is:

[0171] v = ((x1-x2) / w, (y1-y2) / h);

[0172] For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows:

[0173]

[0174] in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors;

[0175] The relationship matrix matching submodule is configured to perform matching calculations on the relationship matrix of the bill to be classified and the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows:

[0176] Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows:

[0177]

[0178] Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is:

[0179]

[0180] in is the number of keyword pairs in the lth category, and the final relationship classification result is w*:

[0181]

[0182] Among them, w* represents the average relationship matching score;

[0183] The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

[0184] Figure 4 This is a schematic diagram of a computing device provided in one embodiment of this specification. Figure 4 As shown, a computing device 400 includes a storage device 410 and a processor 420, wherein the storage device 410 is used to store a computer program, and the processor 420 runs the computer program to enable the computing device to execute the steps of the bill classification method.

[0185] Figure 5 This is a schematic diagram of the structure of a storage medium provided in an embodiment of this specification. Figure 5 As shown, a storage medium 500 stores a computer program 510 used in the computing device. When the computer program 510 is executed by a processor, the steps of the bill classification method are implemented.

[0186] In summary, the embodiments of this specification provide a bill classification method, apparatus, computing device and storage medium, which utilize image classification, content classification, and relationship classification to ultimately achieve bill classification, thereby achieving a deeper use of information, which not only improves classification accuracy, but also has good adaptability to new classification categories.

[0187] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0188] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bill classification method, characterized in that: include: Obtain an input image of the bill to be classified, locate the text line of the input image, obtain the outer frame of the text line area of ​​the input image, perform Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtain image data after removing high-frequency features; The image data after high-frequency features are removed is input into a trained deep learning network model, which calculates the matching degree between the image data and the existing image categories. When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs the image as the corresponding image category as the classification result of the bill; When the matching degree between the image data and one of the image categories is not greater than a preset first threshold, retaining a first number of candidate categories and corresponding posterior probabilities, and arranging the candidate categories from large to small according to the posterior probabilities; According to the text line area identified in the bill to be classified, keyword information is extracted from the bill to be classified, and keyword matching calculation is performed on the keyword information and the existing keyword list of the candidate category established according to the bill type. When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill; When the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, and a relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category to obtain the candidate category with the largest matching relationship, and the candidate category is used as the classification result of the bill to be classified.

2. The method according to claim 1, characterized in that The step of extracting keyword information from the bill to be classified based on the text line area identified by the bill to be classified, performing keyword matching calculation on the keyword information and the existing keyword list of the candidate category established according to the bill type, and taking the corresponding bill type as the classification result of the bill when the matching score of the keyword matching calculation is greater than a second threshold, includes: According to the text line area identified in the bill to be classified, keyword information is extracted from the bill to be classified. The keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified is used as the keyword of the bill name. The keyword of the bill name is matched with the keyword of the candidate category. The matching calculation is performed one by one with the categories in the matching result in the order of the matching results to obtain the matching score of the matching calculation. The calculation method is: Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1} respectively, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1} Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit; when the number of characters in the keyword information is greater than 3, the distance between the characters is less than or equal to 1 and hit; When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

3. The method according to claim 2, characterized in that After the step of extracting keyword information from the bill to be classified based on the text line area identified on the bill to be classified, and using the keyword extracted from the text line area corresponding to the upper middle position of the bill to be classified as the keyword of the bill name, the method further includes: If no keyword is extracted from the text line area corresponding to the middle and upper part of the bill to be classified, that is, no keyword of the bill name is obtained, then The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result, If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold; If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends; If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

4. The method according to claim 1, characterized in that The step of calculating the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified, when the matching score of the keyword matching calculation is not greater than the second threshold, obtaining a relationship matrix between the keywords of the keyword information of the bill to be classified, matching the relationship matrix of the bill to be classified with a known keyword relationship matrix corresponding to the bill type of the candidate category, obtaining a candidate category with the greatest matching relationship, and using the candidate category as the classification result of the bill to be classified includes: When the matching score calculated by keyword matching is not greater than the second threshold, the relative position relationship between every two keywords in the keyword information is calculated based on the keyword information extracted from the bill to be classified, and the relationship matrix between the keywords of the keyword information of the bill to be classified is obtained. The calculation method is: For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is: v = ((x1-x2) / w, (y1-y2) / h); For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows: in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors; The relationship matrix of the bill to be classified is matched with the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows: Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows: Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is: in is the number of keyword pairs in the lth category, and the final relationship classification result is w*: Among them, w* represents the average relationship matching score; The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

5. A bill classification device, characterized in that: It includes image acquisition and preprocessing module, image classification module, content classification module and relationship classification module. The image acquisition and preprocessing module is configured to acquire an input image of a bill to be classified, locate text lines on the input image, obtain an external frame of the text line area of ​​the input image, perform Gaussian blur processing on each text line area according to the text line area to eliminate high-frequency features, and obtain image data after removing high-frequency features; The image classification module is configured to input the image data after removing high-frequency features into a trained deep learning network model, and the model calculates the matching degree between the image data and the existing image categories. When the matching degree between the image data and one of the image categories is greater than a preset first threshold, the model outputs that the image is in the corresponding image category as the classification result of the bill; The content classification module is configured to retain a first number of candidate categories and corresponding posterior probabilities when the matching degree between the image data and one of the image categories is not greater than a preset first threshold, and arrange the candidate categories from large to small according to the posterior probabilities; extract keyword information from the bill to be classified according to the text line area identified in the bill to be classified, perform keyword matching calculation on the keyword information and the existing keyword list of the candidate categories established according to the bill type, and when the matching score of the keyword matching calculation is greater than a second threshold, use the corresponding bill type as the classification result of the bill; The relationship classification module is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, obtain a relationship matrix between the keywords of the keyword information of the bill to be classified, match the relationship matrix of the bill to be classified with the known keyword relationship matrix corresponding to the bill type of the candidate category, obtain the candidate category with the largest matching relationship, and use the candidate category as the classification result of the bill to be classified.

6. The device according to claim 5, characterized in that The content classification module includes a first content classification submodule, wherein The first submodule of content classification is configured to extract keyword information from the bill to be classified according to the text line area identified in the bill to be classified, use the keyword extracted from the text line area corresponding to the middle and upper position of the bill to be classified as the keyword of the bill name, match the keyword of the bill name with the keyword of the candidate category, and perform matching calculations with the categories in the matching results one by one in the order of the matching results to obtain a matching score of the matching calculation, which is calculated as follows: Suppose that the list of bill keywords of Q1 candidate categories is matched with the keyword information identified in the bill image to be classified, and the number of hit keywords is {k1, k2, ..., kQ1} respectively, and the number of keywords corresponding to Q1 bill categories is {K1, K2, ..., KQ1}, then ms=max{k1 / K1,k2 / K2,…kQ1 / KQ1} Among them, ms is the score calculated by matching. When the number of characters in the keyword information is less than or equal to 3, the string is completely matched and hit; when the number of characters in the keyword information is greater than 3, the distance between the characters is less than or equal to 1 and hit; When the match score calculated by keyword matching is greater than the second threshold, the corresponding bill type is used as the classification result of the bill.

7. The device according to claim 5, characterized in that The content classification module also includes a second content classification submodule, wherein The second submodule of content classification is configured to: if no keyword is extracted from the text line area corresponding to the middle and upper position of the bill to be classified, that is, no keyword of the bill name is obtained, then The keyword information of the bill to be classified is matched with the keyword list of the corresponding candidate classification, and according to the matching calculation result, If the matching score is greater than a third threshold, the corresponding bill category with a matching score greater than the third threshold is taken as the classification result of the bill to be classified, and the third threshold is greater than the second threshold; If the matching score is less than the second threshold, the result of rejecting the recognition is output, and the recognition ends; If the matching score is less than or equal to the third threshold and greater than or equal to the second threshold, then, based on the keyword information extracted from the bill to be classified, the relative position relationship between every two keywords in the keyword information is calculated to obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the category is further identified based on the matching of the relationship matrix.

8. The device according to claim 5, characterized in that The relationship classification module includes a relationship matrix calculation submodule and a relationship matrix matching submodule, wherein: The relationship matrix calculation submodule is configured to calculate the relative position relationship between every two keywords in the keyword information extracted from the bill to be classified when the matching score of the keyword matching calculation is not greater than the second threshold value, and obtain the relationship matrix between the keywords of the keyword information of the bill to be classified, and the calculation method is: For two keywords of a bill, the upper left corner points of the two keywords are (x1, y1) and (x1, y2), and the image width and height are w and h respectively. Then the relative position relationship vector of the two keywords on the sample image is: v = ((x1-x2) / w, (y1-y2) / h); For all the keywords in the keyword information of the bill to be classified, the relative position relationship vectors of every two keywords are averaged as the corresponding elements in the relationship matrix, as follows: in, Represents the relative position relationship matrix, v i is the relative position vector between two keywords, and L is the number of vectors; The relationship matrix matching submodule is configured to perform matching calculations on the relationship matrix of the bill to be classified and the known keyword relationship matrix corresponding to the bill type of the candidate category, and the calculation method is as follows: Traverse the position relationship vectors of kQ1 candidate categories, match and calculate the keyword list of Q1 candidate categories with the keyword relationship matrix of the bill to be classified, and the number of hit keyword items is {k1, k2, …, kQ1} respectively. The matching calculation is the angle of the relationship vector, and the formula is as follows: Among them, i and j represent different keywords, l is the lth candidate category, and the average relationship matching of the lth category is: in is the number of keyword pairs in the lth category, and the final relationship classification result is w*: Among them, w* represents the average relationship matching score; The average matching relationship scores between the bills to be classified and the candidate bills are calculated, and the candidate category with the highest average matching relationship score is taken as the classification result of the bills to be classified.

9. A computing device, characterized in that The method comprises a storage device and a processor, wherein the storage device is used to store a computer program, and the processor runs the computer program to enable the computing device to perform the steps of the method according to any one of claims 1 to 4.

10. A storage medium, characterized in that: The computer program used in the computing device according to claim 9 is stored therein, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images