Document classification method, training method, device and storage medium

Abstract

This application provides a document classification method, training method, device, and storage medium. The method comprises: obtaining text recognition results for a document image to be processed; classifying the document image to be processed based on the text recognition results and a preset template library, and outputting the classification results for the document image to be processed, wherein the template library includes at least one type of document template. This application automatically identifies the type of a document image based on the text recognition results and the template library, resolving the drawback of conventional image classification methods that cannot process document images and improving the degree of automation in document image classification processing business scenarios.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a document classification method, training method, device and storage medium. Background Art

[0002] With the increasing degree of office electronicization, document materials originally stored in paper form are gradually converted into image form through electronic means such as scanners and stored and managed in different categories. In actual scenarios, different types of document materials have different uses, so document classification is particularly important in improving work efficiency.

[0003] There are over a hundred common document categories, and the differences between some categories are very small. For example, the difference between an outpatient invoice and an inpatient invoice is often reflected in just a few different characters. This large number of document categories and the small differences between them make the classification task very difficult. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a document classification method, training method, device and storage medium, which automatically identify the type of document image based on text recognition results and template library, solve the disadvantage that conventional image classification methods cannot process document images, and improve the degree of automation in business scenarios of document image classification processing.

[0005] A first aspect of an embodiment of the present application provides a document classification method, comprising: obtaining a text recognition result of a document image to be processed; classifying the document image to be processed according to the text recognition result and a preset template library, and outputting the classification result of the document image to be processed, wherein the template library includes at least one type of document template.

[0006] In one embodiment, obtaining the text recognition result of the document image to be processed includes: using optical character recognition technology to recognize the document image to be processed to obtain an initial recognition result, and the initial recognition result includes: the initial text line position and text content in the document image to be processed; normalizing the initial text line position to obtain the final text line position of the document image to be processed, wherein the text line position corresponds to the text content.

[0007] In one embodiment, the normalization processing of the initial text line position to obtain the final text line position of the document image to be processed includes: determining the base point position in the document image to be processed; calculating the horizontal coordinate difference and the vertical coordinate difference between each of the initial text line positions and the base point position; determining a first proportional value between the horizontal coordinate difference and the horizontal width value of the document image to be processed, and determining a second proportional value between the vertical coordinate difference and the vertical height value of the document image to be processed, using the first proportional value as the horizontal coordinate of the text line position, and using the second proportional value as the vertical coordinate of the text line position.

[0008] In one embodiment, the document image to be processed is classified according to the text recognition result and a preset template library, and the classification result of the document image to be processed is output, including: according to the text recognition result, respectively calculating the matching degree between the document image to be processed and each of the document templates in the template library; selecting the document template with a matching degree greater than a preset threshold as the target document template, and using the type of the target document template as the document type of the document image to be processed.

[0009] In one embodiment, it also includes: when there are multiple target document templates, selecting the document template with the greatest matching degree with the document image to be processed from the multiple target document templates, and using the type of the document template with the greatest matching degree as the document type of the document image to be processed.

[0010] In one embodiment, the degree of matching between the document image to be processed and each of the document templates in the template library is calculated based on the text recognition result, including: for each document template, calculating the degree of overlap and text similarity between each text line in the document image to be processed and each text line in the document template based on the text recognition result; selecting a target row set from the text recognition result, the degree of overlap corresponding to the target rows in the target row set being greater than a first threshold, and the text similarity corresponding to the target rows being greater than a second threshold; and taking the ratio of the number of target rows included in the target row set to the total number of text lines in the document template as the degree of matching between the document image to be processed and the document template.

[0011] In one embodiment, the document image to be processed is classified according to the text recognition result and a preset template library, and the classification result of the document image to be processed is output, including: according to the text recognition result, respectively calculating the matching degree between the document image to be processed and each of the document templates in the template library; selecting the document template with the largest matching degree as the target document template, and using the type of the target document template as the document type of the document image to be processed.

[0012] In one embodiment, the maximum value of the matching degree is greater than a preset threshold.

[0013] A second aspect of an embodiment of the present application provides a method for training a document template, comprising: obtaining sample text recognition results of multiple sample document images and a current template, wherein the document categories of the multiple sample document images are the same, and the current template and the document category of the multiple sample document images are the same; for each sample document image in the multiple sample document images, calculating the overlap between the kth sample text line in the current sample document image and each text line in the current template, and judging whether the maximum overlap is greater than a first threshold, wherein k is a positive integer; when the maximum overlap is greater than the first threshold, calculating the text similarity between the kth sample text line and the text line with maximum overlap, the text line with maximum overlap being the text line in the current template corresponding to the maximum overlap; judging whether the text similarity is greater than a second threshold; when the text similarity is greater than the second threshold, updating the current template according to the kth sample text line and the text line with maximum overlap.

[0014] In one embodiment, it also includes: when the maximum overlap is less than or equal to the first threshold, or when the text similarity is less than or equal to the second threshold, a training process is performed between the k+1th sample text line in the current sample document image and each text line in the current template, and all sample text lines in the current sample document image are traversed in turn to obtain the trained document template.

[0015] In one embodiment, the current template is updated based on the kth sample text line and the maximum overlap text line, including: determining the average text line position of the kth sample text line position and the maximum overlap text line position, and using the average text line position as the updated text line position; using the text content with higher confidence in the kth sample text line and the maximum overlap text line as the text content of the updated text line, to generate an updated document template.

[0016] In one embodiment, obtaining sample text recognition results of multiple sample document images includes: using optical character recognition technology to recognize the multiple sample document images to obtain initial recognition results, the initial recognition results including: the initial text line position and text content in the sample document images; normalizing the initial text line positions of the multiple sample document images to obtain the final text line positions of the multiple sample document images, wherein the text line positions correspond to the text content.

[0017] In one embodiment, the normalization processing of the initial text line positions of the multiple sample document images to obtain the final text line positions of the multiple sample document images includes: determining the base point position in the sample document image for each sample document image; calculating the horizontal coordinate difference and the vertical coordinate difference between each of the initial text line positions and the base point position; determining a first ratio value between the horizontal coordinate difference and the horizontal width value of the sample document image, and determining a second ratio value between the vertical coordinate difference and the vertical height value of the sample document image, using the first ratio value as the horizontal coordinate of the final text line position, and using the second ratio value as the vertical coordinate of the final text line position.

[0018] The third aspect of an embodiment of the present application provides a document classification device, including: a first acquisition module, used to obtain the text recognition result of the document image to be processed; a classification module, used to classify the document image to be processed according to the text recognition result and a preset template library, and output the classification result of the document image to be processed, wherein the template library includes at least one type of document template.

[0019] In one embodiment, the first acquisition module is used to: use optical character recognition technology to identify the document image to be processed to obtain an initial recognition result, and the initial recognition result includes: the initial text line position and text content in the document image to be processed; normalize the initial text line position to obtain the final text line position of the document image to be processed, wherein the text line position corresponds to the text content.

[0020] In one embodiment, the first acquisition module is also used to: determine the base point position in the document image to be processed; calculate the horizontal coordinate difference and the vertical coordinate difference between each initial text line position and the base point position; determine a first proportional value between the horizontal coordinate difference and the horizontal width value of the document image to be processed, and determine a second proportional value between the vertical coordinate difference and the vertical height value of the document image to be processed, and use the first proportional value as the horizontal coordinate of the text line position, and use the second proportional value as the vertical coordinate of the text line position.

[0021] In one embodiment, the classification module is used to: calculate the matching degree between the document image to be processed and each of the document templates in the template library based on the text recognition result; select the document template with a matching degree greater than a preset threshold as the target document template, and use the type of the target document template as the document type of the document image to be processed.

[0022] In one embodiment, the classification module is also used to: when there are multiple target document templates, select the document template with the greatest matching degree with the document image to be processed from the multiple target document templates, and use the type of the document template with the greatest matching degree as the document type of the document image to be processed.

[0023] In one embodiment, the classification module is used to: for each of the document templates, calculate the overlap and text similarity between each text line in the document image to be processed and each text line in the document template based on the text recognition results; select a target line set from the text recognition results, the overlap corresponding to the target lines in the target line set is greater than a first threshold, and the text similarity corresponding to the target lines is greater than a second threshold; and take the ratio of the number of target lines included in the target line set to the total number of text lines in the document template as the matching degree between the document image to be processed and the document template.

[0024] In one embodiment, the classification module is used to: calculate the matching degree between the document image to be processed and each of the document templates in the template library based on the text recognition result; select the document template with the largest matching degree as the target document template, and use the type of the target document template as the document type of the document image to be processed.

[0025] In one embodiment, the maximum value of the matching degree is greater than a preset threshold.

[0026] In a fourth aspect, an embodiment of the present application provides a training device for a document template, comprising: a second acquisition module for acquiring sample text recognition results of a plurality of sample document images and a current template, wherein the document categories of the plurality of sample document images are the same, and the current template is the same as the document category of the plurality of sample document images; a first calculation module for calculating, for each of the plurality of sample document images, the degree of overlap between the kth sample text line in the current sample document image and each text line in the current template, and determining whether the maximum degree of overlap is greater than a first threshold, wherein k is a positive integer; a second calculation module for calculating, when the maximum degree of overlap is greater than the first threshold, the text similarity between the kth sample text line and the text line with the maximum degree of overlap, the text line with the maximum degree of overlap being the text line in the current template corresponding to the maximum degree of overlap; a determination module for determining whether the text similarity is greater than a second threshold; and an update module for updating the current template according to the kth sample text line and the text line with the maximum degree of overlap when the text similarity is greater than the second threshold.

[0027] In one embodiment, an iterative module is also included, which is used to: when the maximum overlap is less than or equal to the first threshold, or when the text similarity is less than or equal to the second threshold, perform a training process between the k+1th sample text line in the current sample document image and each text line in the current template, and traverse all sample text lines in the current sample document image in turn to obtain the trained document template.

[0028] In one embodiment, the update module is used to: determine the average text line position of the kth sample text line position and the maximum overlap text line position, and use the average text line position as the updated text line position; use the text content with higher confidence in the kth sample text line and the maximum overlap text line as the text content of the updated text line to generate an updated document template.

[0029] In one embodiment, the second acquisition module is used to: use optical character recognition technology to identify the multiple sample document images to obtain an initial recognition result, which includes: the initial text line position and text content in the sample document image; normalize the initial text line positions of the multiple sample document images to obtain the final text line positions of the multiple sample document images, wherein the text line position corresponds to the text content.

[0030] In one embodiment, the second acquisition module is also used to: determine the base point position in each sample document image; calculate the horizontal coordinate difference and the vertical coordinate difference between each initial text line position and the base point position; determine a first proportional value between the horizontal coordinate difference and the horizontal width value of the sample document image, and determine a second proportional value between the vertical coordinate difference and the vertical height value of the sample document image, and use the first proportional value as the horizontal coordinate of the final text line position, and use the second proportional value as the vertical coordinate of the final text line position.

[0031] The fifth aspect of the embodiments of the present application provides an electronic device, comprising: a memory for storing a computer program; a processor for executing the computer program to implement the method of the first aspect of the embodiments of the present application and any embodiment thereof, or to implement the method of the second aspect of the embodiments of the present application and any embodiment thereof.

[0032] The sixth aspect of the embodiments of the present application provides a non-transitory electronic device readable storage medium, including: a program, which, when run by an electronic device, enables the electronic device to execute the method of the first aspect of the embodiments of the present application and any of its embodiments, or implement the method of the second aspect of the embodiments of the present application and any of its embodiments.

[0033] The document classification method, training method, device and storage medium provided in this application pre-set a template library containing multiple types of document templates. When there is a document image to be processed, text recognition is first performed on the document image, and then the type of the document image is automatically identified based on the text recognition result of the document image and the template library. This solves the problem that conventional image classification methods cannot process document images, improves the degree of automation in business scenarios of document image classification processing, and improves the accuracy of document type recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 This is a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0036] Figure 2 A flowchart of a document classification method according to an embodiment of the present application is shown;

[0037] Figure 3 A flowchart of a document classification method according to an embodiment of the present application is shown;

[0038] Figure 4A This is a schematic diagram of a short purchase order according to an embodiment of the present application;

[0039] Figure 4B A schematic diagram of a long purchase order according to an embodiment of the present application;

[0040] Figure 5 A schematic structural diagram of a document classification device according to an embodiment of the present application;

[0041] Figure 6 A schematic structural diagram of a document template training device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. In the description of the present application, the terms "first", "second", etc. are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0043] like Figure 1 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 1In the example, a processor 11 and a memory 12 are connected via a bus 10. Memory 12 stores instructions executable by processor 11. The instructions are executed by processor 11 so that electronic device 1 can execute all or part of the process of the method in the following embodiment. This solves the problem that conventional image classification methods cannot process document images, improves the degree of automation in business scenarios of document image classification processing, and improves the accuracy of document type recognition.

[0044] In one embodiment, the electronic device 1 may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, or the like.

[0045] Please see Figure 2 , which is a document classification method of an embodiment of the present application, the method can be Figure 1 The electronic device 1 shown is used to perform the method to solve the problem that conventional image classification methods cannot process document images, improve the business scenario automation level of document image classification processing, and improve the accuracy of document type recognition. The method includes the following steps:

[0046] Step 201: Obtain text recognition results of the document image to be processed.

[0047] In this step, the image of the document to be processed can be obtained in real time from the image data of the document to be classified. For example, in a real-world scenario, if hospital bill documents need to be classified, the image of the bill can be obtained in real time by scanning or taking a photo of the document to be processed. The image of the document to be processed can also be read from a pre-prepared image database. The text recognition result can be obtained by performing text recognition on the image of the document to be processed, or it can be directly read from a specified database. Different acquisition methods can be selected according to actual needs to increase flexibility.

[0048] In one embodiment, step 201 may specifically include: using optical character recognition technology to recognize the document image to be processed to obtain an initial recognition result, wherein the initial recognition result includes: initial text line positions and text content in the document image to be processed; normalizing the initial text line positions to obtain final text line positions in the document image to be processed, wherein the text line positions correspond to the text content.

[0049] In this step, OCR (Optical Character Recognition) technology can be used to recognize the document image to be processed in real time to obtain the text recognition result of the document image to be processed. At this time, the text recognition result includes at least: the text line position in the document image to be processed and the text content of each text line, wherein the text line position can be represented by the coordinates of the key points of the text line. For example, during the OCR recognition process, the minimum bounding box of a text line can be predicted based on the contour positioning mechanism. The corner points of the bounding box can be used as key points. Assuming that the bounding box is a rectangular box, the coordinate combination of the upper left corner and the lower right corner of the bounding box can be used to represent the text line position. Assuming that the coordinates of the upper left corner of the minimum bounding box of a text line are (x1, y1) and the coordinates of the lower right corner are (x2, y2), then its text line position can be represented by (x1, y1, x2, y2). In order to avoid the influence of offset and image scale inconsistency, the initial text line position can be normalized to improve the calculation accuracy.

[0050] Here, a text line refers to a text line recognized by the OCR algorithm. Its line distribution may be different from or the same as the line distribution of the actual text in the document to be processed. For example, if a line of actual text in the document to be processed is long, it may be divided into three text lines by the OCR algorithm.

[0051] In one embodiment, the initial text line position is normalized to obtain the final text line position of the document image to be processed, including: determining the base point position in the document image to be processed. Calculating the horizontal coordinate difference and the vertical coordinate difference between each initial text line position and the base point position. Determining a first ratio value between the horizontal coordinate difference and the horizontal width value of the document image to be processed, determining a second ratio value between the vertical coordinate difference and the vertical height value of the document image to be processed, using the first ratio value as the horizontal coordinate of the text line position, and using the second ratio value as the vertical coordinate of the text line position.

[0052] In this embodiment, a normalization operation can be performed based on the OCR recognition result of the document image to be processed: first, a base point position is selected as a reference position. The base point can be any point in the document image to be processed, but for the same document image to be processed, the base point is unique. Preferably, the base point can be a corner point of the minimum bounding box of a text line. For example, the text line located in the upper left corner can be selected, and the upper left corner point of the minimum bounding box of the text line can be used as the base point. The coordinates of the key points corresponding to each text line position are subtracted from the coordinates of the base point to obtain the horizontal coordinate difference and the vertical coordinate difference, and then the horizontal coordinate difference and the vertical coordinate difference are divided by the width and height of the image respectively. Specifically, the horizontal coordinate difference is divided by the width value of the document image to be processed, and the vertical coordinate difference is divided by the height value of the document image to be processed. This operation can avoid the influence of offset and inconsistent image scale and improve the calculation accuracy.

[0053] In one embodiment, assume that for a document image to be processed, the OCR algorithm generates two text lines. The initial text line positions are represented by keypoint coordinates: text line 1 is represented as [10, 6, 20, 30]. Text line 2 is represented as [5, 7, 20, 40]. The width of the document image to be processed is 100, and the height is 200.

[0054] Then the normalization process is:

[0055] 1. Get a point in the upper left corner of the document image to be processed as the base point position, the text line position: [5,6]

[0056] 2. Subtract the coordinates of the key points corresponding to the two text lines from the base point coordinates: [10-5, 6-6, 20-5, 30-6], [5-5, 7-6, 20-5, 40-6]. The difference in the horizontal and vertical coordinates for text line 1 is expressed as: [5, 0, 15, 24]. The difference in the horizontal and vertical coordinates for text line 2 is expressed as: [0, 1, 15, 34].

[0057] 3. Divide the horizontal and vertical coordinate differences by the width and height of the image to get the text line position composed of the first and second proportional values:

[0058] The text line position corresponding to text line 1 is expressed as: [(10-5) / 100, (6-6) / 200, (20-5) / 100,

[0059] The text line position corresponding to text line 2 is expressed as: [(5-5) / 100, (7-6) / 200, (20-5) / 100,

[0060] Step 202: Classify the document image to be processed according to the text recognition result and the preset template library, and output the classification result of the document image to be processed.

[0061] In this step, the template library includes one or more types of document templates, that is, the types of document templates are known. Here, the document template at least includes the position of the text line and the corresponding text content. The text recognition result of the document image to be processed is matched with the document templates in the template library, and then the classification result to which the document image to be processed belongs can be obtained.

[0062] The above document classification method, by presetting a template library containing document templates of multiple types, when there is a document image to be processed, first performs text recognition on the document image, and then automatically identifies the type to which the document image belongs based on the text recognition result of the document image and the template library, solves the drawback that the conventional image classification method cannot process document images, improves the automation degree of the business scenario of classifying and processing document images, and improves the accuracy of document type recognition.

[0063] Please refer to Figure 3 which is the document classification method of an embodiment of the present application. This method can be executed by the Figure 1 shown electronic device 1 to solve the drawback that the conventional image classification method cannot process document images, improve the automation degree of the business scenario of classifying and processing document images, and improve the accuracy of document type recognition. This method also includes a training process for the document template. The method specifically includes the following steps:

[0064] Step 301: Obtain the sample text recognition results and the current template of multiple sample document images.

[0065] In this step, the document categories of multiple sample document images are the same, and the current template is the same as the document categories of multiple sample document images. In the actual scenario, the template library can contain document templates of multiple types. Then, in the training stage, a batch of sample document images can be selected for different document types respectively, and these sample document images are type-labeled. For example, the sample images of outpatient bills are labeled as "outpatient bills", and the sample images of inpatient bills are labeled as "inpatient bills". Each type can include multiple sample document images. For example, this batch of sample document images includes N categories, and each category includes m sample document images, where m is a positive integer. To save training resources, it can be set that 1 < m <= 10. Then, training is performed on the multiple sample document images corresponding to each type to obtain the document template corresponding to that type.

[0066] You can select any sample document image's text recognition result from multiple sample documents of the same type as the current template, or specify another sample document image of the same type as the current template. To maintain consistency in the calculated data, the text line positions in the current template are normalized. The current template provides a starting state for the training process, and the final training results are obtained by continuously updating the current template.

[0067] It should be noted that different base point locations can significantly vary the normalization results. Therefore, for the same document template, the base point locations should be consistent during the normalization process during training and when using the same document template for classification.

[0068] In one embodiment, to avoid the effects of offset and image size, referring to the specific implementation of step 201 above, optical character recognition technology is used to recognize multiple sample document images to obtain initial recognition results, which include the initial text line positions and text content in the sample document images. The initial text line positions of the multiple sample document images are then normalized to obtain final text line positions for the multiple sample document images, where the text line positions correspond to the text content. For details, see the description of step 201 in the above embodiment.

[0069] In one embodiment, the initial text line positions of multiple sample document images are normalized to obtain the final text line positions of the multiple sample document images. Specifically, the normalization process may include: determining the base point position in each sample document image; calculating the horizontal coordinate difference and the vertical coordinate difference between each initial text line position and the base point position; determining a first ratio value between the horizontal coordinate difference and the horizontal width value of the sample document image, and determining a second ratio value between the vertical coordinate difference and the vertical height value of the sample document image, using the first ratio value as the horizontal coordinate of the final text line position, and using the second ratio value as the vertical coordinate of the final text line position. For details, please refer to the description of step 201 in the above embodiment.

[0070] Step 302: For each of the multiple sample document images, calculate the degree of overlap between the kth sample text line in the current sample document image and each text line in the current template. Determine whether the maximum overlap is greater than a first threshold. If so, proceed to step 303; otherwise, proceed to step 306.

[0071] In this step, it is assumed that this batch of sample document images contains N categories. For the i-th (1 <= i <= N) category of sample document images, the text recognition result of the first sample document image is selected as the current template. Then, the remaining m - 1 sample document images are traversed in sequence according to the current template to update the current template.

[0072] Specifically, take the j-th (1 < j <= m) sample document image as the current sample document. For the k-th sample text line of the current sample document, calculate the overlap degree with each text line of the current template. For example, the IOU (Intersection over Union) can be calculated to represent the overlap degree between the two. Here, 1 < k < G, and G is the total number of sample text lines in the OCR recognition result of the j-th sample document image. G is a positive integer. During the IOU calculation process, the input is the normalized positions of the sample text line and the text line of the current template, and the IOU between the two text lines is calculated. Then, select the maximum IOU value from the above calculation results. Determine whether this maximum IOU value is greater than the first threshold. If so, enter step 303; otherwise, enter step 306. The first threshold can be obtained based on empirical data statistics. For example, it can be set to values such as 0.5 or 0.8. For a scenario with strict requirements, it can be set to 0.8, and for a scenario with less strict requirements, it can be set to 0.5.

[0073] Step 303: Calculate the text similarity between the k-th sample text line and the text line with the maximum overlap degree. The text line with the maximum overlap degree is the text line in the current template corresponding to the maximum overlap degree. Then enter step 304.

[0074] In this step, if the maximum value of the IOU is greater than the set threshold 1, select a text line in the current template with the maximum IOU with the k-th sample text line. This text line is the text line with the maximum overlap degree. Calculate the text similarity between the k-th sample text line and the text line corresponding to the maximum overlap degree. The text similarity can be calculated through the corresponding text content. The text similarity between the text line with the maximum overlap degree and the k-th sample text line can further represent the matching degree between these two text lines.

[0075] In one embodiment, during the text similarity calculation process, the text content of the k-th sample text line and the text line with the maximum overlap degree can be extracted respectively, and the text content can be segmented. For example, the text content can be segmented with segmentation granularities such as words, original strings, syntactic analysis results, etc., and based on the segmentation results, text feature vectors corresponding to the k-th sample text line and the text line with the maximum overlap degree are constructed respectively. By calculating the distance between the two text feature vectors, such as calculating the Euclidean distance between the two text feature vectors, the text similarity between the two can be obtained.

[0076] Step 304: Determine whether the text similarity is greater than a second threshold. If so, proceed to step 305; otherwise, proceed to step 306.

[0077] In this step, the second threshold can be set based on actual needs. The larger the second threshold, the higher the accuracy. The second threshold can be obtained based on empirical statistics. The value range of the second threshold can be 0-1. For strict scenarios, the second threshold can be 0.8. If the similarity is greater than the set second threshold of 0.8, it means that the kth sample text line matches the text line with the maximum overlap, and then proceed to step 305.

[0078] Step 305: Update the current template based on the kth sample text line and the text line with the maximum overlap, and then proceed to step 306.

[0079] In this step, when the text similarity exceeds the second threshold, it indicates that the kth sample text line matches the text line with the maximum overlap. These two text lines are weighted and summed to obtain a new text line, which is then stored in the updated current template. This enables online updating of document templates. When new types of documents require classification, the aforementioned training method can be used to update the document templates in the template library, improving the automation level of document image classification in business scenarios.

[0080] In one embodiment, step 305 may specifically include determining an average text line position of the kth sample text line position and the maximum overlap text line position, using the average text line position as the updated text line position, and using the text content of the kth sample text line and the maximum overlap text line with the higher confidence as the text content of the updated text line to generate an updated document template.

[0081] Here we introduce the principle of weighted summation of the k-th sample text line and the text line with maximum overlap, as follows: Similar to the above description of step 201, the key point coordinates of the text line can be used to represent the text line position. For a text line, the coordinate combination of the upper left corner and the lower right corner of the minimum bounding box can be used to represent the text line position. Assuming that the coordinates of the upper left corner of the minimum bounding box of a text line are (x1, y1) and the coordinates of the lower right corner are (x2, y2), then its text line position can be represented by (x1, y1, x2, y2). Then assume that the k-th sample text line position is represented as (a, b, c, d) and the maximum overlap text line position is represented as (e, f, g, h). Then the average text line position is the sum of the corresponding coordinate values ​​of the two divided by 2. That is, the updated text line position can be expressed as For the updated text line content, the OCR algorithm can give the confidence level of each text line content. The higher the confidence level, the more reliable the recognition result. Therefore, the text line with the highest confidence level is selected as the updated text line content to improve the accuracy of the result.

[0082] Step 306: Perform a training process between the k+1th sample text line in the current sample document image and each text line in the current template, and traverse all sample text lines in the current sample document image in turn to obtain a trained document template.

[0083] In this step, if the maximum value of the IOU is determined to be less than or equal to the set first threshold 1 in steps 302 to 305, the iteration of the k+1th sample text line is performed in a manner similar to step 302. Or if the similarity in step 304 is less than or equal to the set second threshold 2, the iteration of the k+1th sample text line is performed in a manner similar to steps 302 to 305. Or after the comparison of the kth sample text line is completed in step 305, the iteration of the k+1th sample text line is performed in a manner similar to steps 302 to 305. After traversing all the sample text lines of the jth sample document image in sequence, the current template is updated to the new document template. After traversing all the sample document images of the i-th category in sequence, the final document template corresponding to the i-th category can be obtained and stored in the template library.

[0084] Step 307: Obtain the image of the document to be processed and use optical character recognition technology to recognize the image of the document to be processed to obtain an initial recognition result. The initial recognition result includes: the initial text line position and text content in the image of the document to be processed. For details, please refer to the description of step 201 in the above embodiment.

[0085] Step 308: Normalize the initial text line position to obtain the final text line position of the document image to be processed, where the text line position corresponds to the text content. For details, please refer to the description of step 201 in the above embodiment.

[0086] Step 309: Based on the text recognition result, the matching degree between the document image to be processed and each document template in the template library is calculated respectively.

[0087] In this step, the matching degree can represent the degree of similarity between the document image to be processed and the document template. The greater the matching degree, the greater the degree of similarity between the document image to be processed and the document template, and the greater the possibility that the document image to be processed and the document template belong to the same type. Therefore, assuming that there are multiple types of document templates in the template library, the matching degree between the document image to be processed and each document template in the template library can be calculated.

[0088] In one embodiment, step 309 may specifically include: for each document template, based on the text recognition results, respectively calculating the overlap and text similarity between each text line in the document image to be processed and each text line in the document template. For example, for the p-th text line in the document image to be processed, the overlap and similarity are calculated one by one with each text line in the document template until each text line in the document image to be processed is calculated. Then, based on the calculated overlap and text similarity, a target line set is selected from the text recognition results, the overlap corresponding to the target line in the target line set is greater than a first threshold, and the text similarity corresponding to the target line is greater than a second threshold. The ratio of the number of target lines included in the target line set to the total number of text lines in the document template is used as the matching degree between the document image to be processed and the document template. One document template corresponds to one such target line set, and the matching degree between each document template and the document image to be processed is calculated in the above manner.

[0089] In this embodiment, the overlap can be represented by IOU, and its calculation method and the setting of the first threshold can refer to the IOU method in the above step 302, and the setting of text similarity and the second threshold can refer to the similarity method in the above step 303.

[0090] For example, suppose the template contains three categories of document templates: A, B, and C. Among them, the document template of type A has 10 text lines (position and text content), the document template of type B has 20 text lines, and the document template of type C has 15 text lines. Input the OCR recognition result of a document image D to be processed. If 6 text lines in the document image D to be processed match those in the document template of type A (that is, the IOU of the text line positions of the two is greater than the first threshold and the similarity of the text content is greater than the second threshold), then the matching degree between the document image D to be processed and the document template of type A is 6 / 10. The matching degree calculation of the document image D to be processed and the document templates of type B and type C is also processed in the same way. This can further improve the calculation accuracy.

[0091] Step 310: Select a document template with a matching degree greater than a preset threshold as a target document template, and use the type of the target document template as the document type of the document image to be processed.

[0092] In this step, the preset threshold can be set based on actual scenarios or statistically derived from empirical data. The preset threshold can range from 0 to 1, for example, 0.8. This threshold is used to characterize the degree of match between documents of the same type. Only two documents with a degree greater than the preset threshold are considered to be of the same type. Therefore, a target document template is selected from the template library whose matching degree with the image of the document to be processed is greater than the preset threshold. The target document is of the same type as the image of the document to be processed, and the type of the image of the document to be processed is output.

[0093] In one embodiment, it also includes: when there are multiple target document templates, selecting the document template with the highest matching degree with the document image to be processed from the multiple target document templates, and using the type of the document template with the highest matching degree as the document type of the document image to be processed.

[0094] In real-world scenarios, there may be multiple target document templates in the template library whose matching degree with the document image to be processed exceeds the preset threshold. In this case, the type of the target document template with the highest matching degree is selected as the output type for the document image to be processed, further improving classification accuracy. If there is no target document template in the template library whose matching degree with the document image to be processed exceeds the preset threshold, "other" may be output, prompting the user that no matching type was found.

[0095] In one embodiment, after obtaining the degree of match between the document image to be processed and each document template in the template library in step 309, the document template with the highest degree of match can be selected from the template library as the target document template, and the type of the target document template can be used as the document type of the document image to be processed. Directly selecting the document template with the highest degree of match can reduce unnecessary calculations.

[0096] In real-world scenarios, the maximum match value might also be less than the preset threshold. In this case, the template library cannot find a document template of the same type as the document image being processed. This means the current template library cannot recognize the document image being processed. Instead of outputting "other," the document template type corresponding to the maximum match value could be displayed, prompting the user that no matching type was found. Therefore, the maximum match value must be greater than the preset threshold to avoid outputting "other" when it should be.

[0097] like Figure 4A This is a diagram of a short purchase order, such as Figure 4B The following is a diagram of a long purchase order. Assuming that the template library only supports the recognition of short purchase orders, the output of the two images above is:

[0098] Figure 4A Corresponding: "clsass_name":short_po.

[0099] Figure 4B Corresponding: "clsass_name": other.

[0100] In one embodiment, if one wants to improve the classification accuracy of an existing category or add a new category, one can refer to the training phase, re-label a small amount of data, perform online training, and update the template library.

[0101] Assume that 3 long purchase order pictures are selected for online real-time update of the template library. After the update, the template library supports the classification of long purchase orders. Then, you still need to enter the above Figure 4A and Figure 4B Two pictures, the output is:

[0102] Figure 4A Corresponding to "clsass_name":short_po.

[0103] Figure 4B Corresponding to "class_name":long_po.

[0104] Please see Figure 5 , which is a document classification device 500 according to an embodiment of the present application, which can be applied to Figure 1 The electronic device 1 shown here addresses the drawback that conventional image classification methods cannot process document images, improves the degree of automation in document image classification processing business scenarios, and improves the accuracy of document type recognition. The device includes: a first acquisition module 501 and a classification module 502. The principle relationship between each module is as follows:

[0105] The first acquisition module 501 is used to obtain the text recognition result of the document image to be processed. The classification module 502 is used to classify the document image to be processed according to the text recognition result and a preset template library, and output the classification result of the document image to be processed, wherein the template library includes at least one type of document template.

[0106] In one embodiment, the first acquisition module 501 is configured to use optical character recognition technology to recognize the document image to be processed and obtain an initial recognition result, which includes the initial text line position and text content in the document image to be processed. The initial text line position is normalized to obtain the final text line position of the document image to be processed, where the text line position corresponds to the text content.

[0107] In one embodiment, the first acquisition module 501 is further configured to: determine a base point position in the document image to be processed; calculate a horizontal coordinate difference and a vertical coordinate difference between each initial text line position and the base point position; determine a first ratio value between the horizontal coordinate difference and the horizontal width of the document image to be processed; determine a second ratio value between the vertical coordinate difference and the vertical height of the document image to be processed; use the first ratio value as the horizontal coordinate of the text line position; and use the second ratio value as the vertical coordinate of the text line position.

[0108] In one embodiment, the classification module 502 is configured to calculate, based on the text recognition results, a degree of match between the document image to be processed and each document template in the template library. The document template with a degree of match greater than a preset threshold is selected as the target document template, and the type of the target document template is used as the document type of the document image to be processed.

[0109] In one embodiment, the classification module 502 is also used to: when there are multiple target document templates, select the document template with the highest matching degree with the document image to be processed from the multiple target document templates, and use the type of the document template with the highest matching degree as the document type of the document image to be processed.

[0110] In one embodiment, classification module 502 is configured to, for each document template, calculate, based on the text recognition results, the degree of overlap and text similarity between each text line in the document image to be processed and each text line in the document template. A target line set is selected from the text recognition results, wherein the overlap corresponding to target lines in the target line set exceeds a first threshold and the text similarity corresponding to the target lines exceeds a second threshold. The ratio of the number of target lines in the target line set to the total number of text lines in the document template is used as the degree of match between the document image to be processed and the document template.

[0111] In one embodiment, the classification module 502 is used to: calculate the matching degree between the document image to be processed and each document template in the template library based on the text recognition results; select the document template with the largest matching degree as the target document template, and use the type of the target document template as the document type of the document image to be processed.

[0112] In one embodiment, the maximum value of the matching degree is greater than a preset threshold.

[0113] For a detailed description of the document classification device 500 , please refer to the description of the relevant method steps in the above embodiment.

[0114] Please see Figure 6 , which is a document template training device 600 of an embodiment of the present application, which can be applied to Figure 1The electronic device 1 shown is used to train a template library to address the drawback of conventional image classification methods that cannot process document images, thereby improving the degree of automation in document image classification processing business scenarios and increasing the accuracy of document type recognition. The device includes: a second acquisition module 601, a first calculation module 602, a second calculation module 603, a judgment module 604, and an update module 605. The principles of each module are as follows:

[0115] The second acquisition module 601 is used to obtain sample text recognition results and a current template of multiple sample document images, wherein the document categories of the multiple sample document images are the same, and the current template and the document category of the multiple sample document images are the same; the first calculation module 602 is used to calculate, for each sample document image in the multiple sample document images, the overlap between the kth sample text line in the current sample document image and each text line in the current template, and to determine whether the maximum overlap is greater than a first threshold, wherein k is a positive integer; the second calculation module 603 is used to calculate the text similarity between the kth sample text line and the text line with the maximum overlap when the maximum overlap is greater than the first threshold, and the text line with the maximum overlap is the text line in the current template corresponding to the maximum overlap; the judgment module 604 is used to determine whether the text similarity is greater than a second threshold; the update module 605 is used to update the current template according to the kth sample text line and the text line with the maximum overlap when the text similarity is greater than the second threshold.

[0116] In one embodiment, an iterative module 606 is also included, which is used to: when the maximum overlap is less than or equal to a first threshold, or when the text similarity is less than or equal to a second threshold, perform a training process between the k+1th sample text line in the current sample document image and each text line in the current template, and traverse all sample text lines in the current sample document image in turn to obtain a trained document template.

[0117] In one embodiment, the update module 605 is used to: determine the average text line position of the kth sample text line position and the maximum overlap text line position, and use the average text line position as the updated text line position; use the text content with higher confidence in the kth sample text line and the maximum overlap text line as the text content of the updated text line to generate an updated document template.

[0118] In one embodiment, the second acquisition module 601 is used to: use optical character recognition technology to identify multiple sample document images to obtain initial recognition results, which include: the initial text line position and text content in the sample document image; normalize the initial text line positions of multiple sample document images to obtain the final text line positions of multiple sample document images, wherein the text line position corresponds to the text content.

[0119] In one embodiment, the second acquisition module 601 is also used to: determine the base point position in the sample document image for each sample document image; calculate the horizontal coordinate difference and the vertical coordinate difference between each initial text line position and the base point position; determine a first ratio value between the horizontal coordinate difference and the horizontal width value of the sample document image, determine a second ratio value between the vertical coordinate difference and the vertical height value of the sample document image, use the first ratio value as the horizontal coordinate of the final text line position, and use the second ratio value as the vertical coordinate of the final text line position.

[0120] For a detailed description of the document template training device 600, please refer to the description of the relevant method steps in the above embodiment.

[0121] An embodiment of the present invention further provides a non-transitory electronic device readable storage medium, comprising: a program, which, when run on an electronic device, enables the electronic device to execute all or part of the process of the method in the above embodiment. The storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD). The storage medium may also include a combination of the above types of memories.

[0122] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A document template training method, characterized in that: include: Obtaining sample text recognition results of a plurality of sample document images and a current template, wherein the plurality of sample document images have the same document category, and the current template has the same document category as the plurality of sample document images; For each of the plurality of sample document images, calculating a degree of overlap between a k-th sample text line in the current sample document image and each text line in the current template, and determining whether a maximum degree of overlap is greater than a first threshold, where k is a positive integer; When the maximum overlap is greater than the first threshold, calculating the text similarity between the kth sample text line and the maximum overlap text line, the maximum overlap text line being the text line in the current template corresponding to the maximum overlap; Determining whether the text similarity is greater than a second threshold; When the text similarity is greater than the second threshold, updating the current template according to the kth sample text line and the text line with the maximum overlap, including: Performing a weighted summation on the kth sample text line and the text line with the maximum overlap to obtain a new text line, and storing the new text line in the current template to update the current template.

2. The method according to claim 1, characterized in that Also includes: When the maximum overlap is less than or equal to the first threshold, or when the text similarity is less than or equal to the second threshold, a training process is performed between the k+1th sample text line in the current sample document image and each text line in the current template, and all sample text lines in the current sample document image are traversed in turn to obtain the trained document template.

3. The method according to claim 1, characterized in that The content of each text line has a confidence level representing a recognition result, and updating the current template according to the kth sample text line and the text line with the maximum overlap includes: Determine an average text line position of the kth sample text line position and the maximum overlap text line position, and use the average text line position as the updated text line position; The text content with a higher confidence level in the k-th sample text line and the text line with the maximum overlap is used as the text content of the updated text line to generate an updated document template.

4. The method according to claim 1, wherein The step of obtaining sample text recognition results of multiple sample document images includes: Using optical character recognition technology to recognize the multiple sample document images to obtain initial recognition results, the initial recognition results including: initial text line positions and text content in the sample document images; The initial text line positions of the multiple sample document images are normalized to obtain final text line positions of the multiple sample document images, wherein the text line positions correspond to the text contents.

5. The method according to claim 4, characterized in that The normalizing of the initial text line positions of the plurality of sample document images to obtain final text line positions of the plurality of sample document images includes: For each sample document image, determining a base point position in the sample document image; Calculating the horizontal coordinate difference and the vertical coordinate difference between each of the initial text line positions and the base point position; Determine a first ratio value between the horizontal coordinate difference and the horizontal width value of the sample document image, determine a second ratio value between the vertical coordinate difference and the vertical height value of the sample document image, use the first ratio value as the horizontal coordinate of the final text line position, and use the second ratio value as the vertical coordinate of the final text line position.

6. A document classification method, using the document template training method according to any one of claims 1 to 5, characterized in that: include: Get the text recognition results of the document image to be processed; Classifying the document image to be processed according to the text recognition result and a preset template library, and outputting the classification result of the document image to be processed, wherein the template library includes at least one type of document template; The step of obtaining the text recognition result of the document image to be processed includes: Using optical character recognition technology to recognize the image of the document to be processed to obtain an initial recognition result, wherein the initial recognition result includes: the initial text line position and text content in the image of the document to be processed; The initial text line position is normalized to obtain the final text line position of the document image to be processed, wherein the text line position corresponds to the text content.

7. The method according to claim 6, characterized in that The normalizing process of the initial text line position to obtain the final text line position of the document image to be processed includes: Determining the base point position in the image of the document to be processed; Calculating the horizontal coordinate difference and the vertical coordinate difference between each of the initial text line positions and the base point position; Determine a first proportional value between the horizontal coordinate difference and the horizontal width value of the document image to be processed, determine a second proportional value between the vertical coordinate difference and the vertical height value of the document image to be processed, use the first proportional value as the horizontal coordinate of the text line position, and use the second proportional value as the vertical coordinate of the text line position.

8. The method according to claim 6, characterized in that The step of classifying the document image to be processed according to the text recognition result and a preset template library, and outputting the classification result of the document image to be processed, includes: Calculating the matching degree between the document image to be processed and each document template in the template library according to the text recognition result; The document template with the matching degree greater than a preset threshold is selected as the target document template, and the type of the target document template is used as the document type of the document image to be processed.

9. The method according to claim 8, characterized in that Also includes: When there are multiple target document templates, the document template with the greatest matching degree with the document image to be processed is selected from the multiple target document templates, and the type of the document template with the greatest matching degree is used as the document type of the document image to be processed.

10. The method according to claim 8, characterized in that The step of calculating the matching degree between the document image to be processed and each document template in the template library according to the text recognition result includes: For each of the document templates, respectively calculating the overlap and text similarity between each text line in the document image to be processed and each text line in the document template according to the text recognition result; Selecting a target row set from the text recognition result, wherein the overlap corresponding to the target row in the target row set is greater than a first threshold, and the text similarity corresponding to the target row is greater than a second threshold; The ratio of the number of target rows included in the target row set to the total number of text rows in the document template is used as the matching degree between the document image to be processed and the document template.

11. The method according to claim 6, characterized in that The step of classifying the document image to be processed according to the text recognition result and a preset template library, and outputting the classification result of the document image to be processed, includes: Calculating the matching degree between the document image to be processed and each document template in the template library according to the text recognition result; The document template with the greatest matching degree is selected as the target document template, and the type of the target document template is used as the document type of the document image to be processed.

12. The method according to claim 11, characterized in that The maximum value of the matching degree is greater than a preset threshold.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 12.

14. A non-transitory electronic device readable storage medium, characterized in that: The invention comprises: a program, which, when executed by an electronic device, causes the electronic device to execute the method according to any one of claims 1 to 12.

Images