Document recognition model training method and related device

Abstract

The invention provides a training method of a document recognition model and a related device, and relates to the technical field of artificial intelligence, and the method of one embodiment comprises the steps: recognizing a first document element of a sample image and a first confidence coefficient of the first document element through a first recognition model; identifying a first document category of the sample image through a second identification model, and determining an element weight of a first document element based on the first document category; calculating training loss based on the first confidence coefficient and the element weight; and updating the first recognition model based on the training loss to obtain an updated document recognition model. According to the scheme of the embodiment, the efficiency of training the document recognition model can be improved, and the accuracy of the document recognition model obtained through training can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for training a document recognition model and related devices. The related devices include a training device for the document recognition model, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence technology, there is a demand for document recognition, such as the need to recover documents. Taking document recovery as an example, it is usually for documents in image format. By recognizing the image, each document element in the image, such as text, title, table, QR code, etc., is extracted, and then the elements of the recognized document elements are recovered. For example, if the document element is text, the text is spliced, and if it is a table, the table is recovered. Then, based on the recovered elements, the final document can be recovered.

[0003] At present, when performing document recognition, it is usually performed through a document recognition model. For example, the document elements in an image are recognized through a document recognition model, and the recognized document elements are then restored using a corresponding element recovery method. However, the existing method of obtaining a document recognition model is inefficient, and the accuracy of the document recognition model needs to be improved. Summary of the invention

[0004] Based on this, it is necessary to provide a document recognition model training method and related devices that can improve the efficiency of determining the document recognition model and, on this basis, improve the accuracy of document recognition in order to address the above-mentioned technical problems. The related devices include a document recognition model training device, a computer device, a computer-readable storage medium and a computer program product.

[0005] In a first aspect, the present application provides a method for training a document recognition model, wherein the method comprises:

[0006] Recognize a first document element of the sample image and a first confidence of the first document element by using a first recognition model;

[0007] identifying a first document category of the sample image by a second recognition model, and determining an element weight of the first document element based on the first document category;

[0008] Calculating a training loss based on the first confidence and the element weight;

[0009] The first recognition model is updated based on the training loss to obtain an updated document recognition model.

[0010] In a second aspect, the present application also provides a training device for a document recognition model, the device comprising:

[0011] An element recognition module, configured to recognize a first document element of a sample image and a first confidence level of the first document element by using a first recognition model;

[0012] A document category recognition module, used for identifying the first document category of the sample image through a second recognition model;

[0013] A weight determination module, configured to determine an element weight of the first document element based on the first document category;

[0014] a loss determination module, configured to calculate a training loss based on the first confidence level and the element weight;

[0015] A model updating module is used to update the first recognition model based on the training loss to obtain an updated document recognition model.

[0016] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any of the above embodiments are implemented.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0018] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0019] Based on the training method, apparatus, computer device, computer-readable storage medium and computer program product of the document recognition model of the embodiment of the present application as described above, in the process of training the document recognition model, on the basis of identifying the first document element of the sample image through the first recognition model, the first document category of the sample image is also identified through the second recognition model, and the element weight of the first document element is determined in combination with the first document category, and the training loss is calculated based on the element weight and the first confidence of the first document element, that is, the influence of the document category is also considered when calculating the loss, so that the calculated training loss takes into account both the confidence of the recognition element category itself and the influence of the document category to which the sample image belongs. The calculated training loss has a high accuracy rate, which can improve the efficiency of determining the document recognition model, and can thereby improve the accuracy of the document recognition model obtained through training. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 is a schematic diagram of an application scenario of a training method for a document recognition model in an embodiment;

[0021] Figure 2 A flowchart of a method for training a document recognition model in one embodiment;

[0022] Figure 3 A flowchart of a method for training a document recognition model in another embodiment;

[0023] Figure 4 A schematic diagram of a process for determining element weights in one embodiment;

[0024] Figure 5 A schematic diagram of a flow chart of a method for determining element weights in another embodiment;

[0025] Figure 6 A schematic diagram of training a document recognition model in another embodiment;

[0026] Figure 7 A schematic diagram of the principle of a training method for a document recognition model in one embodiment;

[0027] Figure 8 A structural block diagram of a training device for a document recognition model in one embodiment;

[0028] Fig. 9 is an internal structure diagram of a computer device in one embodiment;

[0029] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0033] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0036] In the field of artificial intelligence technology, document recognition has become an important part of artificial intelligence. For example, document elements (such as tables, pictures, text, etc.) of each module on an image are identified, and the area of ​​each document element is determined. This application is usually considered a layout analysis task. After the document element is identified, the element recovery can be performed for the identified document element. For example, if the identified document element is a table, the table is restored by the table recovery algorithm. If the identified document element is text, the text recognition is performed by the OCR (Optical Character Recognition) algorithm to identify the text content, and the identified text is spliced ​​to obtain the final OCR recognition result. If the identified document element is a seal, the specific seal is identified by the seal recognition algorithm, and so on.

[0037] In the process of document recognition, images are usually recognized through a trained document recognition model to identify document elements contained in the image. However, in actual use, in the process of training a document recognition model, it takes a lot of time to train the document recognition model, and the possibility of improving the efficiency of training the document recognition model needs to be considered, and the accuracy of the document recognition model also needs to be improved.

[0038] After research, it was found that the document category of the image can be combined to realize the training of the document recognition model, so as to improve the efficiency of training the document recognition model and improve the recognition accuracy of the trained document recognition model.

[0039] Based on this, an embodiment of the present application provides a method for training a document recognition model to solve one or more of the problems mentioned above.

[0040] The training method of the document recognition model provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. In some embodiments, the terminal 10 communicates with the server 20 through the network, and the server 20 can obtain a sample image and obtain a document recognition model based on the sample image training. The document recognition model obtained by training can be deployed on the terminal 10 or on the server 20 to implement the document recognition process, wherein the sample image obtained by the server 20 can be pre-stored in the database or obtained from the terminal 10. In other embodiments, the document recognition model can be obtained by obtaining a sample image through the terminal 10 and training based on the sample image. The document recognition model obtained by training can be deployed on the terminal 10 or on the server 20 to implement the document recognition process. In still other embodiments, the process of training to obtain a document recognition model can also be completed through the cooperation between the terminal 10 and the server 20. The data storage system can store intermediate data that the server 20 needs to use or in the process of training to obtain a document recognition model, such as sample images, document recognition models, initial weights, etc.

[0041] The terminal 10 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 20 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0042] refer to Figure 2 As shown, the embodiment of the present application provides a training method for a document recognition model, which is described by taking application in a server 20 or a terminal 10 as an example, wherein the method includes:

[0043] Step S201: recognizing a first document element of a sample image and a first confidence level of the first document element by using a first recognition model.

[0044] The first recognition model is a model for identifying document elements in an image. In some embodiments, the first recognition model may be a model implemented based on a Region Proposal Network (RPN). The region proposal network uses an image as input data, determines a rectangular suggestion box, identifies a target box from the rectangular suggestion box, and determines a suggested category for each target box. The suggested category is the category of the identified document element. At the same time, the confidence of the identified document element and its category can be output, thereby obtaining the first document element and the first confidence. Among them, the specific model structure of the first recognition model can adopt an existing model structure, such as a region proposal network, or it can be implemented after improvement on the basis of a region proposal network. The embodiments of the present application do not limit this.

[0045] The sample images are images used to train the document recognition model, and each sample image may be annotated with the document elements present in the sample image and the location information of the document elements. The document elements and the location information of the document elements in the sample image may be annotated manually or by calling a preset annotation interface, and the present application embodiment does not impose any specific restrictions on this.

[0046] The first confidence level reflects the probability of the first document element being identified by the first recognition model, and characterizes the credibility of the recognition result of the first recognition model identifying the first document element. The first confidence level can be calculated based on the probability function of the first recognition model outputting the first document element.

[0047] Step S202: identifying a first document category of the sample image by using a second recognition model, and determining an element weight of a first document element based on the first document category.

[0048] The second recognition model is a model for identifying the document category to which the image belongs, where the document category may be, for example, contracts, bank statements, invoices, comics, etc., but not limited thereto. The second recognition model may be a model implemented by a classification network, such as a network including a convolutional layer and a fully connected layer, but not limited thereto, as long as it can achieve document classification recognition.

[0049] The element weight refers to a weight used to evaluate or adjust the first confidence of the first document element recognized by the first recognition model.

[0050] Step S203: Calculate the training loss based on the first confidence and the element weight.

[0051] There is no limitation on the method of calculating the training loss based on the first confidence and the element weight. The first confidence and the element weight may be calculated based on any possible loss function to obtain the training loss.

[0052] Step S204: updating the first recognition model based on the training loss to obtain an updated document recognition model.

[0053] After the training loss is obtained, there is no limitation on the method of updating the first recognition face model based on the training loss, for example, the model parameters of the first recognition model are updated based on the gradient step of the preset model parameters.

[0054] Based on the training method of the document recognition model of the embodiment of the present application as described above, in the process of training the document recognition model, on the basis of identifying the first document element of the sample image through the first recognition model, the first document category of the sample image is also identified through the second recognition model, and the element weight of the first document element is determined in combination with the first document category, and the training loss is calculated based on the element weight and the first confidence of identifying the first document element, that is, the influence of the document category is also considered when calculating the loss, so that the calculated training loss takes into account both the confidence of the recognized element category itself and the influence of the document category to which the sample image belongs. The calculated training loss has a high accuracy rate, which can improve the efficiency of determining the document recognition model, and can thereby improve the accuracy of the document recognition model obtained through training.

[0055] In some embodiments, Figure 3 As shown, the step S203 of the embodiment of the present application calculates the training loss based on the first confidence and the element weight, including step S2030: calculating the training loss based on the first confidence, the second confidence of the first document category, and the element weight;

[0056] At this time, the above step S204 of updating the first recognition model based on the training loss to obtain an updated document recognition model includes step S2040: updating the first recognition model and the second recognition model based on the training loss to obtain an updated document recognition model.

[0057] The second confidence level reflects the probability of the first document category being identified by the second recognition model, and characterizes the credibility of the recognition result of the first document category identified by the second recognition model. The second confidence level can be calculated based on the probability function of the first document category output by the second recognition model.

[0058] There is no limitation on the method of calculating the training loss based on the first confidence, the second confidence and the element weight. The first confidence, the second confidence and the element weight can be calculated based on any possible loss function to obtain the training loss.

[0059] Based on the method of this embodiment, the training patrol can be simultaneously combined with the second confidence calculation of the first document category, and the first recognition model and the second recognition model can be updated based on the training loss to obtain an updated document recognition model, that is, the document recognition model can simultaneously include a first recognition model for identifying document elements and a second recognition model for identifying document categories, so that the document recognition model finally obtained can not only identify document elements but also identify document categories, thereby improving the diversity of document recognition results, and in the case of document recovery, the content corresponding to the identified document elements can be further combined based on the identified document category to obtain the corresponding document of the document category, which helps to improve the efficiency of document recovery.

[0060] In some embodiments, determining the element weight of the first document element based on the first document category in the above step S202 may include:

[0061] When the first document category does not belong to a preset document category, the element weight of the first document element is set to a preset value.

[0062] Among them, the preset document category can be the category of the document that needs to be identified, and can be set based on the needs of document identification. The embodiment of the present application does not impose any specific restrictions on this.

[0063] Among them, the specific numerical value of the preset value is not limited. In some embodiments, the preset value can be set to a value less than 1 to reduce the impact of recognition training that does not belong to the preset document category on model convergence, which helps to further improve the efficiency of training document recognition models.

[0064] Based on the solution of this embodiment, the first document category does not belong to the preset document category, indicating that the identified first document category is not the expected document category, so the document element of the first document element is directly set to the preset value, which is fast and convenient and can help to further improve the efficiency of training the document recognition model.

[0065] In some embodiments, determining the element weight of the first document element based on the first document category in the above step S202 may include:

[0066] In the case that the first document category belongs to a preset document category, an element weight of the first document element is determined based on the document elements included in the first document category.

[0067] Based on the method in this embodiment, when the first document category belongs to the preset document category, it is indicated that the identified first document category is the expected document category, so that the element weight of the first document element can be determined in combination with the elements included in the first document category and the identified first document element, which can improve the accuracy of the determined element weight, and on this basis can improve the accuracy of the obtained training loss, which can help to further improve the efficiency of the training document recognition model.

[0068] In some embodiments, Figure 4 As shown, the above-mentioned determining the element weight of the first document element based on the document elements included in the first document category may include:

[0069] Step S401: obtaining an initial weight, where the initial weight is determined based on whether the document elements included in the first document category include the first document element.

[0070] Based on the first document category and the first document element, the relationship between the first document category and the first document element can be determined. For a certain document category, it may contain multiple document elements. For example, if the document category is a contract, the document elements it contains may include headers, footers, text, titles, tables, etc.; if the document category is a bank statement, the document elements it contains may include text, tables, footers, and seals; if the document category is a comic, the document elements it contains may include pictures, text, and QR codes.

[0071] Taking bank statements as an example, if the identified first document element is text, it can be considered that the document elements included in the first document category include the first document element. If the identified first document element is a picture, since bank statements usually do not include pictures, it can be considered that the document elements included in the first document category do not include the first document element.

[0072] Step S402: Determine the element weight of the first document element based on the initial weight.

[0073] Based on this embodiment, when determining the element weight, an initial weight is determined based on whether the document elements included in the first document category include the first document element, so that the element weight can be determined based on whether the first document element is included in the document elements of the first document category, so that the accuracy of the element weight finally obtained can be further improved, and on this basis, the accuracy of the obtained training loss can be further improved, which can help to further improve the efficiency of training the document recognition model.

[0074] The above method for determining the initial weight is not limited. In some embodiments, the initial weight can be set in the following manner:

[0075] In a case where the document elements included in the first document category include the first document element, the initial weight includes a first initial weight, and the first initial weight is greater than 1;

[0076] In a case where the document elements included in the first document category do not include the first document element, the initial weight includes a second initial weight, and the second initial weight is less than 1.

[0077] Based on this embodiment, when determining element weights, when the document elements included in the first document category include the first document element, it indicates that the currently identified first document element is relatively credible, and thus the initial weight can be set to a first initial weight greater than 1; and when the document elements included in the first document category do not include the first document element, the initial weight can be set to a second initial weight less than 1, so that the document recognition model can be learned more accurately, thereby further improving the accuracy and reliability of the determined element weights.

[0078] In some embodiments, Figure 5 As shown, the step S402 of determining the element weight of the first document element based on the initial weight includes:

[0079] Step S4020: Determine the element weight of the first document element based on the initial weight and the second confidence of the first document category.

[0080] Based on this embodiment, when determining the element weight of the first document element based on the initial weight, the influence of the second confidence in identifying the first document category is further considered, that is, the accuracy of the identified first document category is further considered, which helps to further improve the accuracy of the obtained element weight. On this basis, the accuracy of the obtained training loss can be further improved, which can help to further improve the efficiency of the training document recognition model.

[0081] In the process of determining the element weight of the first document element based on the initial weight and the second confidence, as described above, different initial weights can be set according to whether the document elements included in the first document category include the first document element, thereby determining the element weight in different ways.

[0082] Accordingly, in some embodiments, determining the element weight of the first document element based on the initial weight and the second confidence level includes:

[0083] Based on a processing method corresponding to the initial weight, the initial weight and the second confidence are calculated to obtain the element weight of the first document element.

[0084] Based on this embodiment, when determining element weights based on the initial weights and the second confidence, different processing methods are used based on the differences in the initial weights, that is, whether the document elements included in the first document category include the first document element. That is, different processing methods are used for different situations of whether the first document element is included in the document elements included in the first document category. This can help to further improve the accuracy of the obtained element weights, and on this basis can further improve the accuracy of the obtained training loss, which can help to further improve the efficiency of the training document recognition model.

[0085] The specific method of calculating and processing the initial weight and the second confidence based on the processing method corresponding to the initial weight to obtain the element weight of the first document element may include, in some embodiments:

[0086] When the initial weight is greater than 1, reducing the initial weight based on the second confidence level to obtain an element weight of the first document element;

[0087] When the initial weight is less than 1, the initial weight is increased based on the second confidence level to obtain the element weight of the first document element.

[0088] Based on this embodiment, when the initial weight is greater than 1, it means that the document elements included in the first document category include the identified first document element, and the initial weight itself has been set to a value greater than 1, and then the initial weight is appropriately reduced in combination with the second confidence level, which can avoid the deviation caused by the excessive weight caused by further increasing the initial weight. When the initial weight is less than 1, it means that the document elements included in the first document category do not include the identified first document element, and the initial weight itself has been set to a value less than 1, and then the initial weight is appropriately increased in combination with the second confidence level, which can avoid the deviation caused by the excessive weight caused by further reducing the initial weight, which can further improve the accuracy of the obtained element weight, and on this basis can further improve the accuracy of the obtained training loss, which can help to further improve the efficiency of obtaining the document recognition model.

[0089] There is no limitation on the manner of reducing or increasing the initial weight based on the second confidence level. In some specific examples, the following manner may be used:

[0090] When the initial weight is greater than 1, the difference between the initial weight and the first value is determined as the element weight of the first document element;

[0091] When the initial weight is less than 1, the sum of the initial weight and the first value is determined as the element weight of the first document element;

[0092] The first value is a product of 1 minus the second confidence level and a preset coefficient, and the preset coefficient is less than 1.

[0093] There is no limitation on the way of setting the specific numerical value of the preset coefficient. In some embodiments, the preset coefficient can be set to a value less than 1.

[0094] Based on this embodiment, when the initial weight is increased and then decreased, the product of the difference of 1 minus the second confidence level and the preset coefficient is performed, which helps to make the obtained element weight within a reasonable range and can further improve the accuracy of the obtained element weight.

[0095] In some embodiments, Figure 6 As shown, based on the above embodiment, it can also include:

[0096] Step S205: When the document category label of the sample image is a label annotated through a preset annotation interface, the first document category does not belong to the preset document category, and the second confidence of the first document category is greater than a preset threshold, adjust the document category label of the sample image based on the second confidence.

[0097] Based on this embodiment, when the sample image is not manually annotated but is annotated through a preset annotation interface to determine the document category label of the sample image, if the second confidence level is greater than the preset threshold, it means that the recognition result of the identified first document category is relatively accurate and reliable, and the first document category does not belong to the preset document category. Therefore, the document category label of the sample image can be manually adjusted, which can avoid the impact of sample noise and help improve the efficiency of training the document recognition model.

[0098] Based on the above-mentioned embodiments, some specific application examples are used for illustration. In the following embodiments, the document recognition model includes both the first recognition model and the second recognition model.

[0099] Before training the document recognition model in the embodiment of the present application, sample images may be obtained and initial weights may be determined.

[0100] The acquired sample images may include manually annotated sample data, i.e., the document category of the sample image is manually annotated, as well as the document elements contained in the sample image and the location information of the document elements. The acquired sample images may also include images annotated by calling a preset annotation interface, i.e., the document category of the sample image is annotated by calling a preset annotation interface, as well as the document elements contained in the sample image and the location information of the document elements. The existing method of annotating sample images through a preset annotation interface can already achieve a relatively high accuracy, for example, it can usually achieve an accuracy of more than 90%. Therefore, sample images annotated through a preset annotation interface can also realize the training and determination of a document recognition model.

[0101] After obtaining the sample image, the obtained sample can also be subjected to image enhancement processing to expand the number and richness of the sample image. The specific image enhancement processing method is not limited, such as randomly scaling the sample image, for example, the scaling range is 0.5 times to 2.0 times, and the scaled sample image is randomly rotated at a random angle, for example, the rotation angle is from 10 degrees counterclockwise to 10 degrees clockwise, and based on the randomly scaled size and the rotation angle, the position of the annotated document element is updated to obtain the sample image after image enhancement. In the following embodiment, the sample image used to train the document recognition model can include both the original sample image and the sample image after image enhancement.

[0102] When determining the initial weight, it can be based on the document elements contained in different document categories. If the document category contains a certain document element, the initial weight can be set to the first initial weight, otherwise the initial weight is set to the second initial weight. In view of the fact that the document elements contained in each document category may be different, the determined initial weight can be reflected in the form of a weight matrix. For example, the columns of the weight matrix represent each document element category (such as headers, footers, text, titles, tables, pictures, seals, QR codes, etc.), the rows of the weight matrix represent document categories (such as contracts, bank statements, comics, etc.), and the specific element values ​​in the weight matrix, such as the element value of the i-th row and j-th column, represent the weight of the document element of the j-th column corresponding to the document category of the i-th row.

[0103] The setting rule of the weight matrix may be that if the document category includes the document element, the corresponding weight is the first initial weight, which is greater than 1, for example, set to 1.2. By setting the first initial weight to be greater than 1, the recognition effect of the document recognition model in the document category can be made better. If the document category does not include the document category, the corresponding weight is the second initial weight, which is less than 1, for example, set to 0.8. For example, if the document element of the document category in the i-th row includes the document element in the j-th column, the element value of the i-th row and the j-th column is the first initial weight of 1.2, otherwise it is the second initial weight of 0.8.

[0104] After obtaining the sample image and the set weight matrix, the document recognition model can be trained to obtain the document recognition model. The principle diagram of training the document recognition model is as follows: Figure 7 shown.

[0105] After obtaining the sample image, the image features of the sample image are extracted. The method of extracting the image features can be the same as the existing method of extracting image features, and this embodiment does not impose any specific limitation on this.

[0106] The sample image is recognized by the first recognition model, the sample element in the sample image is recognized, and the recognized first document element, the position information of the first document element (which may be the position information determined by the rectangular frame shape), and the first confidence of the recognized first document element are output. In some embodiments, the first recognition model may first recognize a large number of candidate frames in the sample image, and then further crop the area of ​​the candidate frame on the image features to obtain the target frame, and then recognize and classify the target frame to obtain the first document element and the position information of the first document element. The embodiments of the present application do not impose specific restrictions on this.

[0107] The sample image is recognized by using the second recognition model to identify a first document category of the sample image and a second confidence level of the first document category.

[0108] If the first document category does not belong to the preset document category, the element weight of the first document element is set to a preset value, which is less than 1 and less than the second initial weight, for example, 0.5. In the case where the initial weight is set in the weight matrix as described above, the element values ​​in the weight matrix can be set to 0.5 to obtain the updated element weight. In some embodiments, if the first document category does not belong to the preset document category and the second confidence is greater than a preset threshold, for example, 0.95, the document category label of the sample image can be adjusted based on the second confidence, and then the training loss is calculated. At the same time, the sample image can be recorded to facilitate manual verification and confirmation of the accuracy of the annotation information of the sample image after the training is completed.

[0109] If the first document category belongs to the preset document category, the element weight of the first document element is obtained based on the second confidence and the initial weight. When the initial weight is set in a weight matrix, the element weight of the first document element can be calculated based on the initial weight combined with the second confidence after obtaining the corresponding initial weight from the weight matrix.

[0110] In some embodiments, the element weight of the first document element is calculated based on the initial weight combined with the second confidence level, which can be performed as follows:

[0111]

[0112] Among them, β1 is the weight of the document element determined based on the second confidence and the initial weight, is the initial weight of element category i, is the second confidence level, and k is a preset coefficient. The preset coefficient is a constant and is less than 1. For example, in some embodiments, the preset coefficient k may be 0.2.

[0113] After obtaining the element weight β1 of the document element determined based on the second confidence and the initial weight, the training loss is calculated based on the first confidence, the second confidence and the element weight. The specific method of calculating the training loss is not limited and can be determined based on the selected loss function. For example, in some embodiments, the training loss can be determined based on the product of the first confidence, the second confidence and the element weight.

[0114] Based on the obtained training loss, the first recognition model and the second recognition model are updated to obtain an updated document recognition model. The above process is repeated until the model training end condition is reached to obtain a determined document recognition model. The obtained document recognition model includes the updated first recognition model and the updated second recognition model, that is, the last updated first recognition model and the last updated second recognition model.

[0115] The specific setting of the model training end condition is not limited. For example, in some embodiments, it can be determined that the model training end condition is met when the number of training iterations reaches a preset number. In other embodiments, it can be determined that the model training end condition is met when the model converges, for example, when the calculated model loss reaches a stable state and no longer changes with the input. In still other embodiments, it can be determined that the model training end condition is met when any one of the training iterations reaches a preset number and the model converges. The embodiments of the present application do not impose specific restrictions on this.

[0116] Among them, after the training is completed, or after a round of training is completed and before the next round of training begins, if there are recorded sample images, the document elements and location information of the recorded sample images can be manually confirmed to update the training data set to further improve accuracy.

[0117] After the document recognition model is obtained through training, the document recognition model can be deployed to a device that needs to perform document recognition, such as a terminal, a server, etc., so that the device can recognize the document category, document elements, and location information of the document elements through the document recognition model. Based on the recognition of the document category, document elements, and location information of the document elements, the device can also perform further processing, including but not limited to recovering the document.

[0118] When a new document category needs to be added to the document recognition model, the new document category can be added based on the above preset document category, and the initial weight of the new document category can be added to the weight matrix. Sample images of the new document category are obtained, annotated (manually annotated and / or pre-annotated), and the above training process is performed again to obtain an updated document recognition model that can recognize the new document category and the document elements of the document category.

[0119] Based on the method of the embodiment of the present application as described above, while identifying the document elements, the document category is also identified, and the weight of the document element is determined in combination with the document category to train the loss, which can improve the efficiency of model training, accelerate convergence, and improve the accuracy of the document recognition model obtained by training. And in the case where the sample image is unlabeled data, if the confidence of the document category is greater than the preset threshold, the document category is dynamically adjusted, and the sample image is recorded for manual verification and adjustment, thereby reducing the impact of unlabeled noise data. In addition, the element weight is adjusted according to the confidence of the document category to avoid the problem of inaccurate weights caused by inaccurate document category classification, thereby improving the accuracy of the element weight. When a new document category is added, a new document recognition model can also be quickly iterated.

[0120] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0121] Based on the same inventive concept, the embodiment of the present application also provides a document recognition model training device for implementing the document recognition model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more document recognition models provided below can refer to the limitations of the document recognition model training method above, and will not be repeated here.

[0122] In one embodiment, Figure 8 As shown, a training device for a document recognition model is provided, comprising: an element recognition module 801, a document category recognition module 802, a weight determination module 803, a loss determination module 804, and a model update module 805, wherein:

[0123] An element recognition module 801 is used to recognize a first document element of a sample image and a first confidence level of the first document element by using a first recognition model;

[0124] A document category recognition module 802, configured to recognize a first document category of a sample image by using a second recognition model;

[0125] A weight determination module 803, configured to determine an element weight of a first document element based on a first document category;

[0126] A loss determination module 804, configured to calculate a training loss based on the first confidence level and the element weight;

[0127] The model updating module 805 is used to update the first recognition model based on the training loss to obtain an updated document recognition model.

[0128] In some embodiments, the loss determination module 804 is further configured to calculate a training loss based on the first confidence, the second confidence of the first document category, and the element weight;

[0129] The model updating module 805 is further used to update the first recognition model and the second recognition model based on the training loss to obtain an updated document recognition model.

[0130] In some embodiments, the weight determination module 803 is further configured to set the element weight of the first document element to a preset value when the first document category does not belong to a preset document category.

[0131] In some embodiments, the weight determination module 803 is further configured to determine the element weight of the first document element based on the document elements included in the first document category when the first document category belongs to a preset document category.

[0132] In some embodiments, the weight determination module 803 is further used to obtain an initial weight, where the initial weight is determined based on whether the document elements included in the first document category include the first document element; and based on the initial weight, determine the element weight of the first document element.

[0133] In some embodiments, when the document elements included in the first document category include the first document element, the initial weight includes a first initial weight, and the first initial weight is greater than 1; when the document elements included in the first document category do not include the first document element, the initial weight includes a second initial weight, and the second initial weight is less than 1.

[0134] In some embodiments, the weight determination module 803 is further configured to determine the element weight of the first document element based on the initial weight and the second confidence of the first document category.

[0135] In some embodiments, the weight determination module 803 is further configured to calculate the initial weight and the second confidence level based on a processing method corresponding to the initial weight to obtain the element weight of the first document element.

[0136] In some embodiments, the weight determination module 803 is also used to, when the initial weight is greater than 1, reduce the initial weight based on the second confidence level to obtain the element weight of the first document element; and when the initial weight is less than 1, increase the initial weight based on the second confidence level to obtain the element weight of the first document element.

[0137] In some embodiments, the weight determination module 803 is also used to determine the difference between the initial weight and the first numerical value as the element weight of the first document element when the initial weight is greater than 1; and to determine the sum of the initial weight and the first numerical value as the element weight of the first document element when the initial weight is less than 1; the first numerical value is 1 minus the difference between the second confidence level and the product of a preset coefficient, and the preset coefficient is less than 1.

[0138] In some embodiments, the device also includes: an adjustment module, which is used to adjust the document category label of the sample image based on the second confidence level when the document category label of the sample image is a label annotated through a preset annotation interface, the first document category does not belong to the preset document category, and the second confidence level of the first document category is greater than a preset threshold.

[0139] Each module in the training device of the document recognition model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data, such as sample images, weight matrices, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training method for a document recognition model is implemented.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for a document recognition model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0142] Those skilled in the art will understand that Fig. 9 , Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the document recognition model training method in any of the embodiments described above are implemented.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the document recognition model training method in any of the embodiments described above are implemented.

[0145] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the document recognition model training method in any of the embodiments described above.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training a document recognition model, characterized in that: The method comprises: Recognize a first document element of the sample image and a first confidence of the first document element by using a first recognition model; identifying a first document category of the sample image by a second recognition model, and determining an element weight of the first document element based on the first document category; Calculating a training loss based on the first confidence and the element weight; The first recognition model is updated based on the training loss to obtain an updated document recognition model.

2. The method according to claim 1, characterized in that The calculating the training loss based on the first confidence and the element weight comprises: calculating the training loss based on the first confidence, the second confidence of the first document category and the element weight; The updating of the first recognition model based on the training loss to obtain an updated document recognition model includes: updating the first recognition model and the second recognition model based on the training loss to obtain an updated document recognition model.

3. The method according to claim 1 or 2, characterized in that: The determining the element weight of the first document element based on the first document category includes: When the first document category does not belong to a preset document category, the element weight of the first document element is set to a preset value.

4. The method according to claim 1 or 2, characterized in that: The determining the element weight of the first document element based on the first document category includes: In the case that the first document category belongs to a preset document category, an element weight of the first document element is determined based on the document elements included in the first document category.

5. The method according to claim 4, characterized in that The determining the element weight of the first document element based on the document elements included in the first document category includes: Acquire an initial weight, where the initial weight is determined according to whether the document elements included in the first document category include the first document element; Based on the initial weight, an element weight of the first document element is determined.

6. The method according to claim 5, characterized in that In a case where the document elements included in the first document category include the first document element, the initial weight includes a first initial weight, and the first initial weight is greater than 1; In a case where the document elements included in the first document category do not include the first document element, the initial weight includes a second initial weight, and the second initial weight is less than 1.

7. The method according to claim 5, characterized in that Determining the element weight of the first document element based on the initial weight includes: determining the element weight of the first document element based on the initial weight and a second confidence of the first document category.

8. The method according to claim 7, characterized in that The step of determining the element weight of the first document element based on the initial weight and the second confidence of the first document category includes: Based on a processing method corresponding to the initial weight, the initial weight and the second confidence are calculated to obtain an element weight of the first document element.

9. The method according to claim 8, characterized in that The calculating and processing the initial weight and the second confidence based on the processing mode corresponding to the initial weight to obtain the element weight of the first document element includes: When the initial weight is greater than 1, reducing the initial weight based on the second confidence level to obtain an element weight of the first document element; When the initial weight is less than 1, the initial weight is increased based on the second confidence level to obtain an element weight of the first document element.

10. The method according to claim 9, characterized in that When the initial weight is greater than 1, determining the difference between the initial weight and the first value as the element weight of the first document element; When the initial weight is less than 1, determining the sum of the initial weight and the first value as the element weight of the first document element; The first value is a product of a difference between 1 and the second confidence level and a preset coefficient, and the preset coefficient is less than 1.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: When the document category label of the sample image is a label annotated through a preset annotation interface, the first document category does not belong to the preset document category, and the second confidence of the first document category is greater than a preset threshold, the document category label of the sample image is adjusted based on the second confidence.

12. A training device for a document recognition model, characterized in that: The device comprises: An element recognition module, configured to recognize a first document element of a sample image and a first confidence level of the first document element by using a first recognition model; A document category recognition module, used for identifying the first document category of the sample image through a second recognition model; A weight determination module, configured to determine an element weight of the first document element based on the first document category; a loss determination module, configured to calculate a training loss based on the first confidence level and the element weight; A model updating module is used to update the first recognition model based on the training loss to obtain an updated document recognition model.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images