Document image classification method, system and device, medium and program product

Abstract

The invention provides a document image classification method, system and device, a medium and a program product, and the method comprises the steps: carrying out the image classification of a to-be-classified document image through an image classification model, and obtaining a dimension vector of the image classification model; performing text extraction on the document image to be classified by using the OCR model, and performing word segmentation on the extracted text to obtain a word segmentation set; performing text classification on the word segmentation set by utilizing a text classification model to obtain a dimensional vector of the text classification model; and inputting the dimensional vector of the image classification model and the dimensional vector of the text classification model into a multi-modal model to obtain an image classification result of the document image to be classified. According to the method, the image classification model, the OCR model, the text classification model and the multi-modal model are combined, image classification, text extraction, text classification and multi-modal classification are carried out on the document images to be classified, the problems that real-time response is difficult and adjustment and optimization are difficult are solved, and the classification accuracy and the classification effect are improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, system, device, medium and program product for classifying document images. Background Technology

[0002] In many areas such as insurance claims, voucher redemption, and product after-sales service, it is necessary to classify the document images uploaded by applicants.

[0003] Existing image classification techniques primarily rely on image classification models, OCR (Optical Character Recognition) models, or multimodal models to process these document images individually. However, using these methods alone presents several problems:

[0004] • Difficulty in distinguishing image features: Especially for form-type images, the similarity of the forms is extremely high, making it difficult to distinguish them based on the appearance features of the images alone. This makes it difficult for image classification models to accurately identify them.

[0005] • Incomplete text extraction: Under different shooting conditions, even the best-performing OCR model cannot completely extract all text. Word overlap, character loss, and character recognition errors occur frequently. This affects the accuracy of first extracting text through the OCR model and then classifying it.

[0006] • High computational power requirements and slow response speed: Classification can also be performed by training multimodal models, but the huge number of parameters requires a lot of computational power, making it difficult to meet the needs of real-time response.

[0007] • Difficulty in multimodal optimization: When a multimodal model makes a classification error, due to the black-box nature of neural networks, existing technologies are almost unable to identify the problem. The only option is to accumulate more samples for retraining, but this process can have unpredictable effects on all classifications.

[0008] These issues mean that most self-service claims and reconciliation systems still require extensive manual review after initial model screening. When problems such as missing or incorrect documents are discovered, repeated rework is necessary, resulting in significant time and effort wasted. The inability to achieve real-time automatic image classification further contributes to declining customer satisfaction. Summary of the Invention

[0009] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing technologies that classify document images by using a single model, such as low accuracy, difficulty in real-time response, and difficulty in optimization. This disclosure provides a method, system, device, medium, and program product for classifying document images.

[0010] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0011] The first aspect of this disclosure provides a method for classifying document images, the classification method comprising:

[0012] The image classification model is used to classify the document images to be classified, and the dimensional vector of the image classification model is obtained.

[0013] The text of the document image to be classified is extracted using an OCR model, and the extracted text is segmented into words to obtain a word segmentation set.

[0014] The word segmentation set is classified using a text classification model to obtain the dimensional vector of the text classification model;

[0015] The image classification model's dimension vector and the text classification model's dimension vector are input into the multimodal model to obtain the image classification result of the document image to be classified.

[0016] Preferably, the step of using an image classification model to classify the document image to be classified and obtaining the dimension vector of the image classification model includes:

[0017] The image of the document to be classified is input into the image classification model to obtain the image classification result of the document to be classified.

[0018] In response to the image classification result not belonging to the object class or the image classification result belonging to the object class and the confidence level of the image classification result being less than or equal to a preset confidence level, the dimension vector of the image classification model is obtained.

[0019] Preferably, the step of classifying the word segmentation set using a text classification model to obtain the dimensional vector of the text classification model includes:

[0020] Input the word segmentation set into the text classification model to obtain the text classification result;

[0021] If the confidence level of the text classification result is less than or equal to the preset confidence level, obtain the dimension vector of the text classification model.

[0022] Preferably, the step of using an OCR model to extract text from the document image to be classified, and then segmenting the extracted text to obtain a segmentation set includes:

[0023] The document images to be classified are input into the OCR model for text extraction, and the target text with a text confidence score greater than the preset confidence score is obtained.

[0024] The target text is segmented into words, and words that do not belong to the vocabulary are filtered out to obtain a segmentation set;

[0025] And / or,

[0026] The multimodal model includes an attention module and a classification module; the step of inputting the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified includes:

[0027] The attention weights are obtained by inputting the dimension vector of the image classification model and the dimension vector of the text classification model into the attention module.

[0028] Based on the attention weights, the dimensional vectors of the image classification model and the text classification model are fused to obtain a fused dimensional vector;

[0029] The fused dimension vector is input into the classification module to obtain the image classification result of the document image to be classified.

[0030] Preferably, the classification method further includes:

[0031] In response to the image classification result belonging to the object class and the confidence level of the image classification result being greater than the preset confidence level, the image classification result of the document image to be classified is output;

[0032] And / or,

[0033] The classification method also includes:

[0034] If the confidence level of the text classification result is greater than the preset confidence level, the image classification result of the document image to be classified is output.

[0035] Preferably, the classification method further includes:

[0036] Obtain target text with a text confidence score greater than a preset confidence score, and perform word segmentation on the target text to obtain the word segmentation result;

[0037] The word segmentation results are filtered to obtain a word frequency and category matrix;

[0038] Merge the word segments of the unclassified document images for each category to obtain the merged word segment set;

[0039] Remove the segments that account for less than a preset percentage from the merged segment set to obtain the segment set after removing the segments;

[0040] The merged word segment set, after removing words whose proportion is less than the preset proportion, is segmented and deduplicated to obtain a frequent word set;

[0041] The word frequency and category matrix are filtered based on the frequent word set to remove word segments that are not in the frequent word set, resulting in new word frequencies and new category matrices.

[0042] A chi-square test was performed on the new word frequency and the new category matrix to obtain the test results;

[0043] Obtain the target word segmentation set in the test results where the assumed value is less than or equal to the preset assumed value;

[0044] A vocabulary is obtained based on the target word segmentation set.

[0045] A second aspect of this disclosure provides a classification system for document images, the classification system comprising:

[0046] The image classification module is used to classify the document images to be classified using an image classification model and obtain the dimensional vector of the image classification model.

[0047] The text extraction module is used to extract text from the document images to be classified using an OCR model, and to segment the extracted text into words to obtain a word segmentation set.

[0048] The text classification module is used to classify the word segmentation set using a text classification model and obtain the dimensional vector of the text classification model.

[0049] The multimodal classification module is used to input the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified.

[0050] Preferably, the image classification module includes:

[0051] An image classification unit is used to input the document image to be classified into an image classification model to obtain the image classification result of the document image to be classified.

[0052] The first acquisition unit is configured to acquire the dimension vector of the image classification model in response to the image classification result not belonging to the object class or the image classification result belonging to the object class and the confidence level of the image classification result being less than or equal to a preset confidence level.

[0053] Preferably, the text classification module includes:

[0054] The text classification unit is used to input the word segmentation set into the text classification model to obtain the text classification result;

[0055] The second acquisition unit is used to acquire the dimension vector of the text classification model in response to the confidence level of the text classification result being less than or equal to a preset confidence level.

[0056] Preferably, the text extraction module includes:

[0057] The text extraction unit is used to input the document image to be classified into the OCR model for text extraction and to obtain the target text with a text confidence score greater than the preset confidence score.

[0058] The word segmentation unit is used to segment the target text into words and filter out words that do not belong to the vocabulary to obtain a word segmentation set.

[0059] And / or,

[0060] The multimodal model includes an attention module and a classification module; the multimodal classification module includes:

[0061] The third acquisition unit is used to input the dimension vector of the image classification model and the dimension vector of the text classification model into the attention module to obtain the attention weights;

[0062] The fusion unit is used to fuse the dimensional vector of the image classification model and the dimensional vector of the text classification model based on the attention weights to obtain a fused dimensional vector;

[0063] The fourth acquisition unit is used to input the fused dimension vector into the classification module to obtain the image classification result of the document image to be classified.

[0064] Preferably, the classification system further includes:

[0065] The first output module is used to output the image classification result of the document image to be classified in response to the image classification result belonging to the object class and the confidence level of the image classification result being greater than the preset confidence level;

[0066] And / or,

[0067] The classification system also includes:

[0068] The second output module is used to output the image classification result of the document image to be classified in response to the text classification result having a confidence level greater than the preset confidence level.

[0069] Preferably, the classification system further includes:

[0070] The first acquisition module is used to acquire target text with a text confidence score greater than a preset confidence score, and to perform word segmentation on the target text to obtain the word segmentation result;

[0071] The first filtering module is used to filter the word segmentation results to obtain a word frequency and category matrix;

[0072] The merging module is used to merge the word segmentation sets of the unclassified document images of each category to obtain the merged word segmentation set;

[0073] The removal module is used to remove words whose proportion is less than a preset proportion from the merged word segmentation set, so as to obtain the word segmentation set after word removal;

[0074] The word segmentation and deduplication module is used to perform word segmentation and deduplication on the merged word segmentation set whose removal ratio is less than the preset ratio, and obtain the frequent word set.

[0075] The second filtering module is used to filter the word frequency and the category matrix based on the frequent word set to remove word segments that are not in the frequent word set, so as to obtain new word frequencies and new category matrices.

[0076] The verification module is used to perform a chi-square test on the new word frequency and the new category matrix to obtain the verification results;

[0077] The second acquisition module is used to acquire the target word segmentation set in the test results where the assumed value is less than or equal to the preset assumed value;

[0078] The third acquisition module is used to obtain a vocabulary list based on the target word segmentation set.

[0079] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein the processor executes the computer program to implement the document image classification method described in the first aspect.

[0080] The fourth aspect of this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document image classification method described in the first aspect.

[0081] The fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the document image classification method as described in the first aspect.

[0082] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0083] The positive and progressive effects of this disclosure are as follows:

[0084] This disclosure combines image classification models, OCR models, text classification models, and multimodal models to perform image classification, text extraction, text classification, and multimodal classification on document images to be classified. It avoids the problems of large number of parameters, difficulty in real-time response, and difficulty in optimization caused by processing all categories simultaneously through a single model, thereby improving the accuracy and classification effect of document image classification. Attached Figure Description

[0085] Figure 1 A flowchart illustrating the method for classifying document images provided in Embodiment 1 of this disclosure;

[0086] Figure 2 These are schematic diagrams of the structure of the multimodal models provided in embodiments 1 and 2 of this disclosure;

[0087] Figure 3 A schematic diagram of the modules of the document image classification system provided in Embodiment 2 of this disclosure;

[0088] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the method for classifying document images according to Embodiment 3 of this disclosure. Detailed Implementation

[0089] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0090] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0091] In this embodiment of the disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good morals.

[0092] Example 1

[0093] Figure 1 A flowchart of a method for classifying document images provided in Embodiment 1 of this disclosure is shown below. Figure 1 As shown, the classification method includes:

[0094] S1. Use an image classification model to classify the document images to be classified and obtain the dimension vector of the image classification model;

[0095] In this embodiment, the categories of document images include form-type and physical object-type;

[0096] The forms include ID cards, damage assessment reports, invoices, and vehicle registration certificates, while the physical items include accident scene photos, panoramic vehicle photos, and engine photos.

[0097] S2. Use the OCR model to extract text from the document images to be classified, and then segment the extracted text to obtain a segmentation set.

[0098] S3. Use a text classification model to classify the word segmentation set and obtain the dimensional vector of the text classification model;

[0099] In this embodiment, any text classification model can be used, but h neurons need to be set in the last hidden layer of the model.

[0100] S4. Input the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified.

[0101] In this embodiment, the dimensional vector `embedding_image` of the image classification model and the dimensional vector `embedding_nlp` of the text classification model are input into the multimodal model to obtain the image classification result of the document image to be classified. At this point, regardless of the confidence level, the classification process ends, and the image classification result of the document image to be classified is output.

[0102] In this embodiment, the image classification results include at least one of the following: ID card, damage assessment report, invoice, vehicle registration certificate, accident scene photos, vehicle panoramic photos, and engine photos.

[0103] This embodiment combines image classification models, OCR models, text classification models, and multimodal models to perform image classification, text extraction, text classification, and multimodal classification on the document images to be classified. This avoids the problems of large number of parameters, difficulty in real-time response, and difficulty in optimization caused by processing all categories simultaneously through a single model, thereby improving the accuracy and classification effect of document image classification.

[0104] In an optional embodiment, step S1 includes:

[0105] S11. Input the document image to be classified into the image classification model to obtain the image classification result of the document image to be classified.

[0106] S12. In response to the image classification result not belonging to the object class or the image classification result belonging to the object class and the confidence level of the image classification result being less than or equal to the preset confidence level, obtain the dimension vector of the image classification model.

[0107] In practice, due to significant differences in image features between the object category and the form category, image classification models can distinguish between them and differentiate the various categories within the object category as accurately as possible. Furthermore, the different categories within the form category may be confused with each other.

[0108] A classification model is constructed using a CNN convolutional network (e.g., by adding a fully connected layer after ResNet50 for fine-tuning, where the last hidden layer in the fully connected layer contains h neurons, such as h=64), and the specific image classification results are output (e.g., damage assessment report, vehicle damage photos, invoices, etc.).

[0109] If the obtained image classification result belongs to the object class, and the confidence level of the image classification result is greater than the preset confidence level (e.g., the preset confidence level is 95%), then the classification process ends, and the image classification result is directly output. Otherwise (i.e., the image classification result does not belong to the object class, or the image classification result belongs to the object class and the confidence level of the image classification result is less than or equal to the preset confidence level), the h-dimensional vector embedding_image of the last hidden layer of the image classification model is retained, and subsequent text classification steps continue to be executed.

[0110] It should be noted that the preset reliability should be set according to the actual situation. For example, the preset reliability can be set to 95% or other values. There are no specific restrictions here.

[0111] In an optional embodiment, step S3 includes:

[0112] S31. Input the word segmentation set into the text classification model to obtain the text classification result;

[0113] S32. In response to the confidence level of the text classification result being less than or equal to the preset confidence level, obtain the dimension vector of the text classification model.

[0114] In an optional embodiment, step S2 includes:

[0115] S21. Input the document images to be classified into the OCR model for text extraction, and obtain the target text with a text confidence score greater than the preset confidence score.

[0116] S22. Segment the target text into words and filter out words that do not belong to the vocabulary to obtain a word segmentation set;

[0117] In this embodiment, after the document image to be classified is processed by the OCR model, the target text with a text confidence level greater than a preset confidence level (e.g., a preset confidence level of 95%) is retained and then segmented into words. The intersection of the resulting word segmentation set with the vocabulary list Vocabulary is used as the input word segmentation list for the document image to be classified.

[0118] In the specific implementation process, OCR models (such as open source models like PaddleOCR) are used to extract the text from the document images to be classified, and only the target text with a text confidence level greater than the preset confidence level (e.g., the preset confidence level is 95%) is retained.

[0119] Vocabulary filtering: The target text is segmented, and words that do not belong to the vocabulary Vocab (obtained during training) are removed, resulting in the final word set W.

[0120] Running the text classification model: Input the word segmentation set W into the text classification model to obtain the text classification result c. If the confidence of the text classification result c is greater than the preset confidence (e.g., the preset confidence is 95%), the classification ends, and the image classification result of the document image to be classified is output. Otherwise (i.e., the confidence of the text classification result is less than or equal to the preset confidence), the h-dimensional vector embedding_nlp of the last layer of the text classification model is retained, and subsequent multimodal classification steps are performed.

[0121] It should be noted that even if the text extraction from a form is incomplete, a multimodal model can still be used for the final judgment; the structure of a multimodal model is as follows: Figure 2 As shown.

[0122] This embodiment trains a text classification model to distinguish different categories of form-type images. By establishing a vocabulary and pre-screening the training corpus, the number of invalid word segments entering the model can be reduced, minimizing interference from personalized content entered by users and OCR model recognition errors. This also reduces the number of model parameters and improves computation speed.

[0123] In an optional embodiment, the multimodal model includes an attention module and a classification module; step S4 includes:

[0124] S41. Input the dimension vector of the image classification model and the dimension vector of the text classification model into the attention module to obtain the attention weights;

[0125] In this embodiment, an attention module is constructed, consisting of an input layer, a hidden layer, and an output layer. The input layer receives two h-dimensional vectors (embedding_image and embedding_nlp) (i.e., the h-dimensional vectors of the image classification model and the text classification model), which are concatenated into a 2h-dimensional vector as input. The hidden layer can be a fully connected layer with a certain number of neurons (e.g., h neurons), using an activation function (such as ReLU) to introduce non-linearity. The output layer has h neurons, corresponding to h attention weights attn.

[0126] S42. Based on attention weights, the dimensional vectors of the image classification model and the text classification model are fused to obtain a fused dimensional vector;

[0127] In this embodiment, for the i-th dimension of the vector embedding_image ( The dimensional values ​​after fusion are shown in formula (1):

[0128] (1)

[0129] The goal is to use attention weights to reflect the importance of the vector `embedding_image` relative to the vector `embedding_nlp` in this dimension.

[0130] For the i-th dimension of the vector embedding_image ( The fused dimension values ​​are shown in formula (2):

[0131] (2)

[0132] Through this correspondence and weighted fusion process, the two input vectors can be fused according to the weights output by the attention module, so that the fused vector can better reflect the combined influence of the two vectors on the classification task.

[0133] In this embodiment, embedding_image represents the h-dimensional vector of the last hidden layer of the image classification model, and embedding_nlp represents the h-dimensional vector of the last layer of the text classification model. h represents the dimension of the vector, and attn represents the h-dimensional weight vector obtained by the first part of the attention module of the bimodal model. It is used to perform weighted fusion of the h-dimensional vector embedding_image of the image classification model and the h-dimensional vector embedding_nlp of the text classification model to obtain the fused h-dimensional vector.

[0134] S43. Input the fused dimension vector into the classification module to obtain the image classification result of the document image to be classified.

[0135] In this embodiment, a classification module is constructed, which consists of a multilayer perceptron (MLP). The input to the MLP is a dimensional vector after weighted fusion by an attention module. The MLP may include multiple hidden layers; for example, the first hidden layer has 2h neurons, the second hidden layer has h neurons, and an activation function (e.g., ReLU) is used. The last hidden layer has the same number of neurons as the number of classification categories, and outputs the image classification result of the document image to be classified.

[0136] In an optional embodiment, the classification method further includes:

[0137] In response to an image classification result belonging to the object class and the confidence level of the image classification result being greater than a preset confidence level, the image classification result of the document image to be classified is output.

[0138] In an optional embodiment, the classification method further includes:

[0139] If the confidence level of the text classification result is greater than the preset confidence level, the image classification result of the document image to be classified is output.

[0140] In an optional embodiment, the classification method further includes:

[0141] Obtain target text with a text confidence score greater than a preset confidence score, and perform word segmentation on the target text to obtain the word segmentation results;

[0142] The word segmentation results are filtered to obtain the word frequency and category matrix;

[0143] Merge the word segments of the unclassified document images for each category to obtain the merged word segment set;

[0144] Remove the segments that account for less than the preset percentage in the merged segment set to obtain the segment set after removing the segments;

[0145] The merged word segment set, after removing words whose proportion is less than the preset proportion, is segmented and deduplicated to obtain a frequent word set;

[0146] The word frequency and category matrix are filtered based on the frequent word set to remove word segments that are not in the frequent word set, resulting in a new word frequency and a new category matrix.

[0147] A chi-square test was performed on the new word frequency and the new category matrix to obtain the test results.

[0148] Obtain the target word segmentation set in the test results where the assumed value is less than or equal to the preset assumed value;

[0149] A vocabulary is obtained based on the target word segmentation set.

[0150] In the specific implementation process, the following steps are used to select word segments with strong ability to distinguish various forms of images, forming the final vocabulary list.

[0151] i. For form-type images, after passing through the OCR model, the target text with a text confidence level greater than the preset confidence level (e.g., a preset confidence level of 95%) is retained and word segmentation is performed. The word segmentation results are used to remove stop words, single Chinese characters, English letters, and numbers, resulting in a word frequency and category matrix, as shown in Table 1. 'word' represents word segmentation, and 'c' represents the image category.

[0152] Table 1

[0153] Image number word_1 word_2 … Image categories 1 1 (represents the number of times word_1 appears in the first image) 0 … c_1 2 1 3 … c_2 … … … … …

[0154] ii. For each form-type image category c_i, merge the word segmentation sets of all images under it. For each merged word segmentation set, remove word segments with a percentage less than a preset percentage (e.g., the preset percentage is 1%). Then, merge these word segmentation sets and remove duplicates according to word segmentation to obtain a vocabulary list (frequent word version).

[0155] iii. Filter the word frequency and category matrix obtained in step one (i.e., step i) using the vocabulary (frequent word version), remove those word segments that are not in the vocabulary, and obtain a new word frequency and category matrix (frequent word version).

[0156] iv. Perform a chi-square test on the word frequency and category matrix (frequent word version), with the image category column as the target. Based on the test results, only retain the word segments with a p-value ≤ 0.01. This set of word segments constitutes the final vocabulary, Vocabulary.

[0157] In practical implementation, for example, in an insurance claims scenario, classification tests were conducted on 300,000 photos (pixel range: 100,000-25,000,000 pixels, with a 1:1 ratio of physical object photos to form photos) across 15 types of document images, using an L40 graphics card as an example. The classification accuracy rates were as follows: Image classification + OCR model + keyword filtering: 90.3% accuracy; Image classification + text classification model (without a vocabulary): 95.5% accuracy; Image classification + text model (with a vocabulary): 98.0% accuracy; Image classification + text classification model (with a vocabulary) + bimodal classification: 99.1% accuracy. The computation latency was as follows: the average latency for classifying a single image was 520ms, and the maximum latency was 1150ms, achieving the real-time response requirement of an average latency of less than 1 second.

[0158] This disclosure creatively combines image classification, text classification, and multimodal fusion classification, replacing the traditional single-model processing method, thereby achieving high-precision real-time classification. The approach of this disclosure brings the following three significant advantages: 1) Reduced average computation latency: It can handle approximately 49% and 46% of document images (e.g., in insurance claims scenarios, the ratio of physical object photos to form photos is 1:1) in the image classification and text classification stages, respectively, with only 5% of complex cases requiring the final multimodal classification step. This hierarchical processing mechanism greatly reduces the overall computation time. 2) Improved classification accuracy: Each sub-model specializes in recognizing a specific type of document image; for example, the image classification model focuses on physical object images, while the text classification model focuses on form images. This division of labor avoids the massive number of parameters and tuning difficulty that a single multimodal model must simultaneously handle all categories, thus effectively improving classification accuracy and training efficiency.

[0159] 3) Simplified problem identification and model optimization: When classification errors occur, this method makes problem localization more intuitive. Because each sub-model is responsible for a different task, it is easy to determine which part has deviated. The specific model with the problem can be retrained without affecting other normally functioning parts, which is usually difficult to achieve when using a single multimodal model. Specifically, for vocabulary-based optimized text classification: a dedicated vocabulary, Vocab, is established during the text classification process. This allows for pre-selection of highly discriminative words before classification, improving the utilization of effective text information, reducing interference from irrelevant information, thereby improving classification accuracy and mitigating the impact of text extraction errors on classification accuracy in existing OCR technologies.

[0160] For multimodal fusion with optimized attention mechanisms: The attention module in bimodal classification has a unique design. It can automatically learn and assign attention weights based on the input image and text feature vectors, and better reflect the combined influence of the two vectors on the classification task through weighted fusion. This makes the fused vector more representative, further improving the classification performance and overcoming the problem that existing technologies struggle to effectively integrate information from different modalities.

[0161] Example 2

[0162] Corresponding to the aforementioned embodiment of a classification method for document images, this disclosure also provides an embodiment of a classification system for document images.

[0163] Figure 3 This is a schematic diagram of a module of a classification system for document images provided in Embodiment 2 of this disclosure, as shown below. Figure 3 As shown, the classification system includes: image classification module 21, text extraction module 22, text classification module 23, and multimodal classification module 24;

[0164] Image classification module 21 is used to classify the document images to be classified using an image classification model and obtain the dimension vector of the image classification model.

[0165] In this embodiment, the categories of document images include form-type and physical object-type;

[0166] The forms include ID cards, damage assessment reports, invoices, and vehicle registration certificates, while the physical items include accident scene photos, panoramic vehicle photos, and engine photos.

[0167] The text extraction module 22 is used to extract text from the document images to be classified using an OCR model, and to segment the extracted text to obtain a segmented set.

[0168] The text classification module 23 is used to classify the word segmentation set using a text classification model and obtain the dimensional vector of the text classification model.

[0169] In this embodiment, any text classification model can be used, but h neurons need to be set in the last hidden layer of the model.

[0170] The multimodal classification module 24 is used to input the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified.

[0171] In this embodiment, the dimensional vector `embedding_image` of the image classification model and the dimensional vector `embedding_nlp` of the text classification model are input into the multimodal model to obtain the image classification result of the document image to be classified. At this point, regardless of the confidence level, the classification process ends, and the image classification result of the document image to be classified is output.

[0172] In this embodiment, the image classification results include at least one of the following: ID card, damage assessment report, invoice, vehicle registration certificate, accident scene photos, vehicle panoramic photos, and engine photos.

[0173] This embodiment combines image classification models, OCR models, text classification models, and multimodal models to perform image classification, text extraction, text classification, and multimodal classification on the document images to be classified. This avoids the problems of large number of parameters, difficulty in real-time response, and difficulty in optimization caused by processing all categories simultaneously through a single model, thereby improving the accuracy and classification effect of document image classification.

[0174] In an optional embodiment, the image classification module includes:

[0175] The image classification unit is used to input the document image to be classified into the image classification model to obtain the image classification result of the document image to be classified.

[0176] The first acquisition unit is used to acquire the dimension vector of the image classification model in response to the image classification result not belonging to the object class or the image classification result belonging to the object class and the confidence of the image classification result being less than or equal to a preset confidence.

[0177] In practice, due to significant differences in image features between the object category and the form category, image classification models can distinguish between them and differentiate the various categories within the object category as accurately as possible. Furthermore, the different categories within the form category may be confused with each other.

[0178] A classification model is constructed using a CNN convolutional network (e.g., by adding a fully connected layer after ResNet50 for fine-tuning, where the last hidden layer in the fully connected layer contains h neurons, such as h=64), and the specific image classification results are output (e.g., damage assessment report, vehicle damage photos, invoices, etc.).

[0179] If the obtained image classification result belongs to the object class, and the confidence level of the image classification result is greater than the preset confidence level (e.g., the preset confidence level is 95%), then the classification process ends, and the image classification result is directly output. Otherwise (i.e., the image classification result does not belong to the object class, or the image classification result belongs to the object class and the confidence level of the image classification result is less than or equal to the preset confidence level), the h-dimensional vector embedding_image of the last hidden layer of the image classification model is retained, and subsequent text classification steps continue to be executed.

[0180] It should be noted that the preset reliability should be set according to the actual situation. For example, the preset reliability can be set to 95% or other values. There are no specific restrictions here.

[0181] In an optional embodiment, the text classification module includes:

[0182] The text classification unit is used to input the word segmentation set into the text classification model to obtain the text classification result;

[0183] The second acquisition unit is used to acquire the dimension vector of the text classification model in response to the confidence level of the text classification result being less than or equal to a preset confidence level.

[0184] In an optional embodiment, the text extraction module includes:

[0185] The text extraction unit is used to input the document images to be classified into the OCR model for text extraction and to obtain the target text with a text confidence score greater than the preset confidence score.

[0186] The word segmentation unit is used to segment the target text into words and filter out words that do not belong to the vocabulary to obtain a word segmentation set;

[0187] In this embodiment, after the document image to be classified is processed by the OCR model, the target text with a text confidence level greater than a preset confidence level (e.g., a preset confidence level of 95%) is retained and then segmented into words. The intersection of the resulting word segmentation set with the vocabulary list Vocabulary is used as the input word segmentation list for the document image to be classified.

[0188] In the specific implementation process, OCR models (such as open source models like PaddleOCR) are used to extract the text from the document images to be classified, and only the target text with a text confidence level greater than the preset confidence level (e.g., the preset confidence level is 95%) is retained.

[0189] Vocabulary filtering: The target text is segmented, and words that do not belong to the vocabulary Vocab (obtained during training) are removed, resulting in the final word set W.

[0190] Running the text classification model: Input the word segmentation set W into the text classification model to obtain the text classification result c. If the confidence of the text classification result c is greater than the preset confidence (e.g., the preset confidence is 95%), the classification ends, and the image classification result of the document image to be classified is output. Otherwise (i.e., the confidence of the text classification result is less than or equal to the preset confidence), the h-dimensional vector embedding_nlp of the last layer of the text classification model is retained, and subsequent multimodal classification steps are performed.

[0191] It should be noted that even if the text extraction from a form is incomplete, a multimodal model can still be used for the final judgment; the structure of a multimodal model is as follows: Figure 2 As shown.

[0192] This embodiment trains a text classification model to distinguish different categories of form-type images. By establishing a vocabulary and pre-screening the training corpus, the number of invalid word segments entering the model can be reduced, minimizing interference from personalized content entered by users and OCR model recognition errors. This also reduces the number of model parameters and improves computation speed.

[0193] In an optional embodiment, the multimodal model includes an attention module and a classification module; the multimodal classification module includes:

[0194] The third acquisition unit is used to input the dimensional vector of the image classification model and the dimensional vector of the text classification model into the attention module to obtain the attention weights;

[0195] In this embodiment, an attention module is constructed, consisting of an input layer, a hidden layer, and an output layer. The input layer receives two h-dimensional vectors (embedding_image and embedding_nlp) (i.e., the h-dimensional vectors of the image classification model and the text classification model), which are concatenated into a 2h-dimensional vector as input. The hidden layer can be a fully connected layer with a certain number of neurons (e.g., h neurons), using an activation function (such as ReLU) to introduce non-linearity. The output layer has h neurons, corresponding to h attention weights attn.

[0196] The fusion unit is used to fuse the dimensional vector of the image classification model and the dimensional vector of the text classification model based on attention weights to obtain a fused dimensional vector;

[0197] In this embodiment, for the i-th dimension of the vector embedding_image ( The dimensional value after fusion is shown in formula (1) in Example 1; it is hoped that the importance of vector embedding_image to vector embedding_nlp in this dimension can be reflected by attention weight.

[0198] For the i-th dimension of the vector embedding_image ( The fused dimension value is shown in formula (2) in Example 1:

[0199] Through this correspondence and weighted fusion process, the two input vectors can be fused according to the weights output by the attention module, so that the fused vector can better reflect the combined influence of the two vectors on the classification task.

[0200] In this embodiment, embedding_image represents the h-dimensional vector of the last hidden layer of the image classification model, and embedding_nlp represents the h-dimensional vector of the last layer of the text classification model. h represents the dimension of the vector, and attn represents the h-dimensional weight vector obtained by the first part of the attention module of the bimodal model. It is used to perform weighted fusion of the h-dimensional vector embedding_image of the image classification model and the h-dimensional vector embedding_nlp of the text classification model to obtain the fused h-dimensional vector.

[0201] The fourth acquisition unit is used to input the fused dimension vector into the classification module to obtain the image classification result of the document image to be classified.

[0202] In this embodiment, a classification module is constructed, which consists of a multilayer perceptron (MLP). The input to the MLP is a dimensional vector after weighted fusion by an attention module. The MLP may include multiple hidden layers; for example, the first hidden layer has 2h neurons, the second hidden layer has h neurons, and an activation function (e.g., ReLU) is used. The last hidden layer has the same number of neurons as the number of classification categories, and outputs the image classification result of the document image to be classified.

[0203] In an optional embodiment, the classification system further includes:

[0204] The first output module is used to output the image classification result of the document image to be classified in response to the image classification result belonging to the object class and the confidence of the image classification result being greater than the preset confidence.

[0205] In an optional embodiment, the classification system further includes:

[0206] The second output module is used to output the image classification result of the document image to be classified in response to the text classification result having a confidence level greater than a preset confidence level.

[0207] In an optional embodiment, the classification system further includes:

[0208] The first acquisition module is used to acquire target text with a text confidence score greater than a preset confidence score, and to perform word segmentation on the target text to obtain the word segmentation results;

[0209] The first filtering module is used to filter the word segmentation results to obtain the word frequency and category matrix;

[0210] The merging module is used to merge the word segmentation sets of the unclassified document images of each category to obtain the merged word segmentation set;

[0211] The removal module is used to remove words that account for less than a preset percentage in the merged word segmentation set, thus obtaining the word segmentation set after word removal;

[0212] The word segmentation and deduplication module is used to perform word segmentation and deduplication on the merged word segmentation set whose removal ratio is less than the preset ratio, and obtain the frequent word set.

[0213] The second filtering module is used to filter the word frequency and category matrix based on the frequent word set to remove word segments that are not in the frequent word set, and obtain new word frequencies and new category matrices.

[0214] The testing module is used to perform chi-square tests on the new word frequency and new category matrix to obtain the test results;

[0215] The second acquisition module is used to acquire the target word segmentation set in the test results where the assumed value is less than or equal to the preset assumed value;

[0216] The third acquisition module is used to obtain a vocabulary list based on the target word segmentation set.

[0217] In the specific implementation process, the following steps are used to select word segments with strong ability to distinguish various forms of images, forming the final vocabulary list.

[0218] i. After the form-type images are processed by the OCR model, the target text with a text confidence level greater than the preset confidence level (e.g., the preset confidence level is 95%) is retained and word segmentation is performed. The word segmentation results are used to remove stop words, single Chinese characters, English letters, and numbers, resulting in a word frequency and category matrix, as shown in Table 1 of Example 1, where word represents word segmentation and c represents image category;

[0219] ii. For each form-type image category c_i, merge the word segmentation sets of all images under it. For each merged word segmentation set, remove word segments with a percentage less than a preset percentage (e.g., the preset percentage is 1%). Then, merge these word segmentation sets and remove duplicates according to word segmentation to obtain a vocabulary list (frequent word version).

[0220] iii. Filter the word frequency and category matrix obtained in step one (i.e., step i) using the vocabulary (frequent word version), remove those word segments that are not in the vocabulary, and obtain a new word frequency and category matrix (frequent word version).

[0221] iv. Perform a chi-square test on the word frequency and category matrix (frequent word version), with the image category column as the target. Based on the test results, only retain the word segments with a p-value ≤ 0.01. This set of word segments constitutes the final vocabulary, Vocabulary.

[0222] In practical implementation, for example, in an insurance claims scenario, classification tests were conducted on 300,000 photos (pixel range: 100,000-25,000,000 pixels, with a 1:1 ratio of physical object photos to form photos) across 15 types of document images, using an L40 graphics card as an example. The classification accuracy rates were as follows: Image classification + OCR model + keyword filtering: 90.3% accuracy; Image classification + text classification model (without a vocabulary): 95.5% accuracy; Image classification + text model (with a vocabulary): 98.0% accuracy; Image classification + text classification model (with a vocabulary) + bimodal classification: 99.1% accuracy. The computation latency was as follows: the average latency for classifying a single image was 520ms, and the maximum latency was 1150ms, achieving the real-time response requirement of an average latency of less than 1 second.

[0223] This disclosure creatively combines image classification, text classification, and multimodal fusion classification, replacing the traditional single-model processing method, thereby achieving high-precision real-time classification. The approach of this disclosure brings the following three significant advantages: 1) Reduced average computation latency: It can handle approximately 49% and 46% of document images (e.g., in insurance claims scenarios, the ratio of physical object photos to form photos is 1:1) in the image classification and text classification stages, respectively, with only 5% of complex cases requiring the final multimodal classification step. This hierarchical processing mechanism greatly reduces the overall computation time. 2) Improved classification accuracy: Each sub-model specializes in recognizing a specific type of document image; for example, the image classification model focuses on physical object images, while the text classification model focuses on form images. This division of labor avoids the massive number of parameters and tuning difficulty that a single multimodal model must simultaneously handle all categories, thus effectively improving classification accuracy and training efficiency.

[0224] 3) Simplified problem identification and model optimization: When classification errors occur, this method makes problem localization more intuitive. Because each sub-model is responsible for a different task, it is easy to determine which part has deviated. The specific model with the problem can be retrained without affecting other normally functioning parts, which is usually difficult to achieve when using a single multimodal model. Specifically, for vocabulary-based optimized text classification: a dedicated vocabulary, Vocab, is established during the text classification process. This allows for pre-selection of highly discriminative words before classification, improving the utilization of effective text information, reducing interference from irrelevant information, thereby improving classification accuracy and mitigating the impact of text extraction errors on classification accuracy in existing OCR technologies.

[0225] For multimodal fusion with optimized attention mechanisms: The attention module in bimodal classification has a unique design. It can automatically learn and assign attention weights based on the input image and text feature vectors, and better reflect the combined influence of the two vectors on the classification task through weighted fusion. This makes the fused vector more representative, further improving the classification performance and overcoming the problem that existing technologies struggle to effectively integrate information from different modalities.

[0226] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0227] Example 3

[0228] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this disclosure. The electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor. When the processor executes the computer program, it implements the document image classification method described in any of the above embodiments. Figure 4 The electronic device 90 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0229] like Figure 4As shown, the electronic device 90 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including memory 92 and processor 91).

[0230] Bus 93 includes a data bus, an address bus, and a control bus.

[0231] The memory 92 may include volatile memory, such as random access memory (RAM) 921 and / or cache memory 922, and may further include read-only memory (ROM) 923.

[0232] The memory 92 may also include a program tool 925 (or utility) having a set (at least one) program module 924, such program module 924 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0233] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92, such as the document image classification method provided in any of the above embodiments.

[0234] Electronic device 90 can also communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 95. Furthermore, electronic device 90 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 96. Figure 4 As shown, network adapter 96 communicates with other modules of electronic device 90 via bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0235] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0236] Example 4

[0237] Embodiment 4 of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document image classification method provided in any of the above embodiments.

[0238] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0239] Example 5

[0240] Embodiment 5 of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the document image classification method described in any of the above claims.

[0241] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0242] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for classifying single-document images, characterized in that, The classification method includes: The image classification model is used to classify the document images to be classified, and the dimensional vector of the image classification model is obtained. The text of the document image to be classified is extracted using an OCR model, and the extracted text is segmented into words to obtain a word segmentation set. The word segmentation set is classified using a text classification model to obtain the dimensional vector of the text classification model; The image classification model's dimension vector and the text classification model's dimension vector are input into the multimodal model to obtain the image classification result of the document image to be classified.

2. The classification method for document images as described in claim 1, characterized in that, The steps of classifying the document image to be classified using an image classification model and obtaining the dimension vector of the image classification model include: The image of the document to be classified is input into the image classification model to obtain the image classification result of the document to be classified. In response to the image classification result not belonging to the object class or the image classification result belonging to the object class and the confidence level of the image classification result being less than or equal to a preset confidence level, the dimension vector of the image classification model is obtained.

3. The method for classifying document images as described in claim 2, characterized in that, The step of classifying the word segmentation set using a text classification model and obtaining the dimensional vector of the text classification model includes: Input the word segmentation set into the text classification model to obtain the text classification result; If the confidence level of the text classification result is less than or equal to the preset confidence level, obtain the dimension vector of the text classification model.

4. The method for classifying document images as described in claim 1, characterized in that, The steps of using an OCR model to extract text from the document image to be classified, and then segmenting the extracted text to obtain a segmented set include: The document images to be classified are input into the OCR model for text extraction, and the target text with a text confidence score greater than the preset confidence score is obtained. The target text is segmented into words, and words that do not belong to the vocabulary are filtered out to obtain a segmentation set; And / or, The multimodal model includes an attention module and a classification module; the step of inputting the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified includes: The attention weights are obtained by inputting the dimension vector of the image classification model and the dimension vector of the text classification model into the attention module. Based on the attention weights, the dimensional vectors of the image classification model and the text classification model are fused to obtain a fused dimensional vector; The fused dimension vector is input into the classification module to obtain the image classification result of the document image to be classified.

5. The method for classifying document images as described in claim 3, characterized in that, The classification method also includes: In response to the image classification result belonging to the object class and the confidence level of the image classification result being greater than the preset confidence level, the image classification result of the document image to be classified is output; And / or, The classification method also includes: If the confidence level of the text classification result is greater than the preset confidence level, the image classification result of the document image to be classified is output.

6. The method for classifying document images as described in claim 1, characterized in that, The classification method also includes: Obtain target text with a text confidence score greater than a preset confidence score, and perform word segmentation on the target text to obtain the word segmentation result; The word segmentation results are filtered to obtain a word frequency and category matrix; Merge the word segments of the unclassified document images for each category to obtain the merged word segment set; Remove the segments that account for less than a preset percentage from the merged segment set to obtain the segment set after removing the segments; The merged word segment set, after removing words whose proportion is less than the preset proportion, is segmented and deduplicated to obtain a frequent word set; The word frequency and category matrix are filtered based on the frequent word set to remove word segments that are not in the frequent word set, resulting in new word frequencies and new category matrices. A chi-square test was performed on the new word frequency and the new category matrix to obtain the test results; Obtain the target word segmentation set in the test results where the assumed value is less than or equal to the preset assumed value; A vocabulary is obtained based on the target word segmentation set.

7. A classification system for document images, characterized in that, The classification system includes: The image classification module is used to classify the document images to be classified using an image classification model and obtain the dimensional vector of the image classification model. The text extraction module is used to extract text from the document image to be classified using an OCR model, and to segment the extracted text into words to obtain a word segmentation set. The text classification module is used to classify the word segmentation set using a text classification model and obtain the dimensional vector of the text classification model. The multimodal classification module is used to input the dimension vector of the image classification model and the dimension vector of the text classification model into the multimodal model to obtain the image classification result of the document image to be classified.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the classification method for document images according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the classification method for document images according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the classification method for document images as described in any one of claims 1 to 6.

Images