A document image classification method and apparatus

Abstract

This application provides a document image classification method and apparatus. In this method, firstly, a text feature sequence and a text layout feature sequence are obtained based on the text in the document image and the spatial location index of the corresponding text. Secondly, an image feature sequence and an image layout feature sequence are obtained based on the images in the document image and the spatial location index of the corresponding images. Then, the text layout feature sequence is fused with the text feature sequence, and the image layout feature sequence is fused with the image feature sequence to obtain a global feature sequence. Finally, the document image is classified based on the global feature sequence. This method utilizes not only text and image information but also the layout information of the document image for classification, overcoming the problem in the prior art of difficulty in distinguishing document image categories with different layouts, improving the classification effect of document image categories, and increasing the accuracy of document image classification.

Description

Technical Field

[0001] This application relates to the field of deep learning, and in particular to a document image classification method and apparatus. Background Technology

[0002] In recent years, with the rapid development of science and technology, artificial intelligence has gradually become a decisive force driving humanity into the intelligent era. Document image classification is one of the important research goals of artificial intelligence. As an interdisciplinary field encompassing computer vision, pattern recognition, and machine learning, document image classification aims to extract discriminative features from document images or based on their semantic information to distinguish different categories of document images, minimizing classification errors and enabling machine vision to possess a certain level of recognition capability. With the emergence of big data and continuous breakthroughs in computing power, document image classification has achieved remarkable results in many fields, and is widely used in areas such as medical image processing, intelligent transportation, e-commerce platforms, and facial recognition.

[0003] In existing technologies, document image classification methods typically use CNN models to extract image feature vectors and BERT models to extract text feature vectors. These image and text feature vectors are then concatenated to obtain joint features, which are then used for classification. However, for some document images where image features are not obvious or text content is difficult to discern, it is insufficient to distinguish image categories based solely on image details and text content.

[0004] Therefore, how to improve the classification effect of such difficult-to-distinguish images and increase the accuracy of classification is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a document image classification method aimed at improving image classification performance and enhancing the accuracy of document image classification.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] Firstly, this application provides a document image classification method, the method comprising:

[0008] The text feature sequence is obtained from the text in the document image, and the spatial location index corresponding to the text is mapped to multiple feature vectors to obtain the text layout feature sequence.

[0009] The text layout feature sequence is combined with the text feature sequence to obtain the first feature sequence;

[0010] Based on the images in the document image, an image feature sequence is obtained, and the spatial location index corresponding to the image is mapped into multiple feature vectors to obtain an image layout feature sequence.

[0011] The image layout feature sequence is combined with the image feature sequence to obtain the second feature sequence;

[0012] The first feature sequence and the second feature sequence are processed to obtain a global feature sequence, and the document image is classified according to the global feature sequence.

[0013] Optionally, the process of processing the first feature sequence and the second feature sequence to obtain the global feature sequence includes:

[0014] Construct a standardized feature vector as a classification label feature vector;

[0015] The classification label feature vector, the first feature sequence, and the second feature sequence are concatenated to obtain a local feature sequence;

[0016] The local feature sequence is input into a neural network model for feature fusion to obtain a global feature sequence.

[0017] Optionally, classifying the document image based on the global feature sequence includes:

[0018] The classification label feature vector is selected from the global feature sequence, and the classification label feature vector is processed by the neural network model for feature fusion to fuse the relevant information required for classification.

[0019] The document image is classified based on the classification tag feature vector.

[0020] Optionally, the local feature sequence is input into a neural network model for feature fusion for processing to obtain a global feature sequence, including:

[0021] The local feature sequence is then input into a Transformer neural network model for processing.

[0022] The Transformer neural network model is used to fuse the information in the first feature sequence and the information in the second feature sequence.

[0023] The relevant information required for document image classification is fused into the classification label feature vector using the Transformer neural network model;

[0024] The global feature sequence is obtained and output using a Transformer neural network model.

[0025] Optionally, obtaining the text feature sequence based on the text in the document image includes:

[0026] Recognize text in document images;

[0027] The text is divided into multiple words to obtain a text sequence of a preset length;

[0028] The word segmentation of the text sequence is mapped into multiple feature vectors to obtain the text feature sequence;

[0029] The step of mapping the spatial location index corresponding to the text to multiple feature vectors to obtain a text layout feature sequence includes:

[0030] The spatial location index of the word segmentation is mapped to multiple feature vectors to obtain a text layout feature sequence.

[0031] Optionally, dividing the text into multiple words includes:

[0032] The text is truncated;

[0033] The corresponding truncated text is filled in to form multiple word segments.

[0034] Optionally, obtaining the image feature sequence based on the images in the document image includes:

[0035] The document image is segmented into multiple tiles;

[0036] The information of the image patch is mapped into multiple feature vectors to obtain an image feature sequence;

[0037] The step of mapping the spatial location index corresponding to the image to multiple feature vectors to obtain an image layout feature sequence includes:

[0038] The spatial location index corresponding to the tile is mapped to multiple feature vectors to obtain the image layout feature sequence.

[0039] Optionally, combining the text layout feature sequence with the text feature sequence includes:

[0040] The text layout feature sequence and the text feature sequence are combined by physical concatenation, wherein the feature vector dimension in the text layout feature sequence and the text feature sequence are equal;

[0041] The step of combining the image layout feature sequence with the image feature sequence includes:

[0042] The image layout feature sequence and the image feature sequence are combined by physical splicing, wherein the feature vectors in the image layout feature sequence and the image feature sequence have the same dimension.

[0043] Optionally, classifying the document image based on the global feature sequence includes:

[0044] The global feature sequence is input into the Softmax classifier;

[0045] The Softmax classifier classifies the document image based on the global feature sequence.

[0046] Secondly, this application provides a document image classification device, the device comprising:

[0047] The text information acquisition module is used to map the text in the document image and the spatial location index corresponding to the text into multiple feature vectors to obtain a text feature sequence and a text layout feature sequence, and to combine the text feature sequence and the text layout feature sequence to obtain a first feature sequence;

[0048] The image information acquisition module is used to map the images in the document image and the spatial location index corresponding to the images into multiple feature vectors to obtain an image feature sequence and an image layout feature sequence, and to combine the image feature sequence and the image layout feature sequence to obtain a second feature sequence;

[0049] The global feature sequence acquisition module is used to process the first feature sequence and the second feature sequence to obtain the global feature sequence;

[0050] The classification module is used to classify the document image based on the global feature sequence.

[0051] Compared to existing technologies, this application has the following advantages: This application obtains text feature sequences and text layout feature sequences based on the text in the document image and the corresponding spatial location index of the text; it also obtains image feature sequences and image layout feature sequences based on the images in the document image and the corresponding spatial location index of the images. Feature fusion is performed on the above text feature sequences, text layout feature sequences, image feature sequences, and image layout feature sequences to obtain a global feature sequence. The document image is then classified based on this global feature sequence. Therefore, this application not only extracts image and text information but also extracts the layout information corresponding to the images and text. Feature fusion is performed on the layout information, image information, and text information, overcoming the problem in existing technologies where it is difficult to distinguish different categories of document images with different layout information, thus improving the classification effect and accuracy of document images. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart of a document image classification method provided in this application embodiment;

[0054] Figure 2 Another flowchart of the document image classification method provided in the embodiments of this application;

[0055] Figure 3 This is a flowchart of a method for obtaining a global feature sequence and classifying document images based on the global feature sequence, as provided in an embodiment of this application.

[0056] Figure 4 This is a schematic diagram of a document image classification device provided in an embodiment of this application. Detailed Implementation

[0057] As mentioned earlier, existing technologies primarily employ multimodal methods based on images and text for document image classification. These methods concatenate image and text features extracted from the document image to obtain joint features, which are then used to classify the document image. However, for document images containing similar image details and text content, it is difficult to correctly distinguish their categories without considering the layout information within the document image. Therefore, this application provides a document image classification method to improve the classification performance of document images that are difficult to categorize, thereby increasing classification accuracy.

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0059] See Figure 1 , Figure 1 A flowchart of a document image classification method provided in this application embodiment, the method comprising:

[0060] S101. Obtain a text feature sequence based on the text in the document image, and map the spatial location index corresponding to the text into multiple feature vectors to obtain a text layout feature sequence.

[0061] Specifically, document images refer to the digitization of paper documents into computer systems, enabling secure storage, online sharing, and automatic classification of data within these documents.

[0062] The text feature sequence obtained from the text in the document image can be obtained by the following method: First, the text obtained from the document image is segmented and filled to obtain a text sequence, which includes multiple word segments; then, the corresponding word segments are mapped to multiple feature vectors of preset length to obtain a text feature sequence, which includes multiple feature vectors.

[0063] In this application, text can be obtained by using OCR technology to recognize document images. Of course, other methods can also be used to recognize text images, and this application does not limit this method.

[0064] The spatial location index corresponding to the text is mapped to multiple feature vectors to obtain a text layout feature sequence, which may include: mapping the spatial location index corresponding to the word segmentation in the text sequence to multiple feature vectors, wherein the text layout feature sequence includes multiple feature vectors.

[0065] S102. Combine the text layout feature sequence with the text feature sequence to obtain the first feature sequence.

[0066] The method for combining the text layout feature sequence with the text feature sequence may include physically concatenating the text layout feature sequence and the text feature sequence. Of course, using other combination methods, such as using a specific neural network model for feature fusion, will not affect the normal implementation of the embodiments of this application.

[0067] Specifically, the first feature sequence can refer to a set of feature vectors containing text information and text layout information.

[0068] S103. Obtain an image feature sequence based on the images in the document image, and map the spatial location index corresponding to the image to multiple feature vectors to obtain an image layout feature sequence.

[0069] The image feature sequence can be obtained from the images in the document image in the following way: First, the image of the document image is divided into multiple blocks by scaling and cropping; then, the content information of the blocks is mapped into multiple feature vectors of preset length to obtain the image feature sequence, which contains multiple feature vectors.

[0070] It should be noted that the content information of the blocks in this embodiment is not limited. It can be the pixel information of the blocks or other content information, which does not affect the implementation of this embodiment.

[0071] The image layout feature sequence can be obtained by mapping the spatial location index of the image to multiple feature vectors as follows: the spatial location index of each patch in the image is mapped to multiple feature vectors to obtain the image layout feature sequence, which contains multiple feature vectors.

[0072] S104. Combine the image layout feature sequence with the image feature sequence to obtain the second feature sequence.

[0073] The method of combining the image layout feature sequence with the image feature sequence is the same as the method of combining the text layout feature sequence with the text feature sequence described above, and will not be repeated here.

[0074] The second feature sequence can specifically refer to a set of feature vectors that contain image information and image layout information.

[0075] S105. Process the first feature sequence and the second feature sequence to obtain the global feature sequence.

[0076] There are various methods for processing the first feature sequence and the second feature sequence. In one optional implementation, a neural network model is used to fuse the features of the first feature sequence and the second feature sequence.

[0077] The neural network model can be a Transformer neural network model, or other neural network models that can be used for feature fusion, without affecting the normal implementation of the embodiments of this application.

[0078] S106. Classify the document image according to the global feature sequence.

[0079] The global feature sequence contains multiple feature vectors. Each feature vector not only contains its own local information but also incorporates global information related to the local information, such as classification label feature vectors that integrate relevant information from image information, text information, image layout information, and text layout information required for classification. Of course, the global feature sequence may also contain one or more feature vectors for other purposes, which will not be elaborated upon here.

[0080] It should be noted that the implementation order of steps S101 and S103 is not limited in this embodiment, nor is the specific order of steps S102 and S104 limited. See details below. Figure 2 ,Should Figure 2 This is another flowchart of the document image classification method provided in this application embodiment. That is, in this application embodiment, image feature sequences, text feature sequences, image layout feature sequences, and text layout feature sequences can be obtained simultaneously or sequentially. After obtaining the text feature sequences and text layout feature sequences, they can be combined to obtain a first feature sequence. After obtaining the image feature sequences and image layout feature sequences, they can be combined to obtain a second feature sequence. Alternatively, the first and second feature sequences can be obtained simultaneously or sequentially after obtaining the image feature sequences, text feature sequences, image layout feature sequences, and text layout feature sequences.

[0081] This application extracts image and text information from document images to be classified, as well as the layout information corresponding to the images and text. It then performs feature fusion on the layout information, image information, and text information, and classifies the document image based on a global feature sequence containing rich layout information. This overcomes the problem in the prior art that it is difficult to distinguish different categories of document images with different layout information, improves the classification effect of document images, and enhances classification accuracy.

[0082] In the embodiments of this application, the above Figure 1 There are multiple possible implementations of steps S105 and S106. One implementation is described in detail below. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.

[0083] Figure 3 This is a flowchart illustrating a method for obtaining a global feature sequence and classifying a document image based on the global feature sequence, as provided in an embodiment of this application. This embodiment primarily processes the first and second feature sequences by introducing a classification label feature vector specifically for classification, obtaining a global feature sequence, and then classifying the document image based on this global feature sequence. Figure 1 or Figure 2 Steps S105 and S106 are described in detail below:

[0084] S201. Construct a standardized feature vector as a classification label feature vector.

[0085] The classification label feature vector does not contain any information and is used to store the relevant information required for the classification of the document image during feature fusion in subsequent processes.

[0086] S202. The classification label feature vector, the first feature sequence, and the second feature sequence are concatenated to obtain a local feature sequence.

[0087] The method of concatenating the classification label feature vector, the first feature sequence, and the second feature sequence can be a physical concatenation, which is only used to standardize the format of subsequent inputs into the neural network model used for feature fusion.

[0088] Of course, the classification label feature vector, the first feature sequence, and the second feature sequence can also be directly input into the neural network model for feature fusion without being concatenated, which will not affect the normal implementation of the embodiments of this application.

[0089] Of course, the classification marker feature vector, the first feature sequence, and the second feature sequence can also be concatenated in other ways, which are also within the protection scope of this application embodiment.

[0090] S203. The local feature sequence is input into a neural network model for feature fusion for processing to obtain a global feature sequence.

[0091] The neural network model used for feature fusion can be a Transformer neural network model. This application does not limit the type of neural network model used for feature fusion, and all of them are within the protection scope of this application.

[0092] S204. Select the classification label feature vector from the global feature sequence. The classification label feature vector is processed by the neural network model used for feature fusion and then fused with the relevant information required for classification.

[0093] In addition to the classification label feature vector, the global feature sequence also contains multiple feature vectors. Each feature vector incorporates global messages related to the message it carries, such as global image information or text information.

[0094] S205. Classify the document image according to the classification marker feature vector.

[0095] This application embodiment introduces a classification marker feature vector specifically for classification. This vector can integrate the relevant information required for classification from the image information, text information, and layout information of the document image to be classified into a specific feature vector. This can save classification time in subsequent processes, simplify the classification process, and enhance the classification effect of document images by classifying them according to the relevant layout information, thereby improving the accuracy of judging the category of document images.

[0096] The above are some specific implementations of the document image classification method provided in the embodiments of this application. Based on this, this application also provides a corresponding apparatus. The apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.

[0097] See Figure 4 As shown, Figure 4 This is a schematic diagram of a document image classification device provided in an embodiment of this application. The device includes: a text information acquisition module 301, an image information acquisition module 302, a global feature sequence acquisition module 303, and a classification module 304.

[0098] The text information acquisition module 301 is used to map the text in the document image and the spatial location index corresponding to the text into multiple feature vectors to obtain a text feature sequence and a text layout feature sequence, and to combine the text feature sequence and the text layout feature sequence to obtain a first feature sequence.

[0099] The text information acquisition module 301 can also recognize the document image and divide the recognized text into multiple words by truncation or padding. It maps the information of the word segmentation and the spatial position index of the word segmentation in the text into multiple feature vectors to obtain the text feature sequence and the text layout feature sequence.

[0100] The text information acquisition module 301 can also physically concatenate the text feature sequence and the text layout feature sequence after acquiring them to obtain the first feature sequence.

[0101] The image information acquisition module 302 is used to map the images in the document image and the spatial location index corresponding to the images into multiple feature vectors to obtain an image feature sequence and an image layout feature sequence, and to combine the image feature sequence and the image layout feature sequence to obtain a second feature sequence.

[0102] The image information acquisition module 302 can also divide the image in the document image into multiple tiles by scaling and cropping, and map the information of the tiles and the spatial position index of the tiles in the image into multiple feature vectors to obtain the image feature sequence and the image layout feature sequence.

[0103] The image information acquisition module 302 can also physically splice the image feature sequence and the image layout feature sequence after acquiring them to obtain a second feature sequence.

[0104] The global feature sequence acquisition module 303 is used to process the first feature sequence and the second feature sequence to obtain the global feature sequence.

[0105] The global feature sequence acquisition module 303 can obtain the global feature sequence by constructing a standardized feature vector as a classification label feature vector, and processing the classification label feature vector, the first feature sequence, and the second feature sequence using a neural network model for feature fusion.

[0106] The neural network model used for feature fusion by the global feature sequence acquisition module 303 may include a Transformer neural network model.

[0107] The neural network model used by the global feature sequence acquisition module 303 for feature fusion can fuse the relevant information required for classification into the classification label feature vector through feature fusion.

[0108] The classification module 304 is used to classify the document image based on the global feature sequence.

[0109] The classification module 304 can classify the document image based on the classification marker feature vector in the global feature sequence.

[0110] The classification module 304 can obtain the classification result of the document image by inputting the global feature sequence into a general classifier, such as a Softmax classifier.

[0111] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0113] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.

Claims

1. A document image classification method, characterized in that, The method includes: The text feature sequence is obtained from the text in the document image, and the spatial location index corresponding to the text is mapped to multiple feature vectors to obtain the text layout feature sequence. The text layout feature sequence is combined with the text feature sequence to obtain the first feature sequence; Based on the images in the document image, an image feature sequence is obtained, and the spatial location index corresponding to the image is mapped into multiple feature vectors to obtain an image layout feature sequence. The image layout feature sequence is combined with the image feature sequence to obtain the second feature sequence; The first feature sequence and the second feature sequence are processed to obtain the global feature sequence; The document image is classified based on the global feature sequence; The step of processing the first feature sequence and the second feature sequence to obtain the global feature sequence includes: Construct a standardized feature vector as a classification label feature vector; The classification label feature vector, the first feature sequence, and the second feature sequence are concatenated to obtain a local feature sequence; The local feature sequence is input into a neural network model for feature fusion to obtain a global feature sequence; The step of classifying the document image based on the global feature sequence includes: The classification label feature vector is selected from the global feature sequence, and the classification label feature vector is processed by the neural network model for feature fusion to fuse the relevant information required for classification. The document image is classified based on the classification tag feature vector.

2. The method according to claim 1, characterized in that, The local feature sequence is input into a neural network model for feature fusion to obtain a global feature sequence, including: The local feature sequence is then input into a Transformer neural network model for processing. The Transformer neural network model is used to fuse the information in the first feature sequence and the information in the second feature sequence. The relevant information required for document image classification is fused into the classification label feature vector using the Transformer neural network model; The global feature sequence is obtained and output using a Transformer neural network model.

3. The method according to claim 1, characterized in that, The step of obtaining the text feature sequence based on the text in the document image includes: Recognize text in document images; The text is divided into multiple words to obtain a text sequence of a preset length; The word segmentation of the text sequence is mapped into multiple feature vectors to obtain the text feature sequence; The step of mapping the spatial location index corresponding to the text to multiple feature vectors to obtain a text layout feature sequence includes: The spatial location index of the word segmentation is mapped to multiple feature vectors to obtain a text layout feature sequence.

4. The method according to claim 3, characterized in that, The process of dividing the text into multiple words includes: The text is truncated; The corresponding truncated text is filled in to form multiple word segments.

5. The method according to claim 1, characterized in that, The step of obtaining the image feature sequence based on the images in the document image includes: The document image is segmented into multiple tiles; The information of the image patch is mapped into multiple feature vectors to obtain an image feature sequence; The step of mapping the spatial location index corresponding to the image to multiple feature vectors to obtain an image layout feature sequence includes: The spatial location index corresponding to the tile is mapped to multiple feature vectors to obtain the image layout feature sequence.

6. The method according to claim 1, characterized in that, The step of combining the text layout feature sequence with the text feature sequence includes: The text layout feature sequence and the text feature sequence are combined by physical concatenation, wherein the feature vector dimension in the text layout feature sequence and the text feature sequence are equal; The step of combining the image layout feature sequence with the image feature sequence includes: The image layout feature sequence and the image feature sequence are combined by physical splicing, wherein the feature vectors in the image layout feature sequence and the image feature sequence have the same dimension.

7. The method according to any one of claims 1-6, characterized in that, The step of classifying the document image based on the global feature sequence includes: The global feature sequence is input into the Softmax classifier; The Softmax classifier classifies the document image based on the global feature sequence.

8. A document image classification device, characterized in that, The device includes: The text information acquisition module is used to map the text in the document image and the spatial location index corresponding to the text into multiple feature vectors, obtain a text feature sequence and a text layout feature sequence, and combine the text feature sequence and the text layout feature sequence to obtain a first feature sequence. The image information acquisition module is used to map images in the document image and their corresponding spatial location indices into multiple feature vectors, obtain an image feature sequence and an image layout feature sequence, and combine the image feature sequence and the image layout feature sequence to obtain a second feature sequence. The global feature sequence acquisition module is used to process the first feature sequence and the second feature sequence to obtain the global feature sequence; The classification module is used to classify the document image based on the global feature sequence; The global feature sequence acquisition module is further configured to: Construct a standardized feature vector as a classification label feature vector; The classification label feature vector, the first feature sequence, and the second feature sequence are concatenated to obtain a local feature sequence; The local feature sequence is input into a neural network model for feature fusion to obtain a global feature sequence; The classification module is also used for: The classification label feature vector is selected from the global feature sequence, and the classification label feature vector is processed by the neural network model for feature fusion to fuse the relevant information required for classification. The document image is classified based on the classification tag feature vector.

Images