Multi-modal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling

Abstract

The invention relates to the field of computer vision and artificial intelligence, in particular to a multi-modal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling. According to the method, visual and text information of a document is co-processed through a double-flow architecture; a visual semantic feature of a document image is efficiently extracted from a visual flow by adopting a lightweight convolution operation and multi-scale feature fusion mechanism; an original text is obtained from a text stream through an optical character recognition technology, text error correction and word embedding extraction are carried out through a fine tuning language model, word-level and sentence-level context modeling is realized by inputting a hierarchical semantic coding network, and document-level features are generated. And high-precision document image classification is realized by fusing the double-flow feature vectors and combining a full-connection layer. Experimental results show that the method has an excellent multi-modal feature fusion effect, relatively high classification precision and strong robustness to complex document scenes in a document image classification task, and is suitable for application scenes of various document image classification.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a multimodal fusion document image classification algorithm that combines multi-branch deep convolution and semantic segmentation modeling. Background Technology

[0002] Document image classification, as a key technology in intelligent document processing, is widely used in fields such as archive digitization, intelligent office work, and document management systems. Current mainstream methods are mainly divided into two categories: unimodal classification and multimodal classification. Unimodal classification methods rely solely on visual features for analysis. When faced with documents that have highly similar visual layouts but different semantic content (such as different types of forms, letters, or invoices), they struggle to effectively distinguish their essential differences, leading to a significant decrease in classification accuracy. While multimodal classification methods improve performance by fusing visual and textual information, they suffer from significant bottlenecks: text stream processing relies on OCR technology to extract text, and low-quality document images often lead to text recognition errors, affecting subsequent error correction and embedding quality; when using language models such as BERT for text processing, the computational complexity of long document sequences increases exponentially, resulting in low inference efficiency; simultaneously, traditional hierarchical attention networks (HANs) use bidirectional GRUs as encoders, which are prone to gradient vanishing problems when processing long texts, weakening the deep modeling capabilities at the word and sentence levels, thus reducing classification accuracy. These limitations make it difficult for existing methods to balance computational efficiency and classification robustness in real-world complex document scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal document image classification method based on multi-branch deep convolution and hierarchical semantic modeling. This method aims to solve the problems of low efficiency in processing long texts, sensitivity to OCR noise, and insufficient visual-text modality fusion in existing technologies. This invention significantly improves the semantic understanding capability and computational efficiency of long documents by introducing an efficient multi-scale visual feature extraction mechanism and a hierarchical semantic encoder based on a state-space model. Simultaneously, through a deep cross-modal fusion strategy, it effectively enhances the model's classification accuracy and robustness for complex document images, demonstrating good practicality and application value.

[0004] To achieve the above objectives, this invention provides a multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling, specifically including the following steps:

[0005] Step 1: Document Image Input and Preprocessing. Obtain the document images to be classified. Images can be sourced from scanners, cameras, or existing digital archives. First, standardize the image size, scaling input images of different sizes to a fixed resolution (e.g., 384×384 pixels), and then convert them to grayscale or RGB to suit the subsequent model input requirements. To enhance the model's generalization ability and alleviate data scarcity, various data augmentation strategies are employed to process the training set, including random rotation, horizontal flipping, and brightness and contrast adjustments, to improve the model's robustness to document images of different qualities, layouts, and styles.

[0006] Step 2: Visual Feature Extraction. The preprocessed document image is input into the visual flow network, which is based on a multi-branch deep convolutional network (i.e., the InceptionNeXt architecture). It extracts multi-scale visual features from different receptive fields through parallel multi-branch deep separable convolutions. The network further integrates an efficient multi-scale attention (EMA) mechanism. Through cascaded operations of channel and spatial attention modules, it dynamically calculates the importance weights of feature maps at each scale, enhancing key visual features and suppressing background noise. The final output is a document visual feature vector rich in multi-scale contextual information.

[0007] Step 3: Text Information Extraction and Semantic Encoding. The Tesseract-4 OCR engine is used to perform optical character recognition on the document image, extracting the original text sequence. This sequence is then input into a BERT-Base model fine-tuned on a domain-specific text corpus. Its Masked Language Model (MLM) and self-attention mechanism are used to perform context-aware semantic error correction and ambiguity resolution for OCR recognition errors, generating dynamic word embedding vectors rich in contextual semantic information. This provides a high-precision text representation for subsequent hierarchical semantic modeling.

[0008] Step 4: Hierarchical semantic feature modeling based on Manba. A hierarchical attention network (HAN) improved from the text sequence after error correction and embedding is used for deep semantic modeling. The core innovation of this network lies in using the Manba (Mamba) state-space model instead of the traditional bidirectional GRU as the sequence encoder to efficiently capture long-range dependencies between words and sentences. First, at the word level, the network encodes the word embedding sequence in the sentence into the Manba encoder and generates sentence vector representations through a word-level attention network. Then, at the sentence level, all sentence vectors are again input into the Manba encoder to model the global inter-sentence logical relationships in the document. These are then weighted and aggregated through a sentence-level attention network, ultimately outputting a document-level text feature vector representing the high-level semantic structure of the entire document.

[0009] Step 5: Multimodal feature fusion. The visual feature vector extracted in Step 2 is fused with the document-level text feature vector generated in Step 4. An early fusion strategy based on concatenation is adopted, concatenating the feature vectors of the two modalities along the feature dimension to form a unified, complementary, and comprehensive multimodal feature representation vector, providing a joint representation of both visual layout and semantic content for the final classification decision.

[0010] Step 6: Document classification. The fused multimodal feature vector is input into a fully connected layer classifier. The Softmax function is used to calculate the probability distribution of the document belonging to each predefined category. The category with the highest probability is taken as the final classification result of the document image, thereby realizing end-to-end automatic document image classification.

[0011] Step 7: Model Training and Optimization. An end-to-end supervised learning strategy is adopted, combining the cross-entropy loss function and the Adam optimizer for model training. First, pre-training is performed on large-scale document datasets such as ImageNet and RVL-CDIP to initialize model parameters and learn general feature representations. Subsequently, fine-tuning is performed on the target dataset Tobacco-3482, employing a dynamic learning rate decay strategy (e.g., dynamically adjusting the learning rate based on validation set performance metrics) to ensure training stability and efficiency, ultimately improving the model's classification accuracy and generalization ability in the target scene. Compared with the prior art, the present invention has significant advantages and technical effects:

[0012] By introducing a multi-scale attention mechanism, the model can dynamically focus on key visual features at different scales, effectively enhancing its ability to perceive document layout details. Employing a hierarchical encoder based on the Manba state-space model significantly improves the model's efficiency in processing long document sequences, overcoming the gradient vanishing and computational bottleneck problems of traditional recurrent neural networks in long sequence modeling. Combining a lightweight network design with a multimodal feature fusion strategy, the model significantly reduces computational complexity while maintaining classification accuracy. This enables the model to demonstrate excellent robustness and practicality when facing challenges such as visually similar document classification and OCR noise interference, providing an efficient and reliable solution for document image classification. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings are briefly described below. It should be noted that the following drawings are only used to illustrate some embodiments of the present invention. Those skilled in the art can still design other types of drawings based on these drawings without innovative effort. The contents of the drawings are as follows:

[0014] Figure 1This document illustrates a flowchart of a multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling, as described in an embodiment of the present invention.

[0015] Figure 2 This illustration shows a schematic diagram of the operation of a multi-branch deep convolutional network for extracting visual features from document images in an embodiment of the present invention.

[0016] Figure 3 This illustration shows a schematic diagram of the operation of the hierarchical attention network of the Manba encoder for extracting text features from document images in an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention. Example:

[0018] Reference Figure 1 A document image classification method based on multi-branch depthwise convolution and cross-attention mechanism includes the following steps:

[0019] (1) Acquire the document image to be processed from an image acquisition device (such as a scanner, digital camera, etc.);

[0020] (2) Image visual feature extraction, further referring to Figure 2 This paper employs a multi-branch deep convolutional and multi-scale attention mechanism to extract visual features from document images. The visual flow includes a multi-scale feature extraction branch based on InceptionNeXt and a cross-spatial attention fusion module. First, the input document image is decomposed into four parallel branches by the InceptionNeXt module: a 1×11 convolutional branch, an 11×1 convolutional branch, a 3×3 convolutional branch, and an identity mapping branch. The 1×11 convolutional branch specifically extracts horizontal features to capture text lines and table lines, the 11×1 convolutional branch extracts vertical features to capture text columns and table vertical lines, the 3×3 convolutional branch extracts local detail features such as character strokes and texture information, and the identity mapping branch preserves the original features to ensure gradient stability. Then, the cross-spatial attention fusion module enables efficient interaction of features from multiple branches. Specifically, the output features of the 1×11 and 11×1 branches are spatially compressed to generate directional global context vectors; simultaneously, the output features of the 3×3 branch are reshaped into a spatial feature matrix. Cross-dimensional interaction is achieved through matrix multiplication: horizontal global information is used to weight vertical spatial features, and vertical global information is used to weight horizontal spatial features. The two interaction results are summed and then used to generate joint spatial attention weights through a sigmoid activation function. The calculation formula is as follows:

[0021] in, This represents the channel attention weights based on the features of the left branch. Spatial expansion representing the right-branch features This represents the channel attention weights based on right-branch features. This represents the spatial unfolding of the left branch features. The process employs an efficient multi-scale attention mechanism (EMA) to dynamically adjust the feature weights at different spatial locations, achieving adaptive fusion of multi-scale features. Finally, the attention-weighted branch features are concatenated with the identity mapping features to form a document visual feature representation rich in directional context and local details, significantly enhancing the model's ability to recognize document layout structure and content features.

[0022] (3) Image-text stream feature extraction, further refer to Figure 3 This paper employs a multi-stage text information processing workflow for document semantic feature extraction. The text stream comprises three core modules: optical character recognition (OCR), semantic error correction coding, and hierarchical semantic modeling. First, the original text content is extracted from the input document image using Optical Character Recognition (OCR) technology, transforming visual information into a processable text sequence. Then, the BERT language model, fine-tuned on a specific semantic database, performs semantic error correction and context encoding on the recognition results. Through masked language modeling and next-sentence prediction tasks, potential OCR errors are corrected, and a text embedding representation with rich semantic information is generated. Building upon this, a hierarchical semantic modeling module based on the Mamba architecture is introduced. This module uses a state-space model to efficiently model long text sequences. The core computational process of Mamba can be described by the following state-space equation:

[0023]

[0024] in, This represents the hidden state of the system at time t. Given the input sequence, The output sequence is represented by A, B, C, and D, which are learnable parameter matrices. This equation models sequence dependencies using a linear time-invariant system, offering higher computational efficiency compared to traditional attention mechanisms. Specifically, the Mamba structure dynamically adjusts the attention weights of words at different positions through a selective state-space mechanism, effectively capturing long-distance dependencies in the document. Simultaneously, a hierarchical processing strategy is employed, performing semantic abstraction at the lexical, sentence, and paragraph levels to progressively construct a document semantic representation from local to global perspectives. Finally, the multi-level semantic features are fused to generate text feature vectors rich in contextual information, providing a high-quality semantic representation foundation for subsequent document classification tasks.

[0025] (4) The visual feature vector extracted in step 2 is fused with the document-level text feature vector generated in step 3. An early fusion strategy based on concatenation is adopted to concatenate the feature vectors of the two modalities in the feature dimension to form a unified, complementary comprehensive multimodal feature representation vector. The fused multimodal feature vector is then input into a fully connected layer classifier. The probability distribution of the document belonging to each predefined category is calculated by the Softmax function. The category with the highest probability is taken as the final classification result of the document image, thereby realizing end-to-end automatic classification of document images.

[0026] (5) Model training: First, the model was initialized using pre-trained weights from the ImageNet dataset. To enhance the network's adaptability to various document layouts and content, it was first pre-trained on the large-scale document image dataset RVL-CDIP. Based on this, the model was fine-tuned on the Tobacco3482 document training set in the target domain and evaluated on the corresponding test set to verify the model's classification accuracy and inference efficiency in the document classification task. The model underwent a total of 500,000 iterations. During training, the AdamW optimizer was used to optimize the parameters. The initial learning rate was set to 0.0001, and after 400,000 iterations, it was dynamically adjusted with a decay coefficient of 0.95 to ensure the stability and efficiency of the training. To enhance the model's robustness to changes in document images, various data augmentation techniques were applied during training, including random rotation (±10°), color jitter, random cropping, and perspective transformation, to simulate changes in document image acquisition in real-world scenarios. The batch size was set to 32, mixed-precision training was used to accelerate the computation process, and distributed training was performed on four NVIDIA V100 GPUs. (6) Real-time inference and output: To verify the performance of the algorithm in document image classification, it was evaluated on the validation set of the Tobacco-3482 dataset. Experimental results show that the model trained on the ImageNet and RVL-CDIP datasets achieves a classification accuracy of 94.4% using only the image modality and 95.9% using multimodal (image + text) classification. When trained only on the ImageNet dataset (without RVL-CDIP), the classification accuracy reaches 86.3% using only the image modality and 90.5% using multimodal classification. Simultaneously, the inference speed reaches 14.5 frames per second, fully verifying the high accuracy and real-time performance of the present invention in document image classification. The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling, characterized in that, The process includes the following steps: Step 1: Document image preprocessing, obtaining the input document image, and performing size standardization and grayscale processing to ensure input uniformity; Step 2: Visual feature extraction, inputting the preprocessed image into a multi-branch deep convolution model (integrating depthwise separable convolution and multi-scale attention mechanism), and outputting a high-order visual feature vector; Step 3: Text semantic error correction and encoding. Text content from document images is extracted using Tesseract-4 OCR, and the BERT model is fine-tuned for text error correction and word embedding generation. Step 4: Hierarchical semantic feature modeling. Word embeddings are input into an improved hierarchical attention network (HAN), where the word-level and sentence-level encoders use Manba encoders for semantic encoding, outputting document-level text features. Step 5: Multimodal feature fusion. Visual feature vectors are concatenated and fused with document-level text features to generate a comprehensive feature representation. Step 6: Document classification. The fused features are input into a fully connected layer, and the document category probability distribution is output through the Softmax function, achieving accurate classification of document images. Step 7: Model training. An end-to-end optimization strategy is adopted, with parameter fine-tuning performed on document datasets such as RVL-CDIP, and the cross-entropy loss function is combined to improve classification performance and generalization ability.

2. The multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling according to claim 1, characterized in that, In step (2), the visual feature extraction network adopts a structure combining a multi-branch deep convolutional network and a multi-scale attention mechanism. The multi-branch deep convolutional network (MDCN) extracts multi-scale visual features under different receptive fields by setting multiple depth-separable convolutional branches with different kernel sizes in parallel. The multi-scale attention mechanism (Efficient Multi-Scale Attention, EMA) dynamically calculates the importance weights of feature maps at each scale through the cascaded operation of the channel attention module and the spatial attention module, thereby enhancing key visual features and suppressing background noise. Finally, it generates a document visual feature vector rich in multi-scale contextual information through weighted fusion.

3. The multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling according to claim 1, characterized in that, In step (3), the text semantic error correction and encoding are as follows: First, the original text sequence is identified and extracted from the document image through the Tesseract-4 OCR engine; then, the sequence is input into the BERT-Base model that has been fine-tuned on the domain text corpus, and its masked language model and self-attention mechanism are used to perform context-aware semantic error correction and ambiguity elimination for OCR recognition errors, and dynamic word embedding vectors containing rich contextual semantic information are generated simultaneously.

4. The multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling according to claim 1, characterized in that, In step (4), the hierarchical semantic feature modeling is specifically implemented through an improved hierarchical attention network (HAN). Its innovation lies in using the Manba state space model instead of the traditional recurrent neural network (RNN) as the core sequence encoder. The network first inputs the word embedding sequence in the sentence into the Manba encoder at the word level to model long-distance inter-word dependencies, and generates sentence vector representations through the word-level attention network. Subsequently, at the sentence level, all sentence vectors are fed back into the Manba encoder to capture the global inter-sentence logical relationships in the document. They are then weighted and aggregated via a sentence-level attention network to finally output a document-level text feature vector that represents the high-level semantic structure of the entire document.

5. The multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling according to claim 1, characterized in that, In step (5), the multimodal feature fusion adopts an early fusion strategy based on concatenation, which concatenates the high-order visual feature vector extracted by the visual flow network with the document-level text feature vector output by the text flow network in the feature dimension to form a unified and complementary comprehensive feature representation vector, providing a decision basis for the final classification that combines visual layout and semantic content.

6. The multimodal fusion document image classification method based on multi-branch deep convolution and hierarchical semantic modeling according to claim 1, characterized in that, In step (7), an end-to-end optimization training strategy is adopted to pre-train on ImageNet and RVL-CDIP large-scale document datasets, and fine-tune and finally evaluate the performance on the Tobacco-3482 dataset. During training, the classification performance was optimized by combining the cross-entropy loss function. The model iterated a total of 500,000 times, using the Adam optimizer. The initial learning rate was set to 0.0001, and after 400,000 iterations, it was dynamically adjusted with a decay coefficient of 0.95 to ensure the stability and efficiency of training.

Images