Text image understanding method based on multi-modal large model optimization

Abstract

The invention discloses a text image understanding method based on multi-modal large model optimization, and the method comprises the steps: improving the recognition and reasoning capability of a low-quality text image through designing a data set, an evaluation criterion, a lightweight feature recovery module and a two-stage training strategy; the constructed data set comprises clear-low-quality text image paired samples; the lightweight feature recovery module comprises an input mapping layer, a stack Transform recovery module, an output mapping layer, an addition normalization layer and a fusion layer; the two-stage training strategy comprises a visual alignment stage and a semantic alignment stage. By adopting the technical scheme provided by the invention, the recognition accuracy and reasoning stability of the multi-modal large model under the low-quality text image condition can be remarkably improved, and meanwhile, the performance in a clear image scene is kept not to be reduced.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal artificial intelligence and document image processing technology, and specifically relates to a low-quality text image understanding method based on a multimodal large model, which can improve the robustness of the text image understanding system in low-quality scenarios. Background Art

[0002] As an important carrier of knowledge and information, text images play a vital role in a variety of fields, including digital office, smart education, financial technology, healthcare, and public services. Parsing and understanding text images not only improves the efficiency of information retrieval, knowledge extraction, and automated document processing, but also has important application value in supporting decision-making and intelligent interaction. With the rapid development of deep learning technology, especially the rise of multimodal large language models (MLLMs), the task of understanding text images has gradually expanded from traditional optical character recognition (OCR) to more complex visual question answering (VQA), document understanding, and information reasoning.

[0003] In recent years, large multimodal models have been pre-trained on large amounts of data for text image understanding tasks, significantly improving their understanding capabilities in clear text image scenarios. Representative achievements include the InternVL series, Qwen-VL, Flamingo, and BLIP. These models utilize visual encoders, text encoders, cross-modal fusion modules, and large language model architectures to effectively achieve inference generation from images to natural language. These models have achieved high levels of text extraction and understanding capabilities in clear image conditions. However, in real-world applications, text images often suffer from varying degrees of quality degradation. For example, in document scanning, photo acquisition, and remote transmission, text images are susceptible to noise, blurring, low-resolution compression, underexposure, and deformation folding. These degradation factors not only undermine the clarity and structural coherence of text but also significantly interfere with the visual feature extraction process, resulting in a significant decrease in the recognition and understanding accuracy of large multimodal models in low-quality image conditions.

[0004] To address the performance degradation problem under low-quality images, existing research mainly focuses on two directions:

[0005] On the one hand, image enhancement methods aim to pre-process input images through techniques such as super-resolution restoration, deblurring, denoising, and exposure correction, thereby improving the performance of subsequent understanding tasks. Typical methods include denoising networks, deblurring networks, and image enhancement models designed for various degradation types, or using visual prompting to guide the restoration process with a small amount of prompt information. Recently, methods such as DocRes proposed a unified restoration model based on visual prompts (DTSPrompt), which can handle various degradation types. However, such methods usually use pixel-level reconstruction as the optimization objective, which has huge training overhead and is prone to overfitting. This makes joint optimization with downstream tasks such as text image understanding difficult.

[0006] On the other hand, robustness enhancement methods attempt to improve the adaptability of multimodal models to degraded input text image data through methods such as text image data augmentation, loss design, contrastive learning, and self-supervised learning. Related work includes MaskOCR, which enhances robustness through masking mechanisms, MaskSTR, which improves interference resistance through training by simulating low-quality scenes, and DoCo, which uses contrastive learning to strengthen text region feature modeling. While these methods can directly operate in the feature space, they typically require redesigning the pre-training task or significantly modifying the model structure, making them difficult to be effectively compatible with existing large-scale pre-trained models.

[0007] Furthermore, existing low-quality image evaluation benchmarks mostly target natural images (such as blurred natural scene images and low-resolution object images). There is a lack of systematic benchmarks specifically addressing the quality degradation of text images, making it difficult to fully reflect the unique recognition challenges of text images. For example, traditional degradation types focus on general blur and compression distortion, ignoring complex issues unique to text images, such as structural deformation, occlusion, and noise aliasing.

[0008] In summary, the following outstanding problems still exist in the field of low-quality text image understanding technology:

[0009] 1) There is a lack of systematic evaluation datasets and standardized metrics specifically built for low-quality text images;

[0010] 2) Image restoration methods have high computational overhead and are difficult to coordinate with downstream tasks.

[0011] 3) Model robustness enhancement methods are difficult to be directly compatible with existing multimodal large model architectures, and their practical application and promotion are limited;

[0012] 4) Existing methods still have obvious deficiencies in text understanding capabilities under real complex degradation conditions, and their optimization of text image characteristics is limited.

[0013] Therefore, there is an urgent need for a new optimization method that can effectively improve the model's understanding and reasoning capabilities under low-quality text image conditions without significantly increasing the computational burden or significantly changing the existing multimodal large model structure. Summary of the Invention

[0014] In response to the problems in the existing technology of insufficient robustness of multimodal large models in low-quality text image understanding scenarios, difficulty in balancing image restoration and understanding tasks, and high computational overhead and poor compatibility of existing optimization methods, the present invention proposes a low-quality text image understanding method based on multimodal large model optimization, and designs a multimodal large model optimization method that combines feature recovery and a two-stage training strategy to improve the robustness of low-quality text image understanding. Through specially designed data sets and evaluation benchmarks, a lightweight feature recovery module (DocRobust-Module, DRM) and a two-stage training strategy, it can significantly enhance its recognition and reasoning capabilities under low-quality text image conditions without changing the original multimodal large model structure and parameters as much as possible, and systematically improve the adaptability and performance of the multimodal large model in low-quality scenarios.

[0015] The technical solutions of the present invention are as follows:

[0016] A text image understanding method based on multimodal large model optimization includes the following key parts: construction of a low-quality text image visual question answering dataset, design of a feature recovery module DocRobust-Module (DRM), a two-stage training strategy, construction of a low-quality text image evaluation benchmark, and model performance evaluation.

[0017] 1) Construction of low-quality text image visual question answering dataset:

[0018] To systematically evaluate the performance of large multimodal models in low-quality text image scenarios, we constructed a large-scale dataset of paired clear and low-quality images, DocRobust-VQA, based on a variety of text image sources (such as document images, charts, and scene text). The construction process of DocRobust-VQA can be divided into two steps: collecting and screening clear text image data, and generating low-quality text images.

[0019] ① Collection and screening of clear text image data

[0020] The present invention first selects high-quality, clear image samples from a public text-image visual question-answering dataset. The selection criteria are clear images and no significant occlusion or distortion of text, ensuring a uniform benchmark for the subsequent low-quality image control group. The resulting clear images cover a variety of formats, including office documents, charts, natural scene text, and single-line text clippings, demonstrating good domain coverage and scale diversity. Each image is annotated with a corresponding question and answer, forming a set of question-answer pairs that provide reliable semantic supervision signals for training and evaluation.

[0021] ② Low-quality text image generation

[0022] For the above-mentioned clear text images, the present invention designs a systematic low-quality text image generation process, the specific steps are as follows:

[0023] Degradation category setting: Common degradation types are divided into five categories, including visibility changes (brightness, contrast adjustment, fog, shadow), deformation (soft distortion, rigid perspective change), blur (Gaussian blur, motion blur, defocus blur), noise (Gaussian noise, shot noise, impulse noise) and compression (downsampling, JPEG compression).

[0024] Degradation process: For each clear image, k = 3 different degradation operations are randomly selected and applied in a fixed order to generate the corresponding low-quality image. The strength of each degradation operation is adaptively adjusted based on the image size, for example, the Gaussian kernel size σ∈[1.0,3.0] is set, and the distortion amplitude is set to vary by 5%-10% of the long side of the image.

[0025] Visual quality control: LIQE (Liquid Image Quality Evaluation) is used to screen abnormal samples and eliminate samples that have been completely degraded, resulting in complete information loss or unresolvable samples, ensuring that low-quality images still have a certain degree of recognition.

[0026] The resulting DocRobust-VQA dataset contains paired clear-quality and low-quality samples, creating a realistic training and evaluation environment that systematically covers multiple degradation types.

[0027] 2) Design of the DocRobust-Module (DRM):

[0028] This paper proposes a lightweight feature recovery module (DRM) that is inserted between the visual encoder and visual feature mapping layer of a large multimodal model. The module consists of an input mapping layer, a stacked Transformer recovery module, an output mapping layer, an addition and normalization layer, and a fusion layer. The DRM takes the visual tokens output by the visual encoder portion of the large multimodal model as input and outputs enhanced visual tokens.

[0029] The overall structure of DocRobust-Module consists of five components, as follows:

[0030] Input mapping layer: This structure consists of a linear layer with a bias, and the input is the visual token features output by the visual encoder of the multimodal large model This structure reduces the dimension to Ddr=512, thereby reducing the amount of calculation.

[0031] Restoration module: It consists of stacked N=6 lightweight Transformer blocks, each of which uses a multi-head self-attention mechanism (8 heads) and a feedforward network (FFN dimension 2048). This module takes the output features of the input mapping layer as input and the restored features as output.

[0032] Output mapping layer: This structure consists of a linear layer with a bias, which remaps the features restored by the recovery module to the dimension Dvisual = 4096 to obtain supplementary features.

[0033] Addition and normalization layer: The original visual token features and the supplementary features are added element by element, and LayerNorm normalization is performed to stabilize the feature distribution.

[0034] Fusion layer: It consists of two linear layers (with GELU activated in the middle), takes the output features of the sum and normalization layers as input, and outputs the final enhanced visual Token Fenh.

[0035] DRM effectively compensates for the loss of visual and semantic information due to image degradation with minimal parameter introduction, improving the text and image understanding capabilities of large multimodal models in low-quality scenarios. This module offers plug-and-play functionality and can be seamlessly integrated into mainstream large multimodal model architectures, such as InterVL.

[0036] 3) Design a two-stage training strategy for multimodal large models for model training:

[0037] To further improve the recovery effect and robustness of the feature recovery module, this paper designs a multimodal large model training strategy that includes two stages: visual alignment and semantic alignment:

[0038] Visual alignment stage: Only DRM is trained. Paired clear and low-quality images are used to guide DRM to learn to restore the low-quality input visual token to the feature distribution of the clear image. The optimization goal is to minimize the mean square error (MSE) between the enhanced visual token and the clear visual token.

[0039] Semantic alignment stage: After integrating DRM, the model parameters of the multimodal large model other than DRM (including text encoder, visual encoder, visual feature mapping layer, and base large language model) are frozen, and supervised fine-tuning (SFT) is performed on the low-quality text image question answering task. It is worth noting that in this stage, only the DRM parameters are trained and optimized. This training stage guides DRM to learn the ability to support downstream semantic reasoning tasks. Furthermore, in the overall semantic alignment stage, the overall robustness is further improved by jointly training the parameters of the entire model through efficient parameter fine-tuning (such as LoRA).

[0040] 4) In the inference phase, the text image to be understood is input into the multimodal large model after the trained integrated feature recovery module to obtain the response number sequence corresponding to the answer to the question;

[0041] Model reasoning process: The present invention completes the text image understanding task based on the multimodal large model. In the reasoning stage, the text image to be understood and the question text of the corresponding understanding task are input as input. Among them, the text image to be understood is input into the visual encoder part of the multimodal large model, and the visual encoder calculates and obtains the original visual token features; then, the DRM uses the original visual token features as input to calculate and obtain the enhanced visual token features; then, the enhanced visual token features pass through the visual feature mapping layer to obtain the visual token sequence, which participates in the calculation of the base large language model. The question text is input into the text encoder, and the text encoder calculates and obtains the text token sequence that participates in the calculation of the base large language model. In the base large language model, the text token sequence and the visual token sequence are cascaded to participate in the calculation of the base large language model, and finally the response number sequence corresponding to the answer to the question is obtained. Furthermore, the response number sequence can be transcribed into the answer text through a predefined code table.

[0042] 5) Low-quality text image understanding evaluation benchmark construction and model performance evaluation

[0043] This paper systematically generates low-quality images by simulating multiple real degradation factors (such as visibility changes, deformation, blur, noise, and compression distortion), establishes a comprehensive robustness evaluation benchmark DocRobust-Bench, and designs an evaluation system with three indicators: standard scoring, relative scoring, and instance-level relative scoring.

[0044] Standard scoring: Use official evaluation metrics consistent with those of each dataset, such as relaxed accuracy for ChartQA, average normalized Levenshtein similarity (ANLS) for DocVQA and InfographicVQA, VQA score for TextVQA, and answer coverage rate for OCRBench.

[0045] Relative scoring: A standardized ratio metric is introduced to measure the ratio of the accuracy of low-quality image reasoning to the accuracy of clear image reasoning, eliminating the inherent error interference of the model.

[0046] Instance-level relative scoring: This method uses the image-answer pair granularity to count the proportion of inference failures due to degradation, and the fine-grainedness reflects the robustness of the model.

[0047] Through this design, the present invention significantly improves the recognition accuracy and inference stability of large multimodal models in low-quality text images, while maintaining performance in clear image scenarios. Experiments show that the proposed method achieves superior results over existing technologies on multiple mainstream visual question answering datasets (ChartQA, DocVQA, InfographicVQA, TextVQA, and OCRBench), particularly when using low-quality images.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1) The system constructs a standardized dataset and evaluation benchmark for low-quality text image understanding, filling the gap in existing methods in this field;

[0050] 2) We propose a lightweight, plug-and-play feature recovery module that can compensate for the loss of visual and semantic information without significantly increasing computational overhead.

[0051] 3) A two-stage training strategy is designed to effectively guide the feature recovery process and improve the robustness of the model under real complex degradation conditions;

[0052] 4) The proposed method is compatible with mainstream multimodal large model architectures and does not require re-pretraining or major changes to the original system, facilitating actual engineering deployment and application promotion.

[0053] In summary, the present invention addresses the problem of performance degradation of multimodal large-scale models in low-quality text image scenarios, and proposes an innovative and practical systematic optimization solution. It can be widely used in smart government affairs, financial bill processing, medical image analysis, educational technology and other fields, and has good application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Model structure block diagram of the feature recovery module designed for the method of the present invention.

[0055] Figure 2 This is a flowchart of the phased and progressive training of the overall model of the method of the present invention.

[0056] Figure 3 This is a comparison chart of the scores of the method of the present invention and the benchmark method.

[0057] Figure 4 This is a visual comparison result chart. DETAILED DESCRIPTION

[0058] The present invention will be further described below by way of examples in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0059] This paper addresses the issue of large multimodal models' reduced recognition and comprehension capabilities in low-quality text images. We propose a low-quality text image understanding method based on multimodal large-scale model optimization. This method includes the following key components: constructing a low-quality text image visual question-answering dataset, designing a feature recovery module (DocRobust-Module (DRM),) a two-stage training strategy, constructing a low-quality text image evaluation benchmark, and evaluating model performance.

[0060] 1) Construction of low-quality text image visual question answering dataset:

[0061] To systematically evaluate the performance of large multimodal models in low-quality text image scenarios, we constructed a large-scale dataset of paired clear and low-quality images, DocRobust-VQA, based on a variety of text image sources (such as document images, charts, and scene text). The construction process of DocRobust-VQA can be divided into two steps: collecting and screening clear text image data, and generating low-quality text images.

[0062] ① Collection and screening of clear text image data

[0063] This paper first screened high-quality, clear image samples from publicly available text-image visual question-answering datasets, including ChartQA, DocVQA, InfographicVQA, TextVQA, and OCRBench. The screening criteria required clear, legible images and no significant occlusion or distortion of text, ensuring a consistent baseline for the subsequent low-quality image control group. Ultimately, a total of 189,771 clear images were collected, covering a variety of formats, including office documents, charts, natural scene text, and single-line text clippings, demonstrating good domain coverage and scale diversity.

[0064] Each image is accompanied by corresponding question and answer annotations, totaling 417,502 question and answer pairs, providing reliable semantic supervision signals for training and evaluation.

[0065] ② Low-quality text image generation

[0066] For the above-mentioned clear text images, the present invention designs a systematic low-quality text image generation process, the specific steps are as follows:

[0067] Degradation category setting: Common degradation types are divided into five categories, including visibility changes (brightness, contrast adjustment, fog, shadow), deformation (soft distortion, rigid perspective change), blur (Gaussian blur, motion blur, defocus blur), noise (Gaussian noise, shot noise, impulse noise) and compression (downsampling, JPEG compression).

[0068] Degradation process: For each clear image, k = 3 different degradation operations are randomly selected and applied in a fixed order to generate the corresponding low-quality image. The strength of each degradation operation is adaptively adjusted based on the image size, for example, the Gaussian kernel size σ∈[1.0,3.0] is set, and the distortion amplitude is set to vary by 5%-10% of the long side of the image.

[0069] Visual quality control: LIQE (Liquid Image Quality Evaluation) is used to screen abnormal samples and eliminate samples that have been completely degraded, resulting in complete information loss or unresolvable samples, ensuring that low-quality images still have a certain degree of recognition.

[0070] The resulting DocRobust-VQA dataset contains paired clear-quality and low-quality samples, creating a realistic training and evaluation environment that systematically covers multiple degradation types.

[0071] 2) Design of the DocRobust-Module (DRM):

[0072] This paper proposes a lightweight feature recovery module (DRM) that is inserted between the visual encoder and visual feature mapping layer of a large multimodal model. The module consists of an input mapping layer, a stacked Transformer recovery module, an output mapping layer, an addition and normalization layer, and a fusion layer. The DRM takes the visual tokens output by the visual encoder portion of the large multimodal model as input and outputs enhanced visual tokens.

[0073] The overall structure of DocRobust-Module consists of five components, as follows:

[0074] Input mapping layer: This structure consists of a linear layer with a bias, and the input is the visual token features output by the visual encoder of the multimodal large model This structure reduces the dimension to Ddr=512, thereby reducing the amount of calculation.

[0075] Restoration module: It consists of stacked N=6 lightweight Transformer blocks, each of which uses a multi-head self-attention mechanism (8 heads) and a feedforward network (FFN dimension 2048). This module takes the output features of the input mapping layer as input and the restored features as output.

[0076] Output mapping layer: This structure consists of a linear layer with a bias, which remaps the features restored by the recovery module to the dimension Dvisual = 4096 to obtain supplementary features.

[0077] Addition and normalization layer: The original visual token features and the supplementary features are added element by element, and LayerNorm normalization is performed to stabilize the feature distribution.

[0078] Fusion layer: It consists of two linear layers (with GELU activated in the middle), takes the output features of the sum and normalization layers as input, and outputs the final enhanced visual Token Fenh.

[0079] DRM effectively compensates for the loss of visual and semantic information due to image degradation with minimal parameter introduction, improving the text and image understanding capabilities of large multimodal models in low-quality scenarios. This plug-and-play module seamlessly integrates into mainstream large multimodal model architectures, such as InternVL.

[0080] 3) Two-stage training strategy for multimodal large models:

[0081] To further improve the recovery effect and robustness of the feature recovery module, this paper designs a multimodal large model training strategy that includes two stages: visual alignment and semantic alignment:

[0082] Visual alignment stage: Only DRM is trained. Paired clear and low-quality images are used to guide DRM to learn to restore the low-quality input visual token to the feature distribution of the clear image. The optimization goal is to minimize the mean square error (MSE) between the enhanced visual token and the clear visual token.

[0083] Semantic alignment stage: After integrating DRM, the model parameters of the multimodal large model other than DRM (including text encoder, visual encoder, visual feature mapping layer, and base large language model) are frozen, and supervised fine-tuning (SFT) is performed on the low-quality text image question answering task. It is worth noting that in this stage, only the DRM parameters are trained and optimized. This training stage guides DRM to learn the ability to support downstream semantic reasoning tasks. Furthermore, in the overall semantic alignment stage, the overall robustness is further improved by jointly training the parameters of the entire model through efficient parameter fine-tuning (such as LoRA).

[0084] 4) Low-quality text image understanding evaluation benchmark construction and model performance evaluation

[0085] This paper systematically generates low-quality images by simulating multiple real degradation factors (such as visibility changes, deformation, blur, noise, and compression distortion), establishes a comprehensive robustness evaluation benchmark DocRobust-Bench, and designs an evaluation system with three indicators: standard scoring, relative scoring, and instance-level relative scoring.

[0086] Standard scoring: Use the official evaluation metrics consistent with each dataset, such as ChartQA using relaxed accuracy, DocVQA and InfographicVQA using average normalized Levenshtein similarity (ANLS), TextVQA using VQA score, and OCRBench using answer coverage. The formula is as follows:

[0087] S s (X,MLLM)=F data (MLLM(X cor ), X ans ),

[0088] Among them F data represents a scoring function specific to the dataset, X is the model input, and contains X cor and X ans ;X cor represents low-quality text images, X ans The answer is marked. MLLM stands for Multimodal Large Model; S s (X, MLLM) represents the standard scoring calculation function;

[0089] Relative scoring: A standardized ratio metric is introduced to measure the ratio of the accuracy of low-quality image reasoning to the accuracy of clear image reasoning, eliminating the inherent error interference of the model. The formula is as follows:

[0090]

[0091] Among them, S r (X, MLLM) is the relative score calculation function; Xcle represents the number of clean text images. This metric reflects the ratio of correct answers on low-quality images to correct answers on clean images, providing a more objective measure of the model's ability to understand low-quality images.

[0092] Instance-level relative scoring: This method uses the image-answer pair granularity to count the proportion of inference failures due to degradation, and reflects the robustness of the model at a fine-grained level. The formula is as follows:

[0093]

[0094] Here, 1 is the indicator function, Ss(Xj,MLLM) and Sr(Xj,MLLM) represent the standard score and relative score, respectively, calculated for each image-answer pair Xj in the dataset. The summation effectively represents the proportion of images that can be correctly understood in their clear state but fail to be understood after low-quality processing. A lower proportion indicates greater model robustness, so Sir reflects the model's robustness at the instance level.

[0095] Based on the above indicators, the present invention systematically compared and analyzed the method of the present invention with existing methods. The results showed that the method of the present invention can correctly extract text information and give accurate answers, while the baseline model has a significantly higher output error rate.

[0096] In specific implementation, the training hyperparameter settings, reasoning process, and experimental effect verification of the method proposed in the present invention are as follows:

[0097] 1) Training hyperparameter setting

[0098] The present invention proposes a phased approach, such as Figure 2 As shown, it specifically includes two stages:

[0099] ① Visual alignment stage:

[0100] Training objective: Minimize the mean square error (MSE) between the visual features of the low-quality input text image restored by DRM and the visual features of the clear image.

[0101] Training settings: Freeze the text encoder, visual encoder, visual feature map layer, and base language model of the large multimodal model, and only update the DRM parameters. Use a batch size of 256, an initial learning rate of 0.001, a warmup of 0.5 training epochs, and a 1-cycle scheduling strategy for a total of 5 training epochs.

[0102] ② Local semantic alignment stage:

[0103] Training objective: Under low-quality text image input, train DRM to improve the inference accuracy of the text image VQA task, using cross-entropy loss.

[0104] Training settings: Freeze the text encoder, visual encoder, visual feature map layer, and base language model of the large multimodal model, and only update the DRM parameters. Batch size is 128, and the training strategy follows the InternVL standard fine-tuning configuration.

[0105] ③ Overall semantic alignment stage:

[0106] Training goal: Jointly optimize DRM and all modules of the basic large model (using LoRA parameters for efficient fine-tuning) to further improve global semantic understanding capabilities.

[0107] LoRA settings: rank r = 128, applied to the mapping layer and some Transformer modules, total training is about 10 epochs.

[0108] 6) Reasoning process

[0109] The overall process of the inference phase is as follows:

[0110] Input low-quality text image

[0111] Extract preliminary visual token features Fvis through the visual encoder;

[0112] Input DRM module for feature recovery and enhancement, and output Fenh;

[0113] After the visual feature mapping layer is fused with the text features;

[0114] The base language model performs reasoning to generate the final answer A.

[0115] Inference does not require additional computing overhead. The DRM module executes in parallel with the main model to ensure overall inference efficiency.

[0116] 7) Experimental setup and effect verification

[0117] To verify the effectiveness of our method, we conducted evaluations on low-quality versions of various datasets, including CROHME, DocVQA, and TextVQA. We used the InternVL-2.5 series model as a baseline and compared it with and without integrating our method.

[0118] like Figure 3 As shown in the experimental results,

[0119] On DocRobust-Bench, after adding DRM, the average standard scores of each subset increased by 11.4%-26.8%;

[0120] Both the relative score and the instance-level relative score are significantly higher than the baseline, especially in complex degradation scenarios (such as Gaussian blur + distortion + noise).

[0121] In a clear image scenario, the method of the present invention will not cause performance degradation, ensuring that conventional task requirements are taken into account.

[0122] Figure 4 A practical reasoning example was demonstrated. Under conditions of severe blur, compression and noise interference, the method of the present invention can correctly extract text information and give accurate answers, while the output error rate of the baseline model is significantly higher.

[0123] It should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A text image understanding method based on multimodal large model optimization, characterized in that: By designing a dataset and evaluation benchmark, a lightweight feature recovery module, and a two-stage training strategy, we enhance the recognition and reasoning capabilities of low-quality text images. The steps include: 1) Based on a variety of text image sources, a low-quality text image visual question answering dataset is constructed; the dataset includes paired samples of clear and low-quality images; 2) Design a plug-and-play feature recovery module that can be seamlessly integrated into the multimodal large model architecture; The feature recovery module includes: an input mapping layer, a stacked Transformer recovery module, an output mapping layer, an addition and normalization layer, and a fusion layer; The input mapping layer consists of a linear layer with a bias. The input is the visual token features output by the visual encoder of the multimodal large model. After dimensionality reduction through the input mapping layer, the output features are reduced, thereby reducing the amount of computation. The stacked Transformer recovery module consists of N stacked lightweight Transformer blocks, each of which uses a multi-head self-attention mechanism and a feedforward network. This module takes the output features of the input mapping layer as input and outputs the recovered features. The output mapping layer consists of a linear layer with a bias, which remaps the output features of the stacked Transformer recovery module to a high dimension to obtain supplementary features; The sum normalization layer is used to add the visual token features and the supplementary features element by element and perform normalization to stabilize the feature distribution; The fusion layer consists of two linear layers, which take the output features of the sum and normalization layers as input and output the final enhanced vision; 3) Design a multimodal large model training strategy that includes two stages: visual alignment and semantic alignment, and perform model training; Visual alignment: Only the feature recovery module is trained. Using paired clear and low-quality images, the feature recovery module is guided to learn how to restore the input low-quality visual tokens to the feature distribution of the clear image. The optimization goal of model training is to minimize the mean squared error between the enhanced visual tokens and the clear visual tokens. Semantic alignment: After integrating the feature recovery module into the multimodal large model architecture, the model parameters of the multimodal large model except for the feature recovery module are frozen, including the model parameters of the text encoder, visual encoder, visual feature mapping layer, and the base large language model. Supervised fine-tuning is performed on the low-quality text image question answering task, that is, only the parameters of the feature recovery module are trained and optimized. Then, the parameters of the entire model are jointly trained by adopting an efficient parameter fine-tuning method. 4) In the inference stage, the text image to be understood is input into the trained multimodal large model of the integrated feature recovery module to obtain the response number sequence corresponding to the answer to the question, which can be further transcribed into the answer text.

2. The text image understanding method based on multimodal large model optimization according to claim 1, characterized in that: The multimodal large model specifically adopts InterVL.

3. The text image understanding method based on multimodal large model optimization according to claim 1, characterized in that: The stacked Transformer recovery module specifically stacks 6 lightweight Transformer blocks; each Transformer block adopts a multi-head self-attention mechanism with 8 heads; the feedforward network dimension is 2048.

4. The text image understanding method based on multimodal large model optimization according to claim 1, characterized in that: Step 1) Constructing a low-quality text image visual question answering dataset includes: collecting and screening clear text image data and generating low-quality text images; wherein: ① The clear text image data collection and screening process involves selecting a control group of low-quality images from a public text image visual question-answering dataset. The public text image visual question-answering dataset includes office documents, charts, natural scene text, and single-line text cropped images. Each image is annotated with a corresponding question and answer to form a question-answer pair. ② Low-quality text image generation process, including: Categorize degenerate operations; Degradation operation process: For each clear image, different types of degradation operations are randomly selected and applied in a fixed order to generate the corresponding low-quality image; the intensity of each degradation operation is adaptively adjusted according to the image size; Visual quality control: Screen out abnormal samples and remove samples that are completely lost or uninterpretable due to excessive degradation; Generate a dataset containing paired samples of clear and low-quality images.

5. The text image understanding method based on multimodal large model optimization according to claim 4 is characterized in that: Public text-image visual question answering datasets include ChartQA, DocVQA, InfographicVQA, TextVQA, and OCRBench datasets.

6. The text image understanding method based on multimodal large model optimization according to claim 4, characterized in that: Degradation operations are categorized into: visibility change, deformation, blur, noise, and compression.

7. The text image understanding method based on multimodal large model optimization according to claim 1, characterized in that: A low-quality text image understanding evaluation benchmark is also constructed to evaluate the model performance of a large multimodal model with an integrated feature recovery module; the low-quality text image understanding evaluation benchmark includes standard scoring, relative scoring, and instance-level relative scoring.

8. The text image understanding method based on multimodal large model optimization according to claim 7, characterized in that: The relative score specifically introduces a standardized ratio indicator, that is, the ratio of the low-quality image reasoning accuracy to the clear image reasoning accuracy, which is expressed as: Among them, S r (X, MLLM) is the relative score calculation function; X cle represents a clear text image; the relative score index reflects the ratio between the correct answer on the low-quality image and the correct answer on the clear image; F data represents a scoring function specific to the dataset, X is the model input, and contains X cor and X ans ;X cor represents low-quality text images, X ans is the marked answer; MLLM stands for multimodal large model; Instance-level relative scoring: Counts the proportion of reasoning failures due to degradation at the image-answer pair granularity; expressed as: Where 1 is the indicator function, Ss(Xj,MLLM) and Sr(Xj,MLLM) represent the standard score and relative score calculated for a single image-answer pair Xj in the dataset, respectively.

Images