A complex format test paper content recognition method and system

Abstract

The application provides a complex format test paper content recognition method and system, which introduces a mixed expert low rank adaptation (MoE-LoRA) and an attention mechanism to efficiently fine-tune and reconstruct a pre-trained visual language large model, improves the focusing ability of the model on the answering area and the generalization recognition ability of the model on different question types, and only trains the newly added adaptive parameters in the training process, freezes all original parameters of the basic visual language large model, and greatly reduces the training cost while ensuring the recognition performance. The application improves the "additive fusion" mechanism of the traditional MoE-LoRA into an attention selective fusion mechanism, so that the model can dynamically and selectively integrate pre-training knowledge and newly injected domain knowledge, thereby realizing accurate and low illusion student answer content recognition without format priori.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and in particular relates to a method and system for recognizing the content of complex-format test papers. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent marking systems have become an important application area in the education industry. How to efficiently and accurately identify student answers in different formats is the core challenge in improving the practicality of marking systems. Currently, there are two main technical approaches in this field:

[0003] 1. Traditional OCR analysis methods

[0004] Based on a predefined rule engine and layout analysis, the test paper layout information (such as the coordinates of the question type area and the position of the question number) needs to be manually configured in advance. After the area is cut by the algorithm, the corresponding special recognition model (such as the handwriting, printed text, and formula recognition model) is called for processing.

[0005] 2. End-to-end deep learning methods

[0006] The system uses object detection or instance segmentation techniques to locate the answer region and employs sequence recognition models such as CRNN and Transformer for identification. Some advanced systems attempt to introduce multimodal large models, but still rely on explicit format or question type annotations as input.

[0007] Despite the progress made by the above methods, the following prominent bottlenecks remain:

[0008] (1) Strong dependence on layout

[0009] Traditional methods require manual annotation and adaptation for each test paper template, which makes it difficult to flexibly cope with complex scenarios with diverse formats and inconsistent layouts in actual applications.

[0010] (2) High model switching and system complexity

[0011] Different recognition models need to be called for different question types (such as multiple choice, fill-in-the-blank, problem solving, and drawing questions), which leads to redundancy in system architecture, reduced reasoning efficiency, and high maintenance and expansion costs.

[0012] 3. Rigid expert panel mechanism

[0013] While existing hybrid expert (MoE)-based architectures can integrate multiple recognition models, their expert combination methods are usually fixed, lacking the ability to dynamically respond to input semantics and contextual structure, and are prone to hallucination phenomena due to insufficient understanding of context. Summary of the Invention

[0014] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art, and proposes a method and system for recognizing the content of complex-format test papers. It does not require manual pre-configuration of test paper format information, and can adaptively understand and accurately recognize the student answers of various question types in complex-format test papers.

[0015] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0016] In a first aspect, the present invention provides a method for recognizing the content of complex-format test papers, comprising the following steps:

[0017] Data preprocessing steps: Perform quality enhancement and standardization on the original test paper image to generate a preprocessed image;

[0018] Visual language big model processing steps: Input the preprocessed image into the pre-trained visual language big model with frozen parameters, and extract high-level semantic features. The visual language big model includes a visual encoder, a projection layer and a big language model.

[0019] MoE-LoRA efficient fine-tuning steps: Based on the high-level semantic features extracted in the visual language large model processing steps, the parameters of the visual language large model are efficiently fine-tuned through a low-rank adapting network of a hybrid expert architecture, and expert features are output; the hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and input adaptive expert selection is achieved through a learnable gating network.

[0020] Attention enhancement step: The output of the feedforward neural network of the large language model in the visual language large model processing step is used as the query vector, and the expert features output by the MoE-LoRA efficient fine-tuning step are used as the key vector and value vector. Attention weights are calculated through a cross-modal attention mechanism to achieve dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features.

[0021] Multi-task parsing steps: Based on the enhanced features generated in the attention enhancement step, the user's input natural language instructions are parsed, the task type is identified, and a structured output is generated according to the task type.

[0022] Furthermore, the data preprocessing steps specifically include:

[0023] A deep learning-based Gaussian mixture noise suppression algorithm is used to denoise the image;

[0024] By detecting straight lines at the edges of the test paper using the Hough transform, and calculating the perspective transformation matrix, tilt correction and deformation recovery are achieved.

[0025] By utilizing edge detection and connected component analysis, the effective answer area is automatically located, and background interference and redundant information are removed.

[0026] Furthermore, in the MoE-LoRA high-efficiency fine-tuning step:

[0027] Multiple low-rank adapter experts are configured for different question types, and different low-rank dimensions are set for experts of different question types;

[0028] The learnable gated network employs a Top-K sparse activation strategy to balance computational efficiency and model capacity;

[0029] A two-layer routing structure of group-level routers and expert-level routers is adopted. First, the question types are classified into groups, and then the K most relevant experts are selected within each group.

[0030] Furthermore, the MoE-LoRA efficient fine-tuning step employs a three-stage fine-tuning training strategy:

[0031] Phase A: Single-expert warm-up training, freezing the pre-trained visual language large model, routing network and attention-related parameters; each time only a single LoRA expert and its corresponding question type data are selected for independent training, so that each expert can initially learn the visual features and answer distribution of the corresponding question type. Each expert is trained independently for 3-5 epochs to avoid mutual interference between different question type features.

[0032] Phase B: Router network training, freezing the base visual language large model, freezing all parameters of LoRA experts, training group-level routers and expert-level routers, introducing load balancing loss to avoid expert bias or idleness; under the premise of being aware of expert characteristics, the routing network learns the mapping relationship of "input question type characteristics → expert group → specific expert" and establishes a stable expert selection mechanism.

[0033] Phase C: MoE and attention are jointly fine-tuned, freezing all original parameters of the base visual language large model; all LoRA expert parameters, routing network parameters, attention projection matrix and illusion suppression gating parameters are unfrozen, and end-to-end joint training is performed.

[0034] Furthermore, the MoE-LoRA high-efficiency fine-tuning step also includes:

[0035] Load balancing: Add load balancing loss to routing loss;

[0036] Hallucination suppression: Enhances expert differentiation, making routing choices more explicit.

[0037] Furthermore, the attention enhancement step specifically includes:

[0038] The output of the feedforward neural network of the large visual language model is used as the query vector, and the output of MoE-LoRA for efficient fine-tuning is used as the key vector and value vector.

[0039] The similarity between the query vector and the key vector is calculated by scaling the dot product attention, and then the similarity is summed with the value vector after scaling and normalization.

[0040] A multi-head attention mechanism is used to compute multiple attention heads in parallel, and the outputs of each head are concatenated.

[0041] Information can be transferred across layers and feature distribution can be stabilized through residual connections and layer normalization.

[0042] Furthermore, the attention enhancement step also includes a hallucination suppression gating mechanism:

[0043] During the training phase, the gating parameters can be learned, and the contribution of each attention head can be reduced through backpropagation;

[0044] During the inference phase, a pre-computed illusion head mask is loaded, and the contribution weight of the attention head that generates the illusion is reset to zero.

[0045] Furthermore, the multi-task parsing step specifically includes:

[0046] User intent is classified and task type is identified using a lightweight BERT model;

[0047] Break down the complex intent into a sequence of atomic tasks;

[0048] Generate a routing plan based on the mapping table from task tokens to expert groups and then to specific experts;

[0049] The system uses three levels of prompt templates—system-level, task-level, and instance-level—to constrain the output format to be structured data.

[0050] Secondly, the present invention provides a complex-format test paper content recognition system, comprising:

[0051] The data preprocessing module is used to enhance and standardize the original test paper image to generate a preprocessed image;

[0052] The visual language large model processing module is used to input the pre-processed image into the pre-trained visual language large model with frozen parameters and extract high-level semantic features. The visual language large model includes a visual encoder, a projection layer and a large language model.

[0053] The MoE-LoRA high-efficiency fine-tuning module is used to efficiently fine-tune the parameters of the visual language large model based on the high-level semantic features extracted by the visual language large model processing module, and output expert features through a low-rank adapter network of a hybrid expert architecture. The hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and achieves input-adaptive expert selection through a learnable gating network.

[0054] The attention enhancement module is used to take the output of the feedforward neural network of the large language model in the visual language large model processing module as the query vector, and the expert features output by the MoE-LoRA high-efficiency fine-tuning module as the key vector and value vector. It calculates the attention weight through a cross-modal attention mechanism to realize the dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features.

[0055] The multi-task parsing module is used to parse the natural language instructions input by the user based on the enhanced features generated by the attention enhancement module, identify the task type, and generate structured output according to the task type.

[0056] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above for a method of recognizing the content of a complex-format test paper.

[0057] Compared with existing technologies, the method and system for recognizing complex-format test paper content described in this invention have the following advantages:

[0058] This invention innovatively uses the output of the MoE-LoRA module as the Key and Value of the attention mechanism, and the output of the base model FFN (Feedforward Neural Network) as the Query, to construct a cross-modal attention alignment mechanism. This enables the gating network to dynamically select experts based on the visual features of the question type, and the attention weights to automatically learn the correspondence between visual regions and text semantics, thus avoiding feature interference from irrelevant regions.

[0059] During the training process, this invention only trains the newly added adaptation parameters and freezes all the original parameters of the basic visual language model, thereby significantly reducing training costs while ensuring recognition performance.

[0060] This invention introduces independent illusion suppression mechanisms in the MoE-LoRA efficient fine-tuning step and the attention enhancement step, forming a dual illusion suppression: the former suppresses illusions caused by expert selection errors at the expert routing level, and the latter suppresses content generation illusions caused by improper attention weight allocation at the attention fusion level. The two work together to significantly improve the reliability of the recognition results.

[0061] This invention designs a unified multi-task prompt template that integrates different question types (such as text detection, handwritten character recognition, option judgment, mathematical formula extraction, etc.) into the same model, and guides the model to dynamically adapt to different question types through prompt instructions. Attached Figure Description

[0062] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0063] Figure 1 This is the overall implementation flow of the present invention;

[0064] Figure 2 This is a schematic diagram of the MoE-LoRA high-efficiency fine-tuning process of the present invention;

[0065] Figure 3 This is a schematic diagram illustrating the division of the expert group in this invention;

[0066] Figure 4 This is a schematic diagram of the attention enhancement process of the present invention;

[0067] Figure 5 This is a schematic diagram of the multi-head parallel design of the present invention;

[0068] Figure 6 This is a schematic diagram of the multitasking prompts of the present invention. Detailed Implementation

[0069] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0070] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0071] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0072] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0073] Example 1

[0074] like Figure 1 As shown, this invention provides a method for recognizing content in complex-formatted test papers. By introducing Hybrid Expert Low-Rank Adaptation (MoE-LoRA) and an attention mechanism, it efficiently fine-tunes and restructures a pre-trained visual language model, improving the model's ability to focus on the answer area and its generalization ability to recognize different question types. The traditional "additive fusion" mechanism of MoE-LoRA is improved into an attention-selective fusion mechanism, enabling the model to dynamically and selectively integrate pre-trained knowledge with newly injected domain knowledge, thereby achieving accurate and low-illusion recognition of student answers without prior knowledge of the format. The specific implementation steps of this invention are as follows:

[0075] 1. Data preprocessing steps: Perform quality enhancement and standardization processing on the original test paper image to generate a preprocessed image.

[0076] The original exam paper images undergo quality enhancement and standardization processing to provide high-quality input for subsequent visual understanding. Specifically, this includes:

[0077] Image denoising: A deep learning-based Gaussian mixture noise suppression algorithm is used to eliminate particle noise introduced during scanning / capturing.

[0078] Geometric correction: The Hough transform is used to detect the straight lines at the edges of the test paper, and the perspective transformation matrix is ​​calculated to achieve tilt correction and deformation recovery;

[0079] Intelligent cropping: Automatically locates the effective answer area using edge detection and connected component analysis, removing background interference and redundant information in headers and footers.

[0080] 2. Visual language large model processing steps: Input the preprocessed image into the pre-trained visual language large model with frozen parameters to extract high-level semantic features. The visual language large model includes a visual encoder, a projection layer and a large language model.

[0081] It adopts a mainstream visual language big model as the base, integrating a visual encoder (ViT), a projection layer and a big language model (LLM) to extract high-level semantic features, understand visual-text associations and generate structured descriptions.

[0082] 3. MoE-LoRA Efficient Fine-Tuning Step: Based on the high-level semantic features extracted in the visual language large model processing step, the parameters of the visual language large model are efficiently fine-tuned through a low-rank adapting network of a hybrid expert architecture, and expert features are output. The hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and the expert selection is adaptive to the input through a learnable gating network.

[0083] Construct a low-rank adapting network with a hybrid expert (MoE) architecture, and configure dedicated LoRA experts for different question types such as multiple choice, fill-in-the-blank, short answer, and calculation questions;

[0084] The input-adaptive expert selection is achieved through a learnable gating network, and a Top-K sparse activation strategy is adopted to balance computational efficiency and model capacity. Top-K: K=2 for objective questions and K=3 for subjective questions.

[0085] Different low-rank dimensions are set according to the characteristics of the question type (e.g., r=8 for multiple-choice questions and r=16 for short-answer questions) to optimize the granularity and expressive power of feature extraction;

[0086] Efficient domain transfer is achieved by training only the expert network and gating parameters (accounting for less than 1% of the total base parameters).

[0087] The specific implementation process of this step is as follows: Figure 2 As shown, it includes:

[0088] (1) Data collection and labeling

[0089] It collects student answer sheets in various formats, including scanned test papers, electronic test papers, and handwritten answer sheets. It includes images of various subjects, question types (multiple choice, fill-in-the-blank, short answer, math problems, etc.), and layouts. It labels the overall layout type, column structure, header and footer areas, question types, student answer areas, answer types, and answer content of the test paper.

[0090] (2) Expert pre-allocation and initialization

[0091] Expert group division: Seven parallel low-rank experts are configured, each consisting of a pair of low-rank decomposition matrices with differentiated rank settings, dynamically adjusted according to task complexity. For example... Figure 3 As shown. The objective question expert group includes experts in multiple-choice, true / false, and fill-in-the-blank questions; the subjective question expert group includes experts in essay, short-answer, math, and programming questions. Differentiated experts are constructed through low-rank matrix zero initialization and task embedding k-means pre-assignment.

[0092] (3) Routing network setup

[0093] A hierarchical gating network is constructed to achieve group-level coarse classification and expert-level fine-grained Top-K coefficient routing. Specifically, this includes: building a group router and an expert router. The group router determines whether the input feature belongs to an "objective question group" or a "subjective question group". Based on the coarse classification result, the second-layer router in the corresponding expert group is activated, and the K most relevant experts in the group are selected (Top-K sparse routing).

[0094] (4) Three-stage fine-tuning training

[0095] Phase A: Single-expert warm-up training, freezing the pre-trained visual language large model, routing network, and attention-related parameters; each time, only a single LoRA expert and its corresponding question type data are selected for independent training, allowing each expert to initially learn the visual features and answer distribution of the corresponding question type. Each expert is trained independently for 3–5 epochs to avoid interference between features of different question types;

[0096] Phase B: Routing network training, freezing the base visual language large model; freezing all parameters of LoRA experts, training group-level routers and expert-level routers, introducing load balancing loss to avoid expert bias or idleness; under the premise of being able to perceive expert characteristics, the routing network learns the mapping relationship of "input question type characteristics → expert group → specific expert" to establish a stable expert selection mechanism.

[0097] Freeze all expert parameters, train group-level and expert-level routers, and introduce load balancing losses.

[0098] Phase C: Joint fine-tuning of MoE and attention, freezing all original parameters of the base visual language model; unfreezing all LoRA expert parameters, routing network parameters, attention projection matrix, and hallucination suppression gating parameters, and performing end-to-end joint training.

[0099] (5) Load balancing and hallucination suppression optimization

[0100] Load balancing: Load balancing loss is added to routing loss to penalize experts who are always selected or never selected; router z-loss is used to control the distribution of router output, and the utilization of experts is ensured to be uniform through load balancing loss and router z-loss.

[0101] Hallucination suppression: Enhances expert discrimination and makes routing more explicit. If an expert (or attention head) is found to be generating "hallucinatory" content that is inconsistent with visual evidence, it is set to zero.

[0102] This step outputs expert features obtained after processing by a low-rank adaptive network with a hybrid expert architecture. These expert features carry question-type-specific professional features (such as refined visual semantic representations of different question types, such as multiple choice, fill-in-the-blank, and short answer questions).

[0103] 4. Attention Enhancement Step: The output of the feedforward neural network of the large language model in the visual language large model processing step is used as the query vector, and the expert features output from the MoE-LoRA efficient fine-tuning step are used as the key vector and value vector. Attention weights are calculated through a cross-modal attention mechanism to achieve dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features.

[0104] Achieving dynamic fusion of general and expert knowledge through cross-modal attention mechanisms, and systematically suppressing model illusion:

[0105] Overcoming the limitations of traditional additive fusion, a selective fusion mechanism based on cross-attention is constructed.

[0106] Query-Key-Value configuration:

[0107] Query(Q): Originates from the output of the FFN layer of the base model, carrying general semantic understanding and global contextual information;

[0108] Key (K) and Value (V): Derived from the output of the MoE-LoRA module, carrying specialized features specific to the question type;

[0109] By scaling dot product attention to calculate the association strength between general knowledge queries and expert knowledge sources, task-adaptive feature selection is achieved.

[0110] By leveraging the sparsity and interpretability of attention weights, an explicit mapping constraint from visual regions to generated content is established, blocking the generation path of fictional content lacking visual basis.

[0111] The specific implementation process of this step is as follows: Figure 4 As shown, it includes:

[0112] (1) Input projection

[0113] Separation of Query-Key-Value Sources:

[0114] Query: Base model FFN output

[0115] Role: Information requester, carrying general semantic understanding information ("This is a programming problem").

[0116] Key: MoE-LoRA module output

[0117] Role: Information index, carrying the location of visual features extracted by experts.

[0118] Value: MoE-LoRA output

[0119] Role: Information content, carrying specific feature representations extracted by experts.

[0120] Mathematical calculations:

[0121] ;

[0122] Among them, QK TThis represents the dot product of each query and all keys; a larger value indicates higher relevance. d k It is the dimension of each key vector (K), used as a scaling factor to avoid QK. T If the inner product value is too large, it will prevent the gradient from entering the saturation region of softmax (where the gradient is minimal), thereby maintaining training stability.

[0123] The core task of this step is to prepare three representation vectors—Query, Key, and Value—for subsequent attention calculations. The Query vector originates from the base model and carries general semantic understanding information for the question type. The Key and Value vectors originate from MoE-LoRA experts optimized for specific question types and represent refined visual feature extraction capabilities.

[0124] (2) Scaling Dot Product Attention

[0125] Attention score calculation employs matrix multiplication for batch parallel processing. For each position in the sequence, the model calculates the similarity between its query and the keys of all positions, forming an attention score matrix. This matrix, after scaling and normalization, is multiplied by the value matrix to perform a weighted summation, generating context-aware features that incorporate expert details.

[0126] (3) Parallel computation of multiple attention heads

[0127] Multi-head parallel design such as Figure 5 As shown.

[0128] Calculate separately: ;

[0129] Multi-head output: ;

[0130] Among them, h base This indicates the output of the FFN base model, h moe This indicates the output of the MoE-LoRA expert. , , Let head represent the Q / K / V projection matrix of the i-th head. i W represents the output of the i-th attention head. o This represents the multi-head output projection matrix, where MultiHead represents the final enhanced feature.

[0131] (4) Hallucination suppression gating

[0132] Training phase: The gating parameters (a set of weights) are learnable, and the model automatically learns to reduce the contribution of attention heads that are prone to hallucinations through backpropagation.

[0133] Inference phase: The system loads a pre-computed "hallucination head mask". This mask is obtained by analyzing the validation set: if a certain attention head shows high activity in the samples that generate hallucinations, its contribution weight is set to zero (permanently masked).

[0134] Specific methods include:

[0135] 1. Labeling Hallucination Samples

[0136] Samples whose output content does not match the test paper image or contains unfounded fabricated content are labeled as hallucination samples.

[0137] 2. Calculate the activity level of the attention head.

[0138] ;

[0139] h represents the attention head, AttnWeight h,i Let be the attention weight of the i-th token, and N be the total number of tokens.

[0140] 3. Calculation of hallucination tendencies

[0141] ;

[0142] The average activity levels were measured on hallucination and normal samples, respectively.

[0143] 4. Determine if you are experiencing hallucinations.

[0144] Hallucination activity ;

[0145] Hallucination tendencies ;

[0146] Hallucination samples showed a high activity rate of ≥70%;

[0147] If any two of the criteria are met, the user is identified as having hallucinations.

[0148] 5. Inference shielding

[0149] Set the contribution weight of the illusion head to zero and permanently block it.

[0150] (5) Residual connectivity and layer normalization

[0151] Information is transferred across layers through residual connections, and the distribution range of features is stabilized through layer normalization. This step ensures that the enhanced features contain both the refined information extracted by experts and the general semantic understanding of the base model.

[0152] Output calculation:

[0153]

[0154] Parameter freezing strategy:

[0155] Training only: Attention projection matrix + gating parameters + MoE LoRA expert parameters + routing network parameters;

[0156] Freeze: All original parameters of the base model.

[0157] This step outputs an enhanced feature that retains the general semantic understanding capability of the base model, incorporates expert features optimized for specific question types, and has undergone illusion suppression processing.

[0158] 5. Multi-task parsing step: Based on the enhanced features generated in the attention enhancement step, parse the natural language instructions input by the user, identify the task type, and generate structured output according to the task type.

[0159] As the system's human-computer interaction interface and task scheduling hub, it is responsible for task understanding and execution control.

[0160] Based on the generated enhanced features, user intent is classified using a lightweight BERT model, the natural language instructions input by the user are parsed, and the core task types (layout analysis, content recognition, confidence assessment, etc.) are identified.

[0161] Break down complex requirements into a sequence of executable subtasks, and determine the execution order and dependencies;

[0162] Generate a routing plan based on the mapping table from task tokens to expert groups and then to specific experts;

[0163] The output format is constrained to be structured data through three levels of Prompt templates: system-level, task-level, and instance-level.

[0164] The specific implementation process of this step includes:

[0165] 1. Design multi-task prompts, such as... Figure 6 As shown.

[0166] 2. Task Analysis

[0167] Intent recognition: Use a lightweight BERT model to classify user intent (recognition / verification / statistics).

[0168] Task decomposition: Break down a complex intent into a sequence of atomic tasks.

[0169] Expert Mapping: Generate a routing plan based on the mapping table of task token → expert group → specific expert. (See the mapping table for reference.) Figure 3 .

[0170] Assembly prompt: Assembly complete prompt.

[0171] The following describes this solution through specific implementation scenarios.

[0172] Implementation Scenario 1: Recognition of Hybrid Format Math Exam Papers

[0173] Input: A scanned math exam paper containing multiple choice questions (10 questions), true / false questions (10 questions), fill-in-the-blank questions (5 questions), short answer questions (3 questions), and math questions (1 question), with handwritten formulas and handwritten answers.

[0174] Processing flow:

[0175] S1: The system prompt guides the model to analyze the layout and identify 29 answer areas;

[0176] S2: In MoE-LoRA, the short answer expert is activated to process the short answer question area, the multiple choice expert processes the option area, the true / false expert processes the true / false question area, the fill-in-the-blank expert processes the fill-in-the-blank area, and the math problem expert processes the math problem area.

[0177] S3: Attention mechanisms ensure that no false numbers are generated in the fill-in-the-blank question area (suppressing hallucinations);

[0178] Output: Structured JSON containing coordinates of each question, identified content, confidence level of 0.92, and no hallucination markers.

[0179] Results: No pre-training was required for this layout; the first recognition achieved an accuracy of 91.2%.

[0180] Implementation Scenario Example 2: Hallucination Suppression Verification

[0181] Adversarial example: The area for question 5 on the test paper was heavily smudged with ink, and a traditional VLM might "illusory" the fictional content beneath the smudge.

[0182] This invention processes:

[0183] The attention module found that the visual features (key / value) of this region had a low match with the text query during calculation.

[0184] The hallucination suppression gating detects overactivation of the "text completion" head and reduces its weight;

[0185] The model outputs "Content unrecognizable" for this area with a confidence level of 0.45, and automatically marks it for manual review.

[0186] To avoid generating false answers and ensure fair scoring.

[0187] Example 2

[0188] A complex-format test paper content recognition system includes:

[0189] The data preprocessing module is used to enhance and standardize the original test paper image to generate a preprocessed image;

[0190] The visual language large model processing module is used to input the pre-processed image into the pre-trained visual language large model with frozen parameters and extract high-level semantic features. The visual language large model includes a visual encoder, a projection layer and a large language model.

[0191] The MoE-LoRA high-efficiency fine-tuning module is used to efficiently fine-tune the parameters of the visual language large model based on the high-level semantic features extracted by the visual language large model processing module, and output expert features through a low-rank adapter network of a hybrid expert architecture. The hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and achieves input-adaptive expert selection through a learnable gating network.

[0192] The attention enhancement module is used to take the output of the feedforward neural network of the large language model in the visual language large model processing module as the query vector, and the expert features output by the MoE-LoRA high-efficiency fine-tuning module as the key vector and value vector. It calculates the attention weight through a cross-modal attention mechanism to realize the dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features.

[0193] The multi-task parsing module is used to parse the natural language instructions input by the user based on the enhanced features generated by the attention enhancement module, identify the task type, and generate structured output according to the task type.

[0194] Example 3

[0195] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor as described above for a method of recognizing the content of a complex-format test paper.

[0196] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for recognizing the content of complex-format test papers, characterized in that: Includes the following steps: Data preprocessing steps: Perform quality enhancement and standardization on the original test paper image to generate a preprocessed image; Visual language big model processing steps: Input the preprocessed image into the pre-trained visual language big model with frozen parameters, and extract high-level semantic features. The visual language big model includes a visual encoder, a projection layer and a big language model. MoE-LoRA efficient fine-tuning steps: Based on the high-level semantic features extracted in the visual language large model processing steps, the parameters of the visual language large model are efficiently fine-tuned through a low-rank adapting network of a hybrid expert architecture, and expert features are output; the hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and input adaptive expert selection is achieved through a learnable gating network. In this step, multiple low-rank adapter experts are configured for different question types, and different low-rank dimensions are set for experts of different question types. The learnable gated network employs a Top-K sparse activation strategy to balance computational efficiency and model capacity; A two-layer routing structure of group-level routers and expert-level routers is adopted. First, the question types are classified into groups, and then the K most relevant experts are selected within each group. This step employs a three-stage fine-tuning training strategy: Phase A: Single-expert warm-up training, freezing the pre-trained visual language large model, routing network and attention-related parameters; each time only a single LoRA expert and its corresponding question type data are selected for independent training, so that each expert can initially learn the visual features and answer distribution of the corresponding question type. Each expert is trained independently for 3-5 epochs to avoid mutual interference between different question type features. Phase B: Router network training, freezing the base visual language large model; freezing all parameters of LoRA experts; training group-level routers and expert-level routers, introducing load balancing loss to avoid expert bias or idleness; under the premise of being aware of expert characteristics, the routing network learns the mapping relationship of "input question type characteristics → expert group → specific expert" to establish a stable expert selection mechanism. Phase C: Fine-tuning of MoE and attention, freezing all original parameters of the base visual language large model; unfreezing all LoRA expert parameters, routing network parameters, attention projection matrix and illusion suppression gating parameters, and performing end-to-end joint training; Attention enhancement step: The output of the feedforward neural network of the large language model in the visual language large model processing step is used as the query vector, and the expert features output by the MoE-LoRA efficient fine-tuning step are used as the key vector and value vector. Attention weights are calculated through a cross-modal attention mechanism to achieve dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features. Multi-task parsing steps: Based on the enhanced features generated in the attention enhancement step, the user's input natural language instructions are parsed, the task type is identified, and a structured output is generated according to the task type.

2. The method for recognizing the content of complex-format test papers according to claim 1, characterized in that: The data preprocessing steps specifically include: A deep learning-based Gaussian mixture noise suppression algorithm is used to denoise the image; By detecting straight lines at the edges of the test paper using the Hough transform, and calculating the perspective transformation matrix, tilt correction and deformation recovery are achieved. By utilizing edge detection and connected component analysis, the effective answer area is automatically located, and background interference and redundant information are removed.

3. The method for recognizing the content of complex-format test papers according to claim 1, characterized in that: The MoE-LoRA high-efficiency fine-tuning steps also include: Load balancing: Add load balancing loss to routing loss; Hallucination suppression: Enhances expert differentiation, making routing choices more explicit.

4. The method for recognizing the content of complex-format test papers according to claim 1, characterized in that: The attention enhancement steps specifically include: The output of the feedforward neural network of the large language model in the visual language large model processing step is used as the query vector, and the expert features output in the MoE-LoRA efficient fine-tuning step are used as the key vector and value vector. The similarity between the query vector and the key vector is calculated by scaling the dot product attention, and then the similarity is summed with the value vector after scaling and normalization. A multi-head attention mechanism is used to compute multiple attention heads in parallel, and the outputs of each head are concatenated. Residual connections and layer normalization are used to achieve cross-layer information transfer and stable feature distribution.

5. The method for recognizing the content of complex-format test papers according to claim 4, characterized in that: The attention enhancement steps also include a hallucination suppression gating mechanism: During the training phase, the gating parameters can be learned, and the contribution of each attention head can be reduced through backpropagation; During the inference phase, a pre-computed illusion head mask is loaded, and the contribution weight of the attention head that generates the illusion is reset to zero.

6. The method for recognizing the content of complex-format test papers according to claim 1, characterized in that: The multi-task parsing steps specifically include: User intent is classified and task type is identified using a lightweight BERT model; Break down the complex intent into a sequence of atomic tasks; Generate a routing plan based on the mapping table from task tokens to expert groups and then to specific experts; The system uses three levels of prompt templates—system-level, task-level, and instance-level—to constrain the output format to be structured data.

7. A complex-format test paper content recognition system, characterized in that: include: The data preprocessing module is used to enhance and standardize the original test paper image to generate a preprocessed image; The visual language large model processing module is used to input the pre-processed image into the pre-trained visual language large model with frozen parameters and extract high-level semantic features. The visual language large model includes a visual encoder, a projection layer and a large language model. The MoE-LoRA high-efficiency fine-tuning module is used to efficiently fine-tune the parameters of the visual language large model based on the high-level semantic features extracted by the visual language large model processing module, and output expert features through a low-rank adapter network of a hybrid expert architecture. The hybrid expert architecture includes multiple low-rank adapter experts configured for different question types, and achieves input-adaptive expert selection through a learnable gating network. In this module, multiple low-rank adapter experts are configured for different question types, and different low-rank dimensions are set for experts of different question types. The learnable gated network employs a Top-K sparse activation strategy to balance computational efficiency and model capacity; A two-layer routing structure of group-level routers and expert-level routers is adopted. First, the question types are classified into groups, and then the K most relevant experts are selected within each group. This module employs a three-stage fine-tuning training strategy: Phase A: Single-expert warm-up training, freezing the pre-trained visual language large model, routing network and attention-related parameters; each time only a single LoRA expert and its corresponding question type data are selected for independent training, so that each expert can initially learn the visual features and answer distribution of the corresponding question type. Each expert is trained independently for 3-5 epochs to avoid mutual interference between different question type features. Phase B: Router network training, freezing the base visual language large model; freezing all parameters of LoRA experts; training group-level routers and expert-level routers, introducing load balancing loss to avoid expert bias or idleness; under the premise of being aware of expert characteristics, the routing network learns the mapping relationship of "input question type characteristics → expert group → specific expert" to establish a stable expert selection mechanism. Phase C: Fine-tuning of MoE and attention, freezing all original parameters of the base visual language large model; unfreezing all LoRA expert parameters, routing network parameters, attention projection matrix and illusion suppression gating parameters, and performing end-to-end joint training; The attention enhancement module is used to take the output of the feedforward neural network of the large language model in the visual language large model processing module as the query vector, and the expert features output by the MoE-LoRA high-efficiency fine-tuning module as the key vector and value vector. It calculates the attention weight through a cross-modal attention mechanism to realize the dynamic selective fusion of general knowledge and expert knowledge and generate enhanced features. The multi-task parsing module is used to parse the natural language instructions input by the user based on the enhanced features generated by the attention enhancement module, identify the task type, and generate structured output according to the task type.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images