Answer sheet ocr recognition method and system based on large model detection and character segmentation

Abstract

This invention relates to the field of optical character recognition technology, specifically to a large-model OCR recognition method and system for answer sheets based on detection and character segmentation. The method includes: acquiring answer sheet images and constructing a fine-tuning dataset; fine-tuning a multimodal large model based on this dataset; deploying the fine-tuned model to an edge device; performing lightweight text detection to obtain the text box region for each line; segmenting excessively wide text lines by character spacing, using a vertical projection algorithm to detect long blank areas between characters and dynamically determining the segmentation points; stitching the segmented sub-images together in high resolution according to their original order to generate image fragments that meet the fixed input size of the edge device; performing OCR recognition using the large model and rearranging the character-level recognition results according to their original positions. This invention achieves efficient processing of high-resolution answer sheet images using a large model on an edge device, resolving the contradiction between limited video memory and loss of semantic information, and significantly improving OCR recognition speed and accuracy.

Description

Technical Field

[0001] This invention relates to the field of optical character recognition technology, specifically to a large-scale OCR recognition method and system for answer sheets based on detection and character segmentation. Background Technology

[0002] Optical character recognition (OCR) technology based on multimodal large models significantly outperforms traditional OCR methods in recognition accuracy in complex scenarios. However, in practical applications such as K-12 education and personnel examinations, answer sheet OCR recognition faces the following technical challenges:

[0003] On the one hand, the handwritten fonts on answer sheets vary greatly, with diverse writing styles and special phenomena such as cursive writing, erasures, and blurring. General-purpose OCR models rely too heavily on training data, making it difficult for them to meet the needs of practical applications. While multimodal large-scale models can effectively solve these problems, their massive number of parameters and computational complexity place extremely high demands on hardware resources.

[0004] On the other hand, deploying large models on many edge devices faces severe resource constraints: limited computing power and small memory capacity, insufficient to support direct input of high-resolution images. Large models typically require fixed-size input; directly resizing or cutting high-resolution answer sheet images into fixed windows can lead to character fragmentation and loss of semantic information, resulting in a significant decrease in recognition accuracy. Existing technologies often employ simple uniform segmentation strategies to address such issues, failing to adaptively adjust based on the actual distribution of the text content, easily causing characters to be incorrectly segmented and affecting the final recognition result.

[0005] Therefore, how to fully leverage the OCR capabilities of large models on resource-constrained edge devices while avoiding information loss during image preprocessing has become a key technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention provides a large-scale OCR recognition method and system for answer sheets based on detection and character segmentation. By employing a preprocessing strategy that combines lightweight text detection with character interval segmentation, it achieves efficient processing of high-resolution answer sheet images on edge devices, significantly improving OCR recognition speed and accuracy.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] On one hand, embodiments of the present invention provide a large-scale OCR recognition method for answer sheets based on detection and character segmentation, the method comprising the following steps:

[0009] Obtain the answer sheet image dataset and construct a fine-tuning dataset for OCR recognition. Each sample in the dataset contains the answer sheet image path and the corresponding text content annotation information.

[0010] Based on the fine-tuning dataset, the multimodal large model is fine-tuned to obtain an OCR recognition model specifically for answer sheets. The OCR recognition model is then converted into a format supported by edge devices and deployed to edge devices.

[0011] Lightweight text detection is performed on the input answer sheet image to locate the text box region where each line of text content is located in the answer sheet;

[0012] For text box areas whose width exceeds the input size limit of the edge device, perform character spacing segmentation to obtain several sub-images;

[0013] The sub-images are stitched together in high resolution according to the original text order to generate an image fragment that conforms to the fixed input size of the edge device.

[0014] The image fragment is input into an OCR recognition model deployed on an edge device for OCR recognition to obtain character-level recognition results.

[0015] Based on the original character position information recorded by the character segmentation points, the character-level recognition results are rearranged to restore the output results to the same order as the original text on the answer sheet.

[0016] Optionally, the construction of the fine-tuning dataset for OCR recognition includes:

[0017] A multimodal large model is used to pre-annotate the answer sheet image, and the API interface of the large model is called to obtain the OCR recognition pre-annotation result;

[0018] The pre-annotated results are manually verified and corrected to ensure that the text content accurately corresponds to the answer information in the image.

[0019] Each line of text is labeled as an independent text detection box, and a newline character is added after each line of text to obtain a fine-tuned dataset for OCR recognition.

[0020] Optionally, the step of fine-tuning the multimodal large model based on the fine-tuning dataset to obtain an OCR recognition model specifically for answer sheets includes:

[0021] A multimodal large model that meets the computing power and memory constraints of edge devices is selected as the basic model.

[0022] The LoRA technique is used to fine-tune the base model, injecting trainable parameters only into the attention layer and fully connected layer of the base model;

[0023] The fine-tuned dataset is input into the base model, and the model is trained using preset training parameters until the loss function converges to a preset threshold, thus obtaining an OCR recognition model specifically for answer sheets.

[0024] Optionally, performing lightweight text detection on the input answer sheet image to locate the text box region where each line of text content in the answer sheet is located includes:

[0025] The lightweight detection model was fine-tuned using the aforementioned fine-tuning dataset to obtain a lightweight text detection model.

[0026] A lightweight text detection model is used to process the input answer sheet image and output the rectangular bounding box region of each line of text.

[0027] Optionally, for text box areas whose width exceeds the input size limit of the edge device, a character spacing segmentation operation is performed to obtain several sub-images, including:

[0028] Image preprocessing is performed on the cropped text line image. After converting the text line image into a grayscale image, a binary image is generated using a local window adaptive binarization algorithm.

[0029] Calculate the vertical projection value of the binary image, and perform median filtering on the vertical projection value;

[0030] The gap threshold is calculated based on the mean and standard deviation of the vertical projection values, and projection values ​​smaller than the gap threshold are marked as blank segments.

[0031] Detect consecutive blank segment intervals and filter out consecutive blank segments with a width not less than the minimum blank width, where the minimum blank width is the larger value between a set pixel value and a set ratio of text line width;

[0032] Candidate segmentation points are set at the midpoint of the continuous blank segment interval. Segmentation points with a spacing between adjacent segmentation points not less than a set proportion of the average width of the characters are retained. The text line image is divided into several sub-images based on the segmentation points.

[0033] Optionally, the step of stitching the sub-images together at high resolution according to the original text order to generate an image fragment that conforms to the fixed input size of the edge device includes:

[0034] Iterate through the subgraphs after each line of text is segmented, accumulate the subgraph widths, and when the accumulated width exceeds the input width of the edge device, stitch the preceding subgraphs into a line fragment and record the subgraph index range contained in the line fragment;

[0035] Traverse all line segments in the original order of the answer sheet, accumulate the segment height, and when the accumulated height exceeds the input height of the edge device, vertically stitch the preceding line segments into an image segment. The height of the image segment does not exceed the input height of the edge device.

[0036] Establish a global position table to record the row segment ID to which each subgraph belongs, the starting x-coordinate and the starting y-coordinate within the row segment.

[0037] Optionally, the step of rearranging the character-level recognition results based on the original character position information recorded by the character segmentation points to restore the output result to the same order as the original text on the answer sheet includes:

[0038] Receive character-level recognition results of image segments, wherein the recognition results include characters and confidence levels of sub-image sequences;

[0039] Based on the global location table, the sub-image recognition results are mapped back to the original text lines, and the sub-image results of the same text line are sorted according to the starting x-coordinate;

[0040] Process the boundary regions of adjacent subgraphs. When characters are recognized on both sides of the boundary, select the one with higher confidence. When no characters are recognized on both sides of the boundary, treat it as blank and obtain the rearranged text.

[0041] The rearranged text lines are corrected, and character replacement is performed when the recognition confidence is lower than the confidence threshold and the language model probability is higher, outputting text results that are consistent with the original layout of the answer sheet.

[0042] On the other hand, embodiments of the present invention provide a large-scale OCR recognition system for answer sheets based on detection and character segmentation, comprising:

[0043] At least one processor;

[0044] At least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0046] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0047] The beneficial effects of this invention are as follows: This invention discloses a large-model OCR recognition method and system for answer sheets based on detection and character segmentation. Through a preprocessing strategy combining lightweight text detection and character interval segmentation, it effectively solves the problems of character fragmentation and semantic loss caused by input size limitations in high-resolution answer sheet images on edge devices. Specifically, firstly, a lightweight text detection model is used to accurately locate the text region of each line, ensuring the targeted nature of subsequent processing. Next, for wide text lines, a character interval segmentation algorithm based on vertical projection and adaptive thresholds can intelligently segment characters at gaps, avoiding erroneous truncation and preserving character integrity to the greatest extent. Then, through a high-resolution sub-image stitching strategy, the segmented sub-images are recombined into image fragments that conform to the input size of the edge device, ensuring both the resolution of the input image and full utilization of the edge device's computing resources. Finally, by leveraging a global position table and a character-level recognition result rearrangement mechanism, combined with correction using a 5-gram language model, the recognition result is accurately restored to the original text order of the answer sheet, significantly improving the OCR recognition accuracy in complex handwriting scenarios. Meanwhile, LoRA technology is used to perform lightweight fine-tuning of the large multimodal model, and combined with a lightweight detection model, effectively reducing the computational complexity and memory consumption of the model on edge devices, ensuring the efficiency and real-time performance of the recognition process. This invention effectively solves the problems of character fragmentation and semantic loss caused by fixed-scale cropping, providing a reliable technical solution for automated grading of answer sheets in scenarios such as K12 education and qualification examinations. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the overall process of a large-scale OCR recognition method for answer sheets based on detection and character segmentation, provided in this embodiment of the invention.

[0050] Figure 2 This is a schematic diagram of the lightweight text detection effect provided by an embodiment of the present invention, showing the detection box localization result of each line of text region in the answer sheet image;

[0051] Figure 3 This is a schematic diagram of the character spacing effect provided in an embodiment of the present invention, illustrating the process of detecting character gaps based on vertical projection and performing dynamic segmentation. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of the present invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0053] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts herein, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0054] The terms "at least one", "multiple", "each", "any", etc., used in this invention, "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiple, and "any" refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0056] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0057] The core technical problem that this invention aims to solve is: how to achieve efficient OCR recognition of high-resolution answer sheet images by a large model on edge devices with severely limited computing power and memory, while avoiding the loss of semantic information caused by image preprocessing.

[0058] To address this, this invention designs an adaptive image preprocessing strategy based on text detection. First, a lightweight detection method is performed on the entire answer sheet image to locate the answer area in each row. A character-space-based segmentation algorithm is proposed, which uses vertical projection to statistically analyze character region density to automatically detect long blank areas between characters, and then performs dynamic segmentation. This effectively solves the problems of character breakage and semantic loss caused by fixed-size window segmentation.

[0059] Based on the sub-images segmented by character intervals, local high-resolution stitching is performed, and the images are stitched together in the horizontal and vertical directions to form a fixed input size image required for large model inference on edge devices. This can greatly improve the inference speed and effectively solve the problem of not being able to input ultra-large resolution images.

[0060] refer to Figure 1 ,like Figure 1 The figure shown is a large-scale OCR recognition method for answer sheets based on detection and character segmentation provided by an embodiment of the present invention. The method includes the following steps:

[0061] S100, Obtain the answer sheet image dataset and construct a fine-tuning dataset for OCR recognition. Each sample in the dataset contains the answer sheet image path and the corresponding text content annotation information.

[0062] S200, Based on the fine-tuning dataset, the multimodal large model is fine-tuned to obtain an OCR recognition model specifically for answer sheets. The OCR recognition model is then converted into a format supported by edge devices and deployed to edge devices.

[0063] S300 performs lightweight text detection on the input answer sheet image to locate the text box area where each line of text content in the answer sheet is located;

[0064] S400 performs character spacing segmentation on text box areas whose width exceeds the input size limit of the edge device, resulting in several sub-images;

[0065] S500, the sub-images are stitched together in high resolution according to the original text order to generate an image fragment that conforms to the fixed input size of the edge device;

[0066] S600, the image fragment is input into the OCR recognition model deployed in the edge device for OCR recognition to obtain character-level recognition results;

[0067] S700, based on the original character position information recorded by the character segmentation point, rearrange the character-level recognition result to restore the output result to the same order as the original text on the answer sheet.

[0068] The core innovation of this invention lies in the organic combination of text detection and character segmentation, achieving adaptive processing from the document level to the character level. Lightweight detection ensures accurate positioning of text lines and avoids interference from invalid regions; character spacing segmentation preserves character integrity; intelligent splicing maximizes the preservation of image details while meeting hardware constraints; and character-level recognition result rearrangement ensures the semantic coherence of the final output.

[0069] In some embodiments, S100, constructing the fine-tuning dataset for OCR recognition includes:

[0070] S110, a multimodal large model is used to pre-annotate the answer sheet image, and the API interface of the large model is called to obtain the OCR recognition pre-annotation result;

[0071] S120, The pre-labeling results are manually verified and corrected to ensure that the text content and the answer information in the image correspond accurately;

[0072] S130, each line of text is labeled as an independent text detection box, and a newline character is added after each line of text to obtain a fine-tuned dataset for OCR recognition.

[0073] This embodiment reduces the workload of manual annotation by leveraging the pre-annotation capabilities of a multimodal large model. Meanwhile, manual verification and correction ensure the accuracy of the labeled data. The constructed fine-tuning dataset not only includes image paths but also the coordinate information of each text detection box and the corresponding text content annotation, laying a solid data foundation for subsequent model fine-tuning training.

[0074] In some embodiments, S200, the step of fine-tuning the multimodal large model based on the fine-tuning dataset to obtain an OCR recognition model specifically for answer sheets includes:

[0075] S210 selects a multimodal large model that meets the computing power and memory constraints of edge devices as the basic model;

[0076] S220, the LoRA technique is used to fine-tune the base model, injecting trainable parameters only into the attention layer and fully connected layer of the base model;

[0077] S230, the fine-tuned dataset is input into the base model, and the model is trained using preset training parameters until the loss function converges to a preset threshold, thus obtaining an OCR recognition model specifically for answer sheets.

[0078] This embodiment achieves lightweight fine-tuning of a large model through LoRA technology, injecting only a small number of trainable parameters into key modules such as the attention layer and fully connected layer. While significantly reducing training costs and computational resource consumption, it can accurately adapt to the feature distribution of handwritten characters on the answer sheet.

[0079] In some embodiments, S300, performing lightweight text detection on the input answer sheet image and locating the text box region where each line of text content in the answer sheet is located includes:

[0080] S310, The lightweight detection model is fine-tuned using the fine-tuning dataset to obtain a lightweight text detection model;

[0081] S320 uses a lightweight text detection model to process the input answer sheet image and outputs the rectangular box area of ​​each line of text.

[0082] This embodiment effectively reduces the computational burden on edge devices during image preprocessing by introducing a lightweight text detection model, providing a precise regional range for subsequent character segmentation.

[0083] refer to Figure 2 , Figure 2 To demonstrate the lightweight detection performance, the model can accurately locate the answer area for each line on the answer sheet. Even with slight tilting or local shadows, the detection box can still closely fit the text line.

[0084] In some embodiments, in S400, the character spacing segmentation operation is performed on the text box area whose width exceeds the input size limit of the edge device to obtain several sub-images, including:

[0085] S410, perform image preprocessing on the cropped text line image, convert the text line image into a grayscale image, and then use a local window adaptive binarization algorithm to generate a binary image;

[0086] S420, Calculate the vertical projection value of the binary image, and perform median filtering on the vertical projection value;

[0087] S430, calculate the gap threshold based on the mean and standard deviation of the vertical projection values, and mark the projection values ​​that are less than the gap threshold as blank segments;

[0088] S440, detect continuous blank segment intervals and filter out continuous blank segments with a width not less than the minimum blank width, wherein the minimum blank width is the larger value between the set pixel and the set ratio of the text line width;

[0089] S450, a candidate segmentation point is set at the midpoint of the continuous blank segment interval, and a segmentation point with a spacing between adjacent segmentation points not less than a set ratio of the average width of the characters is retained. The text line image is divided into several sub-images according to the segmentation points.

[0090] The detailed implementation steps of the character spacing algorithm are as follows:

[0091] Step 1: Image preprocessing. The cropped text line image I(x,y) is first converted to grayscale. Then, an adaptive binarization algorithm is used, with the local window size set to 31×31, to obtain the binary image G(x,y). This adaptive method can better handle uneven illumination problems compared to global thresholding.

[0092] Specifically, let the input character region image be I(x,y), the width of the image be W, and the height be H.

[0093] ;

[0094] If a pixel value is greater than the threshold T, it is determined to be a foreground area and the pixel belongs to a character; otherwise, it is a background area.

[0095] Step 2: Vertical projection calculation. For each column x of the binary image, obtain the vertical projection function V(x): , Where V(x) represents the number of pixels of the character in the x-th column.

[0096] To enhance robustness, median filtering is applied to V(x) with a window size of 3.

[0097] Step 3: Adaptive threshold calculation. Calculate the projected mean. and standard deviation To obtain the gap threshold , As an adaptive threshold based on mean and variance, this setting can effectively distinguish between character gaps and internal whitespace within characters.

[0098] Step 4: Detect consecutive blank segments.

[0099] Calculate the gap indicator Z(x) when the column pixel density is below a certain threshold. The column is considered blank at that time.

[0100] ;

[0101] Detect the interval [s,e] (starting point s, ending point e, including the endpoint) on Z(x) where the interval is consecutively 1, and the blank width w = e - s + 1. ,in This dynamic threshold prevents overly wide characters from being incorrectly segmented.

[0102] Step 5: Screening of split points.

[0103] The center of the gap is selected as the dividing point. The dividing point is This yields the set of center point divisions for the continuous blank regions. Then, based on the set minimum cutting distance threshold Perform cutting, when the conditions are met hour, This is a valid split point. Ultimately, points satisfying this condition are retained. The segmentation point is ε = 0.8 × (average character width), and the average character width can be estimated from the width of the first 10 characters.

[0104] refer to Figure 3 , Figure 3 The algorithm accurately sets the splitting points at the character gaps, without misinterpreting the spaces inside the characters as splitting positions.

[0105] In some embodiments, S500, the step of stitching the sub-images together at high resolution according to the original text order to generate an image fragment conforming to the fixed input size of the edge device includes:

[0106] S510: Traverse the subgraphs after each line of text is segmented, accumulate the subgraph width, and when the accumulated width exceeds the input width of the edge device, stitch the previous subgraphs into a line fragment and record the subgraph index range contained in the line fragment.

[0107] S520: Traverse all row segments in the original order of the answer sheet, accumulate the segment height, and when the accumulated height exceeds the input height of the edge device, vertically splice the previous row segments into an image segment. The height of the image segment does not exceed the input height of the edge device.

[0108] S530, establish a global position table to record the row segment ID to which each subgraph belongs, the starting x-coordinate and the starting y-coordinate within the row segment.

[0109] The implementation logic of the image stitching strategy is as follows:

[0110] Horizontal splicing: Assume the input width of the edge device is W=448. For each line of text, traverse the segmented sub-images from left to right, and accumulate the sub-image widths. .when When the first i-1 subgraphs are joined together, they form a single row segment. its width .Record The range of subgraph indices included. Repeat this process for the remaining subgraphs until all subgraphs in the row have been processed.

[0111] Vertical splicing: Assume the input height H = 448. Traverse all line segments in the original order of the answer sheet. Accumulated segment height .when At that time, the first j-1 rows of segments are vertically spliced ​​together to form the final image segment. Its dimensions are W×H', where ≤H. (This likely refers to a specific value or parameter, but without further context, it's difficult to translate accurately.) It is then fed into a large model for recognition.

[0112] Location information recording: A global location table is established to record the row segment ID to which each subgraph belongs, and the starting x and y coordinates within the row segment. This table is used for result rearrangement in S700.

[0113] In some embodiments, S700, the step of rearranging the character-level recognition result based on the original character position information recorded by the character segmentation points to restore the output result to the same order as the original text on the answer sheet includes:

[0114] S710, receive the character-level recognition result of the image segment, the recognition result including the characters and confidence level of the sub-image sequence;

[0115] S720: Based on the global location table, the sub-image recognition results are mapped back to the original text line, and the sub-image results of the same text line are sorted according to the starting x-coordinate;

[0116] S730 processes the boundary region of adjacent sub-images. When characters are recognized on both sides of the boundary, the one with higher confidence is selected. When there are no recognition results on both sides of the boundary, it is regarded as blank, and the rearranged text is obtained.

[0117] S740 corrects the rearranged text lines. When the recognition confidence is lower than the confidence threshold and the language model probability is higher, it performs character replacement and outputs a text result consistent with the original layout of the answer sheet.

[0118] The specific process for result rearrangement and post-processing is as follows:

[0119] Step 1: Receive the large model for each image segment Recognition results , The characters and confidence levels of the subgraph sequence are included.

[0120] Step 2: Based on the global location table, The subplot results are mapped back to the original text line. For multiple subplot results of the same text line, they are sorted by the starting x-coordinate.

[0121] Step 3: For the boundary region (2 pixels wide) between adjacent sub-images, if characters are identified on both sides, select the one with higher confidence; if no identification results are found on either side, it is considered blank.

[0122] Step 4: Perform language model correction on the rearranged text lines. Use a 5-gram language model to detect common errors (such as confusion between "的 / 地 / 得"), and replace when the recognition confidence is lower than 0.8 and the language model probability is significantly higher.

[0123] The final output format is consistent with the original layout of the answer sheet. Each line of text corresponds to a line of recognition result, maintaining the line break markers.

[0124] The following is a specific embodiment provided by the present invention:

[0125] S1. Start;

[0126] S2. Prepare a fine-tuning dataset for OCR recognition. In the jsonl file, the question-and-answer correspondence includes the image path and the text content corresponding to the image. A line break needs to be added after each line of text.

[0127] Among them, when preparing the dataset for large model OCR recognition, a multi-modal large model is used to pre-label the answer sheet images. The api of the multi-modal large model is called to perform OCR recognition on the answer sheet images, and a pre-labeling result with higher accuracy is obtained. Then, the pre-labeling result is manually modified to correct the problematic recognition results. Using a large model for pre-labeling can greatly shorten the time and labor cost of OCR annotation.

[0128] S3. Fine-tune the multi-modal large model using the prepared dataset. The selected multi-modal large model has a computing power requirement and model size for inference that just meet the deployment requirements on edge devices, and can also meet the OCR recognition effect of the answer sheet. Train for 1 epoch, set max_length to 4096, and use the sft lora fine-tuning method.

[0129] S4. Merge the fine-tuned model with the pre-trained model, while improving the OCR recognition ability of the large model for answer sheets and ensuring the generality of the large model.

[0130] S5. Next, deploy the fine-tuned model on edge devices, and convert the visual part and language part of the merged model into the model formats supported by the edge devices respectively.

[0131] When the visual model performs visual feature encoding, due to the memory limitation of the edge device, a fixed input image size is set , and 200 answer sheet images covering most scenarios need to be prepared for quantization when converting the language model.

[0132] S6. Perform large model OCR inference on the edge device. Perform visual feature encoding through the visual model, and then call the language model for text analysis and generation.

[0133] The answer sheet images input during visual encoding have a large resolution and cannot be directly recognized by OCR. The input answer sheet images need to be cut into a fixed input size supported by the edge device during inference.

[0134] Specifically, lightweight text detection is first performed to detect the text box positions of each line of content in the answer sheet. The lightweight detection model is fine-tuned using labeled answer sheet data, which is labeled with a text detection box for each line of text content.

[0135] Next, the text detection bounding boxes obtained from text detection are cropped for each line on the answer sheet. Since the width of some text detection bounding boxes may exceed the fixed input size supported by inference on edge devices, character-spaced segmentation is required for the cropped sub-images.

[0136] After character segmentation, the segmented sub-image regions are re-stitched horizontally according to their left-right order within the original detection box, resulting in multiple row segments of fixed size. The horizontal length of the stitched image is then considered. Apply upper limit constraints: ;in Fixed dimensions for inputting the visual model on the edge device.

[0137] If the text detection box in the above row does not exceed [a certain value], then [the text detection box] is not larger than [a certain value]. If the text box is not segmented, then character splitting is unnecessary; the entire line of text is treated as a single line fragment. Subsequently, all line fragments are vertically concatenated according to the original document layout. During vertical concatenation, if the height of the concatenated area exceeds H pixels, the current concatenated area is output as a final image fragment, and the construction of the next concatenated area begins anew. This process ultimately generates several images that meet the fixed input size. The image fragments are then fed into the large-scale OCR recognition module for inference.

[0138] Finally, for image segments with character segmentation, the recognition results obtained through reasoning are rearranged according to the original character position order recorded during character segmentation, thereby restoring the character-level recognition output to the true text order in the original image.

[0139] Compared with related technologies, the present invention has the following advantages:

[0140] This invention proposes a method and system for OCR recognition of large-scale answer sheets based on detection and character segmentation. It effectively solves the problem of large-resolution image size limitation by using lightweight text detection, character spacing segmentation and dynamic text box recombination, while also improving the speed of large-scale OCR recognition.

[0141] This invention proposes an adaptive image preprocessing method based on text detection. A lightweight text detection model, fine-tuned line by line, is used to adaptively locate the answer sheet region, providing text position information for subsequent character spacing segmentation and recognition result rearrangement.

[0142] This invention designs a vertical projection segmentation algorithm based on character spacing to detect continuous blank regions between characters. It achieves lossless character segmentation and performs high-resolution local stitching on the segmented sub-images to construct a large-scale model inference image that conforms to a fixed input size. This method improves inference speed and enhances the accuracy of character segmentation and OCR recognition without relying on high-resolution input.

[0143] This invention also provides a large-scale OCR recognition system for answer sheets based on detection and character segmentation, comprising:

[0144] At least one processor;

[0145] At least one memory for storing at least one program;

[0146] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0147] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0148] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be a server, an edge computing device, a smart camera, a scanner all-in-one machine, or a dedicated OCR review terminal.

[0149] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0150] Those skilled in the art will understand that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A large-scale OCR recognition method for answer sheets based on detection and character segmentation, characterized in that, The method includes the following steps: Obtain the answer sheet image dataset and construct a fine-tuning dataset for OCR recognition. Each sample in the fine-tuning dataset contains the answer sheet image path and the corresponding text content annotation information. Based on the fine-tuning dataset, the multimodal large model is fine-tuned to obtain an OCR recognition model specifically for answer sheets. The OCR recognition model is then converted into a format supported by edge devices and deployed to edge devices. Lightweight text detection is performed on the input answer sheet image to locate the text box region where each line of text content is located in the answer sheet; For text box areas whose width exceeds the input size limit of the edge device, perform character spacing segmentation to obtain several sub-images; The sub-images are stitched together in high resolution according to the original text order to generate an image fragment that conforms to the fixed input size of the edge device. The image fragment is input into an OCR recognition model deployed on an edge device for OCR recognition to obtain character-level recognition results. Based on the original character position information recorded by the character segmentation points, the character-level recognition results are rearranged to restore the output results to the same order as the original text on the answer sheet.

2. The method according to claim 1, characterized in that, The fine-tuned dataset for OCR recognition includes: A multimodal large model is used to pre-annotate the answer sheet image, and the API interface of the large model is called to obtain the OCR recognition pre-annotation result; The pre-annotated results are manually verified and corrected to ensure that the text content accurately corresponds to the answer information in the image. Each line of text is labeled as an independent text detection box, and a newline character is added after each line of text to obtain a fine-tuned dataset for OCR recognition.

3. The method according to claim 1, characterized in that, The process of fine-tuning the multimodal large model based on the fine-tuned dataset to obtain a dedicated OCR recognition model for answer sheets includes: A multimodal large model that meets the computing power and memory constraints of edge devices is selected as the basic model; The LoRA technique is used to fine-tune the base model, injecting trainable parameters only into the attention layer and fully connected layer of the base model; The fine-tuned dataset is input into the base model, and the model is trained using preset training parameters until the loss function converges to a preset threshold, thus obtaining an OCR recognition model specifically for answer sheets.

4. The method according to claim 1, characterized in that, The step of performing lightweight text detection on the input answer sheet image to locate the text box region containing each line of text in the answer sheet includes: The lightweight detection model was fine-tuned using the aforementioned fine-tuning dataset to obtain a lightweight text detection model. A lightweight text detection model is used to process the input answer sheet image and output the rectangular bounding box region of each line of text.

5. The method according to claim 1, characterized in that, For text box areas whose width exceeds the input size limit of the edge device, a character spacing segmentation operation is performed to obtain several sub-images, including: The text box regions obtained from text detection are cropped, and for each line on the answer sheet, the cropped text line image is obtained; Image preprocessing is performed on the cropped text line image. After converting the text line image into a grayscale image, a binary image is generated using a local window adaptive binarization algorithm. Calculate the vertical projection value of the binary image, and perform median filtering on the vertical projection value; The gap threshold is calculated based on the mean and standard deviation of the vertical projection values, and projection values ​​smaller than the gap threshold are marked as blank segments. Detect consecutive blank segment intervals and filter out consecutive blank segments with a width not less than the minimum blank width, where the minimum blank width is the larger value between a set pixel value and a set ratio of text line width; Candidate segmentation points are set at the midpoint of the continuous blank segment interval. Segmentation points with a spacing between adjacent segmentation points not less than a set proportion of the average width of the characters are retained. The text line image is divided into several sub-images based on the segmentation points.

6. The method according to claim 1, characterized in that, The step of stitching the sub-images together at high resolution according to the original text order to generate an image fragment that conforms to the fixed input size of the edge device includes: Iterate through the subgraphs after each line of text is segmented, accumulate the width of the subgraphs, and when the accumulated width exceeds the input width of the edge device, stitch the preceding subgraphs into a line fragment and record the subgraph index range contained in the line fragment; Traverse all line segments in the original order of the answer sheet, accumulate the segment height, and when the accumulated height exceeds the input height of the edge device, vertically stitch the preceding line segments into an image segment. The height of the image segment does not exceed the input height of the edge device. Establish a global position table to record the row segment ID to which each subgraph belongs, the starting x-coordinate and the starting y-coordinate within the row segment.

7. The method according to claim 1, characterized in that, The process of rearranging the character-level recognition results based on the original character position information recorded by the character segmentation points to restore the output result to the same order as the original text on the answer sheet includes: Receive character-level recognition results of image segments, wherein the recognition results include characters and confidence levels of sub-image sequences; Based on the global location table, the sub-image recognition results are mapped back to the original text lines, and the sub-image results of the same text line are sorted according to the starting x-coordinate; Process the boundary regions of adjacent subgraphs. When characters are recognized on both sides of the boundary, select the one with higher confidence. When no characters are recognized on both sides of the boundary, treat it as blank and obtain the rearranged text. The rearranged text lines are corrected, and character replacement is performed when the recognition confidence is lower than the confidence threshold and the language model probability is higher, outputting text results consistent with the original layout of the answer sheet.

8. A large-scale OCR recognition system for answer sheets based on detection and character segmentation, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images