An end-cloud collaborative handwriting document intelligent recognition method and system

Abstract

This invention discloses a terminal-cloud collaborative intelligent handwritten document recognition method and system, comprising the following steps: Step 1: Acquire the original image of the handwritten document and transmit the original image to the terminal; Step 2: The original image is processed locally by a lightweight handwritten recognition model on the terminal side, and the structured data of the locally processed recognition is uploaded to the cloud; Step 3: The cloud side performs cloud recognition and calibration on the uploaded locally processed structured data through a cloud-side large model handwritten recognition collaborative correction method; Step 4: The structured data of the locally processed recognition and the cloud recognition and calibration data are fused to output the document recognition result and are sent back to the terminal in JSON format. This invention constructs a novel recognition system architecture in which a lightweight terminal model and a large cloud model work collaboratively, realizing optimized allocation of computing resources and intelligent allocation of recognition tasks.

Description

Technical Field

[0001] This invention belongs to the field of education and relates to a method and system for intelligent recognition of handwritten documents through edge-cloud collaboration. Background Technology

[0002] In the context of the deep integration of artificial intelligence technology into the education field, large-scale model-driven homework grading systems are gradually becoming important tools for improving teaching efficiency and achieving precision teaching. However, facing the needs of recognizing and grading students' handwritten homework, large-scale model solutions deployed purely in the cloud or locally have the following problems: First, handwriting recognition requires high model accuracy, especially in complex scenarios such as Chinese cursive writing, messy handwriting, and formula symbols. If all recognition tasks are handled by large-scale models in the cloud, it will lead to a shortage of computing resources and significant response delays under high concurrency. Second, student homework contains a large amount of personal learning data, involving the privacy of minors. Uploading the original document images to the cloud poses a risk of privacy and security leaks. Third, local deployment of large-scale models requires high-performance GPU servers, which are costly and difficult to popularize. Fourth, conventional large-scale model handwriting recognition relies on computing resources and takes a long time. Conventional OCR recognition models are unable to understand deep semantics such as mathematical derivations, essay logic, and subjective question scoring standards, and must rely on the cognitive capabilities of large-scale models. This invention effectively solves the above problems. Summary of the Invention

[0003] In order to overcome at least one deficiency of the prior art, the present invention provides a method and system for intelligent recognition of handwritten documents in a cloud-edge collaborative manner.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: a method for intelligent recognition of handwritten documents through end-to-end cloud collaboration, comprising the following steps: Step 1: Capture the original image of the handwritten document and transmit the original image to the terminal; Step 2: The original image is processed locally using the lightweight handwriting recognition model and handwriting recognition method on the terminal side, and the locally processed structured data is uploaded to the cloud. Step 3: The structured data uploaded from local recognition and processing is then recognized and calibrated in the cloud using a cloud-based large-model handwritten character recognition collaborative correction method; Step 4: Integrate the structured data processed locally with the cloud-based recognition and calibration data to output the document recognition results and send them back to the terminal in JSON format.

[0005] Furthermore, the terminal-side lightweight handwriting recognition model handwriting recognition method includes the following steps for localizing the original image: Step S1: Perform adaptive grayscale conversion and image noise reduction on the original image of the input handwritten document on the terminal; Step S2: Perform adaptive background normalization on the image processed in step S1 on the terminal; Step S3: Perform adaptive tilt correction on the image processed in step S2 on the terminal; Step S4: Perform text region detection on the image processed in step S3 based on the local lightweight OCR recognition model, obtain the location information of each text region, and initially classify the obtained text regions into 'printed' or 'handwritten', while processing overlapping regions and excessively small fragment regions; Step S5: Perform boundary point aggregation and convex hull processing on the detected text regions to extract regional geometric features; Step S6: Based on geometric features and OCR confidence, classify the text region into printed / handwritten characters and calculate the confidence of printed / handwritten character features; Step S7: Based on the preliminary classification results of printed / handwritten text, feature confidence and terminal network status, make a decision on whether to perform local recognition or upload to the cloud for recognition of all 'printed' and 'handwritten' text areas using an intelligent routing method; Step S8: After completing the local terminal identification process through the intelligent routing method, structured data for local terminal identification processing is generated.

[0006] Furthermore, step 3 involves recognizing and calibrating the received data using a cloud-based large-model handwritten character recognition collaborative correction method. Step 31: Local recognition and processing of structured data output. All data containing "cloud recognition request" is recognized by the large model on the cloud side; Step 32: Output the structured data from the local recognition process. All 'handwritten' handwriting types that have been locally recognized and identified and whose confidence level is higher than the preset handwriting confidence threshold are then recognized and corrected by the large model on the cloud side.

[0007] Furthermore, step 4 includes Step 41: Merge the uploaded locally processed structured data and cloud-based recognition and calibration data according to the unique identifier ID of the text region; Step 42: The cloud-based large model performs semantic correction on the recognition results based on contextual information; Step 43: Maintain the original document layout in the structured output; Step 44: Send the structured data back to the terminal in JSON format.

[0008] Furthermore, the adaptive grayscale and noise reduction processing in step S1 includes... Step S11: Select an optimized grayscale conversion method based on the image lighting conditions to convert the original RGB image into a grayscale image; Step S12: Denoising the grayscale image; The noise intensity of grayscale images is determined by Laplacian variance, and the noise is divided into three levels: high, medium, and low. High-noise images are denoised using the nonlocal mean-based denoising algorithm in OpenCV. Medium-noise images are denoised using the bilateral filtering algorithm in OpenCV. Low-noise images are preserved.

[0009] Furthermore, step S2, adaptive background normalization processing, includes... Step S20: Set the document classification type and the classification rules for the type; Document classification types include old paper document types, modern printed document types, and mixed handwritten document types; classification rules are set for each type to categorize documents; Step S21: Set the adaptive background normalization processing method according to the document classification type; Step S22: Calculate the background brightness, contrast, and edge density of the original image; Step S23: Classify the documents based on calculated background brightness, contrast, edge density, and type classification rules; Step S24: Perform adaptive background normalization on the image based on the document's classification type.

[0010] Furthermore, step S5 performs boundary point aggregation and convex hull processing on the detected text regions, and the method for extracting region geometric features includes... Step S51: Traverse each detection region; Step S52: Use the Douglas-Peucker algorithm to simplify the contours of all detection regions to obtain polygonal detection regions; Step S53: Convert the simplified polygon into a convex polygon; Step S54: Extract the convex hull and extract the smallest rotation rectangle that can enclose the convex hull; Step S55: Calculate the geometric feature parameters of the region based on the convex hull and the minimum rotation rectangle.

[0011] A handwritten document intelligent recognition system with edge-cloud collaboration includes a terminal-side system module and a cloud-side system module. The terminal-side system module includes a handwritten document image scanning module, an image processing module, a text recognition and classification module, a local recognition and processing module, and an edge-cloud collaborative communication module. The cloud-side system module includes a cloud recognition module and a cloud data storage module.

[0012] A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the aforementioned end-to-cloud collaborative intelligent recognition method for handwritten documents.

[0013] A computer program, when executed by a processor, is used to perform the aforementioned end-to-end cloud collaborative intelligent recognition method for handwritten documents.

[0014] In summary, the advantages of this invention are: This invention constructs a novel recognition system architecture that enables lightweight terminal models and large cloud-based models to work collaboratively, thereby achieving optimized allocation of computing resources and intelligent distribution of recognition tasks.

[0015] This invention constructs a dynamic routing decision mechanism: based on text type recognition results, confidence assessment, and network status monitoring, it dynamically determines the recognition execution location for each text region.

[0016] This invention avoids the leakage of original image data by uploading only necessary feature data and recognition results, thus meeting the privacy protection requirements of sensitive scenarios such as education and healthcare.

[0017] This invention uses a vision-language model based on the Transformer architecture as a cloud-based recognition model, which improves upon the shortcomings of existing solutions that lack semantic understanding of handwritten characters.

[0018] This invention achieves end-to-cloud collaboration by using a lightweight OCR recognition model for image text recognition on the terminal side and a Transformer architecture-based visual-language model for handwritten text recognition on the cloud side. This enables rapid processing of printed text on the local terminal and recognition of complex and difficult handwritten text on the cloud side. This not only improves the recognition efficiency for mixed documents but also reduces the hardware requirements for intelligent document recognition on the terminal, thus addressing the cost issue of existing terminals for intelligent document recognition.

[0019] The present invention provides a cloud-edge collaborative intelligent handwritten document recognition system that achieves high-precision and fast intelligent recognition of handwritten documents by integrating the functional model library of the terminal system and the cloud system.

[0020] The printed / handwritten text local recognition and intelligent routing method and system included in this invention, along with the end-to-cloud collaborative communication module, enhance network adaptability and system reliability.

[0021] This invention addresses document features that combine printed and handwritten text, improving the accuracy of handwritten text recognition through multiple methods including adaptive grayscale and noise reduction, background normalization, tilt angle estimation and correction, text region detection improved based on PaddleOCR Lite, boundary point aggregation / convex hull processing, and printed / handwritten text region classification. Attached Figure Description

[0022] Figure 1 This is a flowchart of the intelligent recognition method for handwritten documents using edge-cloud collaboration according to the present invention.

[0023] Figure 2 This is a flowchart of the handwriting recognition method of the terminal-side lightweight handwriting recognition model of the present invention.

[0024] Figure 3 This is a schematic diagram of the structured data elements for local identification processing according to the present invention.

[0025] Figure 4 This is a flowchart of the handwriting recognition method of the cloud-based handwriting recognition model of the present invention.

[0026] Figure 5 This is a schematic diagram of the end-to-end cloud collaborative intelligent handwritten document recognition system of the present invention. Detailed Implementation

[0027] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0028] Example: like Figures 1-4 As shown, a method for intelligent recognition of handwritten documents through edge-cloud collaboration includes the following steps: Step 1: Capture the original image of the handwritten document and transmit the original image to the terminal; For any handwritten document that requires handwriting recognition processing, the original image of the handwritten document is processed by using an image acquisition device such as a scanner (scanner), high-speed document scanner, or mobile phone with a camera, in a well-lit and flat position. Step 2: The original image is processed locally using the lightweight handwriting recognition model and handwriting recognition method on the terminal side, and the locally processed structured data is uploaded to the cloud. Step 3: The structured data uploaded from local recognition and processing is then recognized and calibrated in the cloud using a cloud-based Transformer architecture-based visual-language model collaborative correction method for handwritten character recognition. Step 4: Integrate the structured data processed locally with the cloud-based recognition and calibration data to output the document recognition results and send them back to the terminal in JSON format.

[0029] The steps involved in localizing the original image in the lightweight handwriting recognition model and handwriting recognition method on the terminal side include: Step S1: Perform adaptive grayscale conversion and image noise reduction on the original image of the input handwritten document on the terminal; Step S2: Perform adaptive background normalization on the image processed in step S1 on the terminal; Adaptive background normalization is designed to address the shadow / uneven lighting issues associated with handwritten documents. By uniformly processing the image background, it facilitates subsequent document recognition.

[0030] Step S3: Perform adaptive tilt correction on the image processed in step S2 on the terminal; The device scans handwritten documents at a certain tilt angle. After performing adaptive grayscale, noise reduction, and background normalization on the handwritten document, in order to improve the accuracy of handwritten document recognition, it is necessary to perform adaptive tilt angle estimation and correction on the image. Step S4: Perform text region detection on the image processed in step S3 based on the local lightweight OCR recognition model, obtain the position information of each text region, and initially classify the obtained text regions into 'printed' or 'handwritten', while processing overlapping regions and excessively small fragmented regions. Step S5: Perform boundary point aggregation and convex hull processing on the detected text regions to extract regional geometric features; Step S6: Based on geometric features and OCR confidence, classify the text region into printed / handwritten characters and calculate the confidence of printed / handwritten character features; After text region detection via PaddleOCR Lite in step S4 and boundary point aggregation / convex hull processing in step S5, the current document is processed into the smallest rotated rectangular text region labeled with 'printed' and 'handwritten' by PaddleOCR Lite. The width and height, aspect ratio, area, perimeter, ratio of convex hull area to axis-aligned circumscribed rectangle area, and roundness features of the smallest rotated rectangular text region patch are obtained. Classification confidence is calculated based on regional geometric features, and the printed / handwritten text regions are quickly classified based on the classification confidence results. The classification confidence score is calculated using a machine learning model prediction method, and all tiles are quickly classified based on the classification confidence score.

[0031] Calculate the confidence scores of printed / handwritten features: The confidence scores of printed text regions are calculated based on features such as contour regularity and area ratio; the confidence scores of handwritten regions are calculated based on features such as contour complexity and roundness.

[0032] Features of contour regularity include the ratio of the convex hull area to the area of ​​the axis-aligned circumscribed rectangle; features of shape complexity include roundness and boundary smoothness. Text layout features, including aspect ratio and area percentage; OCR identifies confidence features.

[0033] Step S7: Based on the preliminary classification results of printed / handwritten text, feature confidence and terminal network status, make a decision on whether to perform local recognition or upload to the cloud for recognition of all 'printed' and 'handwritten' text regions using an intelligent routing method; Step S8: After completing the local terminal identification and processing through the intelligent routing method, structured data for local terminal identification and processing is formed; The structured data processed by the local terminal includes the following elements: unique identifier (id) of the text region, bounding box of the rotated rectangle of the text region, local identification identifier, cloud identification request, local identification result, and confidence level. 'printed' text region unique identifier id: text region rotating rectangle bounding box: local identification identifier: local identification result: confidence level; 'printed' text region unique identifier id: text region rotating rectangle bounding box: cloud recognition request: confidence level; 'handwritten' text region unique identifier id; text region rotating rectangle bounding box; local recognition identifier; local recognition result; confidence level; 'handwritten' text region ID: unique identifier: text region rotating rectangle bounding box: cloud recognition request: confidence level.

[0034] Step S1, adaptive grayscale and noise reduction, includes Step S11: Select an optimized grayscale conversion method based on the image lighting conditions to convert the original RGB image into a grayscale image; Different grayscale conversion algorithms are selected based on the illumination uniformity of the original RGB image; When determining the uniformity of illumination in the original RGB image, an efficient grayscale conversion method based on OpenCV standards is used. When it is determined that the lighting of the original RGB image is uneven, an optimized grayscale conversion function, optimized_grayscale_conversion(img), is used to convert the original image of the document to grayscale.

[0035] Step S12: Denoising the grayscale image; The steps for noise reduction of grayscale images include Step S121: Use Laplacian variance to determine the noise intensity of the grayscale image and classify the noise into three levels: high, medium, and low. Step S122: Denoise high-noise images using the Non-Local Means Denoising algorithm in OpenCV (adaptively adjust filtering parameters according to the image noise level, where: the filtering intensity parameter of the Non-Local Means Denoising is adjusted within the range of [10,20] according to the noise level, the template window size is adaptively determined according to the image resolution and content complexity, and the search window size maintains an appropriate proportional relationship with the template window); denoise medium-noise images using the bilateral filter algorithm in OpenCV (the bilateral filter parameters are dynamically adjusted according to the image characteristics: the filter diameter is determined based on the stroke width feature, and the standard deviation of the color space and coordinate space is adaptively configured according to the image contrast); preserve the original image for low-noise images. Based on the background brightness (old paper is usually darker and yellower), estimated contrast (standard deviation), and edge density, the original documents are categorized in the image after adaptive grayscale and noise reduction processing. Then, background normalization is performed according to the classification. Adaptive background normalization defines three document types for document shadows / uneven lighting: i. severely degraded, low-contrast documents (old paper document type); ii. slightly uneven, requiring fidelity preservation documents (modern printed document type); iii. documents with large dynamic range and complex structure (mixed handwritten document type). The original documents are first categorized, and then adaptive background normalization is performed based on the categorized type. Specifically, this includes: Step S20: Set the document classification type and the classification rules for the type; Document classification types include old paper document types, modern printed document types, and mixed handwritten document types; The characteristics of old paper document type are severe degradation and low contrast: set when the background brightness is <180 and the contrast is <40, it is old paper (aged_paper). The characteristics of modern print documents are slight unevenness and a requirement for fidelity: a document is considered a modern print document if its edge density is greater than 0.08 and its contrast is greater than 60. The characteristics of mixed handwritten document types are large dynamic range and complex structure: when the edge density is <0.05, it is defined as a mixed handwritten document type. Step S21: Set the adaptive background normalization processing method according to the document classification type; The adaptive background normalization method for documents categorized as "old paper" is as follows: Adaptive histogram equalization (skimage.exposure.equalize_adapthist) is used, with clip_limit=0.03 limiting the contrast enhancement to avoid over-amplifying noise while enhancing the difference between foreground text and background; The adaptive background normalization method for documents categorized as modern print documents is as follows: We used cv2.GaussianBlur to perform a large kernel Gaussian blur (sigmaX=25) to extract the background trend. At the same time, we used cv2.divide to divide the original image and the background image, and then scaled it to the range of 0-255 to cancel out uneven background lighting and make the overall background more uniform. The adaptive background normalization method for documents categorized as mixed handwritten document types is as follows: Use cv2.createCLAHE (limited contrast adaptive histogram equalization), control local contrast with clipLimit=2.0, and divide the image into blocks with tileGridSize=(8,8) for local enhancement to highlight the edges of handwritten strokes; For other document types that are not categorized, use cv2.cvtColor to convert the color image (BGR format) to a grayscale image as the basis for subsequent processing (most document analysis tasks are based on grayscale images).

[0036] Step S22: Calculate the background brightness, contrast, and edge density of the original image; Step S23: Classify documents based on calculated background brightness, contrast, edge density, and type classification rules: Step S24: Perform adaptive background normalization on the image according to the document classification type.

[0037] The device scans handwritten documents at a certain tilt angle. After adaptive grayscale, noise reduction, and background normalization of the handwritten documents, to improve the accuracy of handwritten document recognition, key features are calculated on the image output from the previous step. Based on the calculation results, the images are divided into three types: dense text, table / form, and sparse text. Then, tilt angle estimation and correction are performed according to the type, including the following steps: Step S31: Set the document content type; Document content types are categorized into dense text, table / form, and sparse text. Set up a content categorization mechanism: Set a threshold for the text coverage ratio. If the calculated text coverage ratio is greater than the set threshold, it is determined to be a dense text type. Set a threshold for the proportion of text area in a document (30%). When the calculated text area is greater than the set threshold, it is determined that the main content is text (such as book pages or document body). When the proportion of text area is less than the threshold, it is determined whether there are a large number of horizontal lines in the image (such as tables, musical staves, forms, etc.). If they exist, it is a table / form type; if they do not exist, it is a sparse text type. Step S32: Set the adaptive tilt angle correction method according to the type of document content; An adaptive tilt angle correction method based on dense text types: Utilizing horizontal / vertical projection contours: When the text line is horizontal, the variance of the horizontal projection is the largest (or the energy is most concentrated). By rotating the image and calculating the projection change, the optimal angle is found to complete the correction.

[0038] An adaptive tilt angle correction method based on table / form types: The Hough transform is used to detect straight lines, the angles of the main straight lines are statistically analyzed, and the average value is taken as the tilt angle for correction. An adaptive tilt angle correction method based on sparse text types: The correction is completed by fitting the principal axis direction based on the direction of the circumscribed rectangle of the connected components.

[0039] Step S33: Calculate the key features of the grayscale image: edge density, text coverage ratio, and strong horizontal lines. Based on the calculation results, categorize the document content. Step S34: Based on the document content classification type, perform adaptive tilt angle estimation and correction on the image.

[0040] Methods for text region detection based on a local lightweight OCR recognition model on the image output from the previous step include: Step S41: Perform all text region detection on the image and output all candidate text boxes. Step S42: Based on font texture, stroke continuity, and OCR confidence features, perform a preliminary classification of each detected candidate text box: whether it is 'printed' or 'handwritten'. Step S43: Use a double loop to traverse all regions and use IoU (Intersection over Union) to measure the degree of overlap between two bounding boxes. When the overlapping area exceeds the overlap threshold set for either region, it is considered "significant overlap". Identify the classification of the overlapping regions. If both are printed or both are handwritten (which may be normal text line overlap), keep them as they are. If one is printed and the other is handwritten (handwritten annotations on printed text), it is considered "annotation covering the main text". Separate the printed and handwritten text, delete the printed text region, and keep the handwritten text region (annotation). The overlap threshold is set to 30%. Step S44: Set a pixel threshold for excessively small regions, traverse each region, filter out excessively small regions smaller than the pixel threshold, and retain the remaining detection regions; Filtering out very small areas can prevent meaningless tiny fragments (such as only a few pixels remaining) from being created by cropping. These areas are of no value to OCR and increase noise. After completing text region detection, geometric normalization is required to prepare well-structured and feature-clear input data for the subsequent "printed / handwritten character classification" module. The specific method is as follows: Step S51: Traverse each detection region; Step S52: Use the Douglas-Peucker algorithm to simplify the contours of all detection regions to obtain polygonal detection regions; The adaptive threshold for the perimeter of the text region's outline during simplification is set to 2.5%. Step S53: Convert the simplified polygon into a convex polygon (without concavity). Step S54: Extract the convex hull and extract the smallest rotation rectangle that can enclose the convex hull; The convex hull is the contour feature of a convex polygon; Step S55: Calculate the geometric feature parameters of the region based on the convex hull and the minimum rotation rectangle; Geometric feature parameters include Calculate the width and height, aspect ratio, area, perimeter, ratio of the convex hull area to the area of ​​the axis-aligned bounding rectangle, and roundness (ratio of the convex hull area to the area of ​​the largest bounding circle of the convex hull) of the axis-aligned bounding rectangle. Printed / handwritten text region classification is based on the following combination of features: Features of contour regularity include the ratio of the convex hull area to the area of ​​the axis-aligned circumscribed rectangle; features of shape complexity include roundness and boundary smoothness. Text layout features, including aspect ratio and area percentage; OCR identifies confidence features.

[0041] The method for local recognition and intelligent routing of printed / handwritten text in step S7 is as follows: The system considers the text region type (printed / handwritten) and its feature confidence levels, as well as the terminal's network status (including latency and bandwidth). It then decides whether to perform local recognition or upload the text to the cloud for recognition of all 'printed' and 'handwritten' text regions. The decision threshold is dynamically adjusted based on network conditions and user selection, specifically including: Step S71: Generate structured data for local terminal recognition processing from the data output by the printed / handwritten text region classification in step S6; Step S72: Determine if the local terminal is offline. If yes, process and output the result using the lightweight OCR recognition model on the local terminal. If no, proceed to step S73: Step S73: Determine whether to select local recognition mode. If yes, process and output the result using the lightweight OCR recognition model on the local terminal. If no, proceed to step S74: Step S74: Based on the machine learning model, adaptively determine whether the printed text area and the handwritten text area need to be uploaded to the cloud for recognition and correction. If yes, proceed to step S75. If no, use the local terminal lightweight OCR recognition model for recognition and processing. The adaptive judgment method of the machine learning model is to set confidence thresholds for printed text regions and handwritten text regions. Based on the confidence thresholds for printed text regions, handwritten text regions, and the classification confidence of each text region calculated in step S6, it is determined whether to upload to the cloud for recognition and correction. For example, high-confidence printed text regions with a calculated classification confidence greater than the confidence threshold for printed text regions are processed locally first. This can be set as needed.

[0042] In this embodiment, the local terminal lightweight OCR recognition model used is PaddleOCR Lite. Step S75: For handwritten or printed areas that need to be uploaded to the cloud for recognition and correction, the routing decision is adaptively adjusted based on network conditions. The decision threshold is dynamically adjusted based on network conditions and user selection. When the network quality assessment (which includes latency, bandwidth, and packet loss rate) meets the requirements, the data that needs to be uploaded to the cloud for recognition and correction is processed using a hash algorithm and then uploaded to the cloud for recognition and calibration. When the network quality assessment does not meet the requirements, the data that needs to be uploaded to the cloud for recognition and correction is processed using a lightweight OCR recognition model on the local terminal and labeled.

[0043] The large model on the cloud side is a visual-language model with a Transformer architecture on the cloud side; in this example, it is TroOCR. The received data is identified and calibrated using a cloud-based large-model handwritten character recognition collaborative correction method: Step 31: Local recognition and processing of structured data output. All data containing "cloud recognition request" is recognized by the large model on the cloud side; Step 32: For all 'handwritten' handwriting types that have been locally identified and marked, and whose confidence level is higher than the preset handwriting confidence threshold (hf > or = 0.48), the structured data output from the local recognition process is then recognized and corrected through the cloud-based large model.

[0044] The fusion output includes: semantic correction of the recognition results based on contextual information; and structured output of the terminal recognition results according to the original document layout while saving the data in the cloud. Specifically: Step 41: Merge the uploaded locally processed structured data and cloud-based recognition and calibration data according to the unique identifier ID of the text region; Step 42: The cloud-based large model performs semantic correction on the recognition results based on contextual information; Step 43: Maintain the original document layout in the structured output; Step 44: Send the structured data back to the terminal in JSON format.

[0045] like Figure 5 As shown, this application also provides a cloud-based collaborative intelligent handwritten document recognition system, including a terminal-side system module and a cloud-side system module. The terminal-side system modules include a handwritten document image scanning module, an image processing module, a text recognition and classification module, a local recognition and processing module, and an end-to-cloud collaborative communication module; The handwritten document image scanning module is used to process handwritten documents into image data; it has multiple functions, including support for multiple image input sources, including real-time camera capture, album import, scanner input, etc.; it can automatically detect image quality and prompt users to retake blurry or poorly lit images; it can support batch processing and can process multiple pages of documents at once.

[0046] The image processing module is used for image data preprocessing. It has multiple functions, including integrated adaptive grayscale conversion, noise removal, and background normalization algorithms; it can display the preprocessing effect in real time, and users can manually adjust the parameters; it supports saving preprocessing parameters to adapt to different scenario requirements. The text recognition and classification module is used for text region detection and classification in image data. It is based on a lightweight OCR recognition model for text region detection; it integrates a boundary aggregation optimization algorithm to improve the accuracy of region classification; it displays the detection and classification results in real time and supports manual correction. The local recognition processing module is used for local recognition processing of image data, enabling rapid recognition of printed text, providing confidence assessment and recognition result preview, as well as manual editing and correction of the recognition results.

[0047] The edge-cloud collaborative communication module is used to realize secure and reliable data transmission between the terminal and the cloud, supporting breakpoint resume and network status adaptation, as well as providing transmission progress display and error retry mechanism.

[0048] The cloud-side system module includes a cloud recognition module and a cloud data storage module. The cloud recognition module is used to perform complex handwritten character recognition, recognition result correction, and overall recognition result fusion output on image data. It deploys a Transformer architecture vision-language model for handwritten character recognition, which is used to achieve semantic correction of recognition results, support both batch processing and real-time processing modes, intelligently fuse terminal and cloud recognition results, perform confidence-weighted and contextual consistency checks, and maintain the original layout of the document for structured output.

[0049] The cloud data storage module is used to save all document recognition results structured data, realize the storage of recognition results and process data, support data encryption and access control, and provide data backup and recovery mechanisms.

[0050] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A method for intelligent recognition of handwritten documents using edge-cloud collaboration, characterized in that: Includes the following steps: Step 1: Capture the original image of the handwritten document and transmit the original image to the terminal; Step 2: The original image is processed locally using the lightweight handwriting recognition model and handwriting recognition method on the terminal side, and the locally processed structured data is uploaded to the cloud. Step 3: The structured data uploaded from local recognition and processing is then recognized and calibrated in the cloud using a cloud-based large-model handwritten character recognition collaborative correction method; Step 4: Integrate the structured data processed locally with the cloud-based recognition and calibration data to output the document recognition results and send them back to the terminal in JSON format.

2. The intelligent recognition method for handwritten documents using edge-cloud collaboration according to claim 1, characterized in that: The terminal-side lightweight handwriting recognition model handwriting recognition method includes the following steps for localizing the original image: Step S1: Perform adaptive grayscale conversion and image noise reduction on the original image of the input handwritten document on the terminal; Step S2: Perform adaptive background normalization on the image processed in step S1 on the terminal; Step S3: Perform adaptive tilt correction on the image processed in step S2 on the terminal; Step S4: Perform text region detection on the image processed in step S3 based on the local lightweight OCR recognition model, obtain the location information of each text region, and initially classify the obtained text regions into 'printed' or 'handwritten', while processing overlapping regions and excessively small fragmented regions; Step S5: Perform boundary point aggregation and convex hull processing on the detected text regions to extract regional geometric features; Step S6: Based on geometric features and OCR confidence, classify the text region into printed / handwritten characters and calculate the confidence of printed / handwritten character features; Step S7: Based on the preliminary classification results of printed / handwritten text, feature confidence and terminal network status, make a decision on whether to perform local recognition or upload to the cloud for recognition of all 'printed' and 'handwritten' text areas using an intelligent routing method; Step S8: After completing the local terminal identification process through the intelligent routing method, structured data for local terminal identification processing is generated.

3. The intelligent recognition method for handwritten documents using edge-cloud collaboration according to claim 1, characterized in that: Step 3 involves recognizing and calibrating the received data using a cloud-based large-model handwriting recognition collaborative correction method. Step 31: Local recognition and processing of structured data output. All data containing "cloud recognition request" is recognized by the large model on the cloud side; Step 32: Output the structured data from the local recognition process. All 'handwritten' handwriting types that have been locally recognized and identified and whose confidence level is higher than the preset handwriting confidence threshold are then recognized and corrected by the large model on the cloud side.

4. The intelligent recognition method for handwritten documents using edge-cloud collaboration according to claim 1, characterized in that: Step 4 includes Step 41: Merge the uploaded locally processed structured data and cloud-based recognition and calibration data according to the unique identifier ID of the text region; Step 42: The cloud-based large model performs semantic correction on the recognition results based on contextual information; Step 43: Maintain the original document layout in the structured output; Step 44: Send the structured data back to the terminal in JSON format.

5. The intelligent recognition method for handwritten documents using end-to-end cloud collaboration according to claim 2, characterized in that: The adaptive grayscale and noise reduction processing in step S1 includes Step S11: Select an optimized grayscale conversion method based on the image lighting conditions to convert the original RGB image into a grayscale image; Step S12: Denoising the grayscale image; The noise intensity of grayscale images is determined by Laplacian variance, and the noise is divided into three levels: high, medium, and low. High-noise images are denoised using the nonlocal mean-based denoising algorithm in OpenCV. Medium-noise images are denoised using the bilateral filtering algorithm in OpenCV. Low-noise images are preserved.

6. The intelligent recognition method for handwritten documents using end-to-end cloud collaboration according to claim 2, characterized in that: The adaptive background normalization process in step S2 includes... Step S20: Set the document classification type and the classification rules for the type; Document classification types include old paper document types, modern printed document types, and mixed handwritten document types; classification rules are set for each type to categorize documents; Step S21: Set the adaptive background normalization processing method according to the document classification type; Step S22: Calculate the background brightness, contrast, and edge density of the original image; Step S23: Classify the documents based on calculated background brightness, contrast, edge density, and type classification rules; Step S24: Perform adaptive background normalization on the image based on the document's classification type.

7. The intelligent recognition method for handwritten documents using end-to-end cloud collaboration according to claim 2, characterized in that: The method for S5 to perform boundary point aggregation and convex hull processing on the detected text regions and extract regional geometric features includes: Step S51: Traverse each detection region; Step S52: Use the Douglas-Peucker algorithm to simplify the contours of all detection regions to obtain polygonal detection regions; Step S53: Convert the simplified polygon into a convex polygon; Step S54: Extract the convex hull and extract the smallest rotation rectangle that can enclose the convex hull; Step S55: Calculate the geometric feature parameters of the region based on the convex hull and the minimum rotation rectangle.

8. A cloud-edge collaborative intelligent handwritten document recognition system, characterized in that: It includes a terminal-side system module and a cloud-side system module. The terminal-side system module includes a handwritten document image scanning module, an image processing module, a text recognition and classification module, a local recognition and processing module, and an end-to-cloud collaborative communication module. The cloud-side system module includes a cloud recognition module and a cloud data storage module.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the end-to-cloud collaborative intelligent recognition method for handwritten documents as described in any one of claims 1-7.

10. A computer program, characterized in that, When the computer program is executed by a processor, it is used to perform the intelligent handwritten document recognition method for end-to-end cloud collaboration as described in any one of claims 1-7.

Images