Design method of intelligent handwritten form recognition system fused with image modal perception

Abstract

The invention discloses a design method of an intelligent handwritten recognition system fused with image modal perception, and the method achieves the recognition of a scanned image of a handwritten text and the digital storage of a recognition result through the construction of the intelligent handwritten recognition system fused with image modal perception. The system is formed by combining an OCR (Optical Character Recognition) model, a MobileNetV3 model and a graph convolution model. And training the intelligent handwritten form recognition system model fused with image modal perception, selecting optimal model training parameters, and determining a final system model. The recognition process of the system specifically comprises the following steps: inputting a to-be-recognized handwritten text scanning image, and carrying out optimization processing; a handwritten text image template is selected, related information is selected in a frame mode, and corresponding position coordinate information is stored; intercepting information to be identified from the handwritten text scanning image after pixel point alignment; identifying the to-be-identified information by using the intelligent handwritten form identification system model fused with image modal perception; a system identification result is stored in an Excel table form; and marking an anomaly detection result of the Excel table by the system.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a design method for an intelligent handwriting recognition system integrating image modality perception. Background Art

[0002] In the context of modern social development, the need for intelligent processing of handwritten text images is becoming increasingly prominent. This is not only because intelligent processing can significantly improve the speed and accuracy of data processing through optical character recognition technology, meeting the high-precision processing requirements of massive questionnaire data and paper marking, and effectively reducing manual data entry errors, but also because it facilitates storage and retrieval, saves physical space, protects the environment, and conforms to the trend of sustainable development. In addition, digitized data can be directly used in advanced analytical tools to facilitate information mining and decision-making. The digital processing of handwritten text images is an inevitable choice to adapt to the development trend of an information-based and intelligent society, aiming to better meet the needs of social and economic development.

[0003] Although a variety of handwriting recognition methods already exist in the prior art, these methods are still insufficient in terms of accuracy and are prone to errors, which not only reduces work efficiency but also restricts the potential of handwritten text in intelligent processing and application. Therefore, there is an urgent need for a new technical solution to overcome the problem of handwriting recognition accuracy and improve the overall performance of handwriting recognition. This is also the background of the present invention. Compared with existing patented technologies, the present invention demonstrates unique innovations in the design of functional modules and the application of technical means, and can more effectively cope with various complex situations and achieve more accurate and efficient handwritten text recognition. The present invention aims to break through the limitations of current handwriting recognition technology and promote further development in this field. Summary of the Invention

[0004] The purpose of the present invention is to provide a design method for an intelligent handwriting recognition system that integrates image modality perception, so as to solve the problems of handwriting recognition accuracy and digital storage of recognition results.

[0005] To achieve the above object, the present invention adopts the following technical solution, including the following steps:

[0006] A method for designing an intelligent handwriting recognition system integrating image modality perception includes the following steps:

[0007] Step 1: Construct the information recognition part of the intelligent handwriting recognition system that integrates image modality perception to perform handwriting recognition on scanned images of handwritten text.

[0008] Wherein, the step 1 specifically includes the following steps:

[0009] Step 1.1: Collect handwritten font dataset and preprocess the dataset.

[0010] Step 1.2: Train the OCR model, MobileNetV3 model, and graph convolution model included in the intelligent handwriting recognition system that integrates image modality perception, select the optimal training parameters for each model, and determine the final system model.

[0011] Step 1.3: Evaluate the intelligent handwriting recognition system model that integrates image modality perception.

[0012] Step 1.4: Optimize the scanned image of the handwritten text to be recognized.

[0013] Step 1.5: Before performing text recognition on the handwritten text scanned image, determine the template keyword information and the location coordinates of the area to be filled in by the user based on the handwritten text image template provided by the user.

[0014] Step 1.6: Capture the area to be filled in by the user in the handwritten text image to be recognized.

[0015] Step 1.7: Use a two-stage hierarchical processing mechanism: First, use the OCR model to perform preliminary detection and recognition of text and numbers in the entire area of ​​the handwritten text image template. Then, for the number-sensitive recognition results whose detection confidence reaches the threshold, call the lightweight model MobileNetV3 for refined secondary recognition.

[0016] Step 2: The information recognition part of the intelligent handwriting recognition system integrating image modality perception transmits the recognition results to the information storage part and stores them in the form of Excel tables.

[0017] Preferably, step 1.1 specifically includes the following steps:

[0018] Step 1.1.1: Use the PaddleLabel annotation tool to draw detection boxes and generate label files for the cross-language handwritten character recognition dataset composed of EMNIST, MNIST, and CASIA–HWDB, and construct character localization and recognition data suitable for OCR model training.

[0019] Step 1.1.2: In order to obtain the best training results in graph convolution model training, a standardized graph structure completion dataset is established.

[0020] Step 1.1.3: In the MobileNetV3 model training task, use the MNIST handwritten digit image dataset as the sole training data. Resize the images to a uniform size suitable for model training, convert grayscale to RGB format, and normalize the pixel values.

[0021] Preferably, step 1.2 specifically includes the following steps:

[0022] Step 1.2.1: Divide the preprocessed dataset into a training set and a test set in a ratio of 7:3 for model training.

[0023] Step 1.2.2: Train the OCR model, MobileNetV3 model, and graph convolution model through an iterative training strategy, and systematically evaluate and optimize the trained model parameters to determine the optimal parameter combination for each model.

[0024] Step 1.2.3: Based on the optimal configuration of each model obtained through parameter optimization, the OCR model, MobileNetV3 model, and graph convolution model are integrated by designing a model integration architecture to form an intelligent handwriting recognition system that integrates image modality perception.

[0025] Preferably, step 1.3 specifically includes the following steps:

[0026] Step 1.3.1: Use the test set to perform a final evaluation of the system model. By calculating core performance indicators such as accuracy, recall, and F1 value, combined with confusion matrix visualization analysis, the system quantifies the generalization ability of the model under unknown sample distributions to ensure the reliability and stability of the model in actual application scenarios.

[0027] Preferably, step 1.4 specifically includes the following steps:

[0028] Step 1.4.1: Perform crease, tilt restoration, brightness enhancement, and image resolution enhancement on the scanned image of the handwritten text to be recognized.

[0029] Preferably, step 1.5 specifically includes the following steps:

[0030] Step 1.5.1: Select and identify the keyword information in the template. The selected area is determined by the system's automatic detection and positioning and the user's interactive manual annotation.

[0031] Step 1.5.2: After confirming that the keyword information is captured correctly, the system automatically locates the area to be filled in by the user and saves the coordinate information of the four corners of the selected area.

[0032] Step 1.5.3: Align the pixels of the optimized handwritten text scan image with the corresponding image template provided by the user. By establishing an affine transformation relationship between the handwritten text scan image and the template, a sub-pixel spatial mapping of the template's preset ROI coordinates to the handwritten text scan image is achieved, ensuring accurate capture of the target area of ​​the handwritten text scan image.

[0033] Preferably, step 1.6 specifically includes the following steps:

[0034] Step 1.6.1: Based on the provided coordinate information, the area to be filled in by the user in the scanned image of the handwritten text to be recognized is captured, and the captured image is stored.

[0035] Step 1.6.2: To address the issue of underlines interfering with handwritten text recognition in the user-filled area image, the system uses morphological operations and edge detection image processing technology to remove underlines, and builds a stroke completion algorithm based on a graph convolution model. By topological analysis and feature reconstruction of the damaged text structure, the system restores the complete form of the handwritten text.

[0036] Step 1.6.3: Recognize the captured handwritten text scan image information and implement binary classification of the text information into numbers and non-numbers in the system.

[0037] Preferably, step 1.7 specifically includes the following steps:

[0038] Step 1.7.1: Use the OCR model to recognize the captured handwritten text image and compare the recognized text with the provided standard text library to determine the handwritten content.

[0039] Step 1.7.2: Reuse the MobileNetV3 model to identify the digital part classified in step 1.6.3 above, and output the recognition result.

[0040] Step 1.7.3: Compare and analyze the digit recognition results output by the OCR model and the MobileNetV3 model.

[0041] Preferably, step 2 specifically includes the following steps:

[0042] Step 2.1: Automatically fill the identified keyword information into the first row of the Excel table as the header category information.

[0043] Step 2.2: For each keyword corresponding to the area to be filled in, the recognition results are automatically classified and entered under the corresponding header category.

[0044] In step 2.3, the system will use the anomaly marking algorithm to visually mark the abnormal data items in the generated Excel table.

[0045] Preferably, step 2.3 specifically includes the following steps:

[0046] Step 2.3.1: If there is any numerical data outside the predefined threshold range or the two models’ output results for the digital part recognition do not match, it will be marked in the generated Excel table.

[0047] Step 2.3.2: After completing the recognition processing of the handwritten text scan image dataset, the system automatically extracts the questionnaire names corresponding to the records marked as abnormal or erroneous in the Excel table and creates a dedicated abnormal data archiving folder in the specified storage path to facilitate subsequent manual inspection and error correction.

[0048] The beneficial effects of the present invention are as follows:

[0049] 1. Combining the OCR model, MobileNetV3 model, and graph convolutional model, we designed a model integration architecture to complement each other's strengths. For sensitive digital information, we employed a two-stage mechanism: preliminary recognition using the OCR model and precise recognition using MobileNetV3. Through confidence threshold filtering and model result comparison, we significantly reduced digital recognition errors.

[0050] 2. Build a cross-language dataset including EMNIST, MNIST, and CASIA-HWDB, covering multilingual handwritten characters, and enhancing the system's adaptability to complex scenarios.

[0051] 3. Combine automatic detection with user interactive annotation to determine the template keyword area, and use affine transformation to achieve sub-pixel mapping of the template ROI coordinates to the handwritten image, ensuring high-precision interception of the area to be filled in and avoiding recognition errors caused by positioning deviation.

[0052] 4. Optimize the image to be recognized by repairing creases, correcting tilt, enhancing brightness, and increasing resolution to improve the quality of handwritten text scans. To address underline interference, morphological operations and edge detection are used to remove interfering lines. A stroke completion algorithm is constructed using a graph convolutional model to restore damaged text structure and improve character recognition accuracy.

[0053] 5. Automated Excel table generation: Automatically fill in the table header with the results of keyword recognition based on the template, and enter the corresponding data in categories, reducing manual sorting costs and improving data processing efficiency. It also automatically marks abnormal data and creates a dedicated filing folder to facilitate manual review and error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A model training flowchart for a design method of an intelligent handwriting recognition system that integrates image modality perception.

[0055] Figure 2 Schematic diagram of the recognition process for a design method of an intelligent handwriting recognition system that integrates image modality perception. DETAILED DESCRIPTION

[0056] In order to make the embodiments of the present invention and the prior art solutions clearer and easier to understand, the present invention is described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 1 The steps of the model training flowchart of the intelligent handwriting recognition system based on fusion image modality perception and deep learning are as follows:

[0058] Step 1: Construct the information recognition part of the intelligent handwriting recognition system that integrates image modality perception to perform handwriting recognition on scanned images of handwritten text.

[0059] Wherein, step 1 includes the following steps:

[0060] Step 1.1: Collect handwritten font dataset and preprocess the dataset.

[0061] Wherein, step 1.1 includes the following steps:

[0062] Step 1.1.1: For the cross-language handwritten character recognition dataset composed of EMNIST, MNIST, and CASIA-HWDB, the annotation process begins by converting and preprocessing the combined dataset to ensure that parameters such as image resolution and color space meet PaddleLabel's input specifications. Use the PaddleLabel annotation tool to generate detection boxes and label files, constructing character localization and recognition data suitable for OCR model training.

[0063] Step 1.1.2: During the training of the graph convolutional neural network model, in order to improve the generalization ability and training stability of the model and obtain the best training results, it is necessary to preprocess and structurally optimize the original graph data and establish a standardized and structurally complete graph structure completion dataset.

[0064] Step 1.1.3: Because MobileNetV3 is a lightweight convolutional neural network model, its recommended input size is 224×224 pixels to meet the design requirements of its feature extraction module. Therefore, the original MNIST images must be preprocessed to a uniform training size and pixel value normalization to ensure smooth input and training.

[0065] Step 1.2: Train the OCR model, MobileNetV3 model, and graph convolution model included in the intelligent handwriting recognition system that integrates image modality perception, select the optimal training parameters for each model, and determine the final system model.

[0066] Wherein, step 1.2 includes the following steps:

[0067] Step 1.2.1: Divide the preprocessed dataset into a training set and a test set in a ratio of 7:3 for training each model. Set the number of iterations to 200 and the batch size to 16.

[0068] The OCR model was trained on a comprehensive dataset consisting of three public datasets: EMNIST, MNIST, and CASIA-HWDB. The MobileNetV3 model was trained on the MNIST dataset containing handwritten digits. The graph convolutional network was trained on a self-constructed dataset of fonts with discontinuous strokes.

[0069] Step 1.2.2: Train the OCR model, MobileNetV3 model, and graph convolution model through an iterative training strategy, and systematically evaluate and optimize the trained model parameters to determine the optimal parameter combination for each model.

[0070] Among them, the key evaluation indicators for the OCR model and MobileNetV3 model after training are mainly the following: Precision (P) and Recall (R); the key evaluation indicators for the graph convolutional neural network include: Node Accuracy and Edge Accuracy. The specific calculation formula is as follows:

[0071]

[0072]

[0073] Among them, TP (True Positives) is the number of correct positive samples, FN (False Negatives) is the number of negative samples predicted incorrectly, and FP (False Positives) is the number of positive samples predicted incorrectly.

[0074]

[0075]

[0076] Step 1.2.3: After completing independent training of the OCR recognition model, MobileNetV3 model, and graph convolution model, optimize hyperparameters such as the learning rate, batch size, and regularization coefficient to determine the optimal model training parameters. By designing a model integration architecture, the OCR model, MobileNetV3 model, and graph convolution model are integrated to form an intelligent handwriting recognition system that integrates image modality perception. The interface of this intelligent handwriting recognition system that integrates image modality perception is set up using PyQt software. This system uses visual language to simplify cognitive load, improves operational efficiency through interactive design, and assists decision-making through data visualization.

[0077] Step 1.3: Evaluate the intelligent handwriting recognition system model that integrates image modality perception.

[0078] Wherein, step 1.3 includes the following steps:

[0079] Step 1.3.1: To scientifically evaluate the generalization performance of the system model, perform a final evaluation of the system model using the test set. Obtain various performance indicators on unseen data to accurately measure the model's generalization ability. Deploy the optimal model to the actual application environment for online testing and actual use. During the deployment process, consider the model's operational efficiency and resource usage to ensure that the model can run stably and efficiently in real-world scenarios.

[0080] Figure 2 This is a flow chart of the recognition design method for an intelligent handwriting recognition system that integrates image modality perception.

[0081] Step 1.4: Optimize the scanned image of the handwritten text to be recognized.

[0082] Among them, step 1.4 includes the following steps:

[0083] Step 1.4.1: Perform crease, tilt restoration, brightness enhancement, and image resolution enhancement on the scanned image of the handwritten text to be recognized.

[0084] Among them, the optimization processing operation of the scanned image of handwritten text is essentially to make up for the quality defects of the original image through technical means, and optimize the visual effect from the four dimensions of "denoising (creases) - correction (tilt) - enhancement (brightness) - refinement (resolution)", so that the text information is closer to the ideal input state of the OCR system, and ultimately achieve the goal of "improving recognition accuracy, reducing processing costs, and expanding application scenarios."

[0085] Step 1.5: Before performing text recognition on the handwritten text scanned image, determine the template keyword information and the location coordinates of the area to be filled in by the user based on the handwritten text image template provided by the user.

[0086] Among them, step 1.5 includes the following steps:

[0087] Step 1.5.1: Before performing the handwriting recognition task, the user can use the system interface to perform overall recognition of the handwritten text image template or partial keyword recognition based on actual needs. After the system completes template recognition, all keywords in the template will be selected. If there are omissions or errors in the recognition of keywords in the template, the user will need to reselect the missing keywords or delete the incorrectly recognized keywords or text on the system interface.

[0088] Step 1.5.2: When the keyword information is verified, the system will make a preliminary selection of the area to be filled in corresponding to the keyword based on the preset visual feature model. After completing the initial selection, the system will expand the selection range based on the initial selection boundary through the morphological expansion algorithm until the boundary contacts the non-blank pixel area for the first time. Finally, the system performs standardized cropping according to the shortest side length of the expanded selection area to form a square interception area, and saves the pixel coordinate information of the four corners of the area; if two areas to be filled in are selected for a keyword, the system automatically marks the suspicious area and attempts to correct it automatically. Secondly, if only part of the keyword information in the questionnaire template needs to be recognized, the user is required to manually select part of the keyword area. The system retrieves the area to be filled in set around the keyword, selects the area to be filled in, expands the selection area, and saves the coordinate information of the four corners of the selection area.

[0089] Step 1.5.3: After confirming the keyword information and the area to be filled in, select the handwritten text scan image and perform pixel alignment between the optimized handwritten text scan image and the user-provided template. By establishing an affine transformation relationship between the handwritten text scan image and the template, a sub-pixel spatial mapping is achieved from the template's preset ROI coordinates to the handwritten text scan image, ensuring accurate capture of the target area in the handwritten text scan image. The images of the same questionnaire to be filled in are saved in a new folder created by the system, named corresponding to the questionnaire image.

[0090] Step 1.6: Capture the area to be filled in by the user in the handwritten text image to be recognized.

[0091] Among them, step 1.6 includes the following steps:

[0092] Step 1.6.1: Based on the provided coordinate information, the area to be filled in by the user in the scanned image of the handwritten text to be recognized is captured, and the captured image is stored.

[0093] Step 1.6.2: After capturing the corresponding filled-in areas in the scanned handwritten text image, the captured filled-in areas contain underlines that interfere with recognition. The system uses morphological operations and edge detection algorithms to accurately remove the underlines. At the same time, the graph convolutional model repairs the discontinuous strokes caused by underline removal. By deeply modeling the text structure features, the integrity of the discontinuous strokes is restored, thereby improving the text image quality.

[0094] Step 1.6.3: The system batch processes the intercepted target folders through a traversal algorithm. For all the intercepted images in the folder, the trained OCR model and MobileNetV3 model are called to recognize the intercepted images, and the system implements binary classification of digital and non-digital text information.

[0095] Step 1.7: Use a two-stage hierarchical processing mechanism: First, use the OCR model to perform preliminary detection and recognition of text and numbers in the entire area of ​​the handwritten text image template. Then, for the number-sensitive recognition results whose detection confidence reaches the threshold, call the lightweight model MobileNetV3 for refined secondary recognition.

[0096] Among them, step 1.7 includes the following steps:

[0097] Step 1.7.1: For the target area recognition task, use the OCR model to perform preliminary detection and recognition of the handwritten content in the entire template area, distinguishing between numbers and text. For text, compare the text recognized by the OCR model with the provided standard text library to determine the text content.

[0098] Step 1.7.2: Use the lightweight MobileNetV3 model to perform refined secondary recognition on the digital information and compare the recognition results of the OCR model and the MobileNetV3 model.

[0099] Step 1.7.3: Compare the recognition results of the OCR model and the MobileNetV3 model. If they are the same, output the correct result; if they are different, output the recognition result of the MobileNetV3 model as the correct result.

[0100] Step 2: The information recognition part of the intelligent handwriting recognition system integrating image modality perception transmits the recognition results to the information storage part and stores them in the form of Excel tables.

[0101] Wherein, step 2 includes the following steps:

[0102] Step 2.1: Place the identified keyword information in the first row of the Excel table as header category information.

[0103] Step 2.2: For the screenshots of the area to be filled in corresponding to each keyword, the system automatically classifies the recognition results and enters them under the corresponding header category.

[0104] Step 2.3: The system will use the anomaly marking algorithm to visually mark the abnormal data items in the generated Excel table.

[0105] Wherein, step 2.3 includes the following steps:

[0106] Step 2.3.1: For text recognition tasks, the OCR model output is used directly as the baseline. For digit recognition tasks, verification is performed by comparing the OCR model's results with those of the MobileNetV3 model. Specifically, if the outputs of the two models match, they are recorded directly in an Excel spreadsheet as the correct recognition result without any special annotation. If the recognition results do not match, the MobileNetV3 model output is used as the final judgment and the recognition result is visually marked in the Excel spreadsheet. An abnormal data detection algorithm is established. When a recognition result with a non-standard category label appears, or when numerical data (such as age) exceeds a predefined threshold, the abnormality annotation process in the Excel spreadsheet is automatically triggered. Specifically, in the gender binary classification scenario, if the recognition result does not match the standard "male" or "female" label set, or if the age data record value exceeds the reasonable range of 0-150, the system will assign a special mark to the corresponding data item to distinguish it from normal data. This facilitates subsequent analysis and manual review.

[0107] Step 2.3.2: After intelligent recognition and processing of the handwritten text scan image, the system automatically extracts the associated file names based on the exception tags in the Excel annotation file and creates a dedicated exception data archiving folder in the specified storage path, providing efficient and convenient retrieval and processing support for subsequent manual review and data correction.

Claims

1. A design method for an intelligent handwriting recognition system integrating image modality perception, characterized in that: The steps include: Step 1: Construct the information recognition part of the intelligent handwriting recognition system that integrates image modality perception to perform handwriting recognition on scanned images of handwritten text; Wherein, the step 1 specifically includes the following steps: Step 1.1: Collect handwritten font dataset and preprocess the dataset; Step 1.2: Train the OCR model, MobileNetV3 model, and graph convolution model included in the intelligent handwriting recognition system that integrates image modality perception, select the optimal training parameters for each model, and determine the final system model; Step 1.3: Evaluate the intelligent handwriting recognition system model that integrates image modality perception; Step 1.4: Optimize the scanned image of the handwritten text to be recognized; Step 1.5: Before performing text recognition on the handwritten text scanned image, determine the template keyword information and the location coordinates of the area to be filled in by the user based on the handwritten text image template provided by the user; Step 1.6: Capture the area to be filled in by the user in the handwritten text image to be recognized; Step 1.7: A two-stage hierarchical processing mechanism is used: First, the OCR model performs preliminary detection and recognition of text and numbers across the entire handwritten text image template. Then, for number-sensitive recognition results that reach a detection confidence threshold, the lightweight MobileNetV3 model is used for refined secondary recognition. Step 2: The information recognition part of the intelligent handwriting recognition system integrating image modality perception transmits the recognition results to the information storage part and stores them in the form of Excel tables.

2. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.1 specifically includes the following steps: Step 1.1.1: Use the PaddleLabel annotation tool to draw detection boxes and generate label files for the cross-language handwritten character recognition dataset composed of EMNIST, MNIST, and CASIA–HWDB, and construct character localization and recognition data suitable for OCR model training; Step 1.1.2: In order to obtain the best training results in graph convolution model training, a standardized graph structure completion dataset is established; Step 1.1.3: In the MobileNetV3 model training task, use the MNIST handwritten digit image dataset as the sole training data. Resize the images to a uniform size suitable for model training, convert grayscale to RGB format, and normalize the pixel values.

3. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.2 specifically includes the following steps: Step 1.2.1: Divide the preprocessed dataset into a training set and a test set in a ratio of 7:3 for model training; Step 1.2.2: Train the OCR model, MobileNetV3 model, and graph convolution model using an iterative training strategy. Systematically evaluate and optimize the trained model parameters to determine the optimal parameter combination for each model. Step 1.2.3: Based on the optimal configuration of each model obtained through parameter optimization, the OCR model, MobileNetV3 model, and graph convolution model are integrated by designing a model integration architecture to form an intelligent handwriting recognition system that integrates image modality perception.

4. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.3 specifically includes the following steps: Step 1.3.1: Use the test set to perform a final evaluation of the system model. By calculating the core performance indicators of accuracy, recall, and F1 value, combined with confusion matrix visualization analysis, the system quantifies the generalization ability of the model under unknown sample distributions to ensure the reliability and stability of the model in actual application scenarios.

5. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.4 specifically includes the following steps: Step 1.4.1: Perform crease, tilt restoration, brightness enhancement, and image resolution enhancement on the scanned image of the handwritten text to be recognized.

6. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.5 specifically includes the following steps: Step 1.5.1: Select and identify the template keyword information. The selected area is determined by the system's automatic detection and positioning and the user's interactive manual annotation. Step 1.5.2: After confirming that the keyword information is correctly captured, the system automatically locates the area to be filled in by the user and saves the coordinates of the four corners of the selected area; Step 1.5.3: Align the pixels of the optimized handwritten text scan image with the corresponding image template provided by the user. By establishing an affine transformation relationship between the handwritten text scan image and the template, a sub-pixel spatial mapping of the template's preset ROI coordinates to the handwritten text scan image is achieved, ensuring accurate capture of the target area of ​​the handwritten text scan image.

7. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.6 specifically includes the following steps: Step 1.6.1: Capture the area to be filled in by the user in the scanned image of the handwritten text to be recognized based on the provided coordinate information, and store the captured image; Step 1.6.2: To address the issue of underlines interfering with handwritten text recognition in the user-filled area image, the system uses morphological operations and edge detection image processing techniques to remove underlines. It also builds a stroke completion algorithm based on a graph convolutional model. This algorithm restores the complete form of the handwritten text by topological analysis and feature reconstruction of the damaged text structure. Step 1.6.3: Recognize the captured handwritten text scan image information and implement binary classification of the text information into numbers and non-numbers in the system.

8. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 1.7 specifically includes the following steps: Step 1.7.1: Use an OCR model to recognize the captured handwritten text image and compare the recognized text with the provided standard text library to determine the handwritten content; Step 1.7.2: Use the MobileNetV3 model again to identify the classified digits and output the recognition results. Step 1.7.3: Compare and analyze the digit recognition results output by the OCR model and the MobileNetV3 model.

9. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 1 is characterized by: Step 2 specifically includes the following steps: Step 2.1: Automatically fill the identified keyword information into the first row of the Excel table as the header category information; Step 2.2: For each keyword corresponding to the area to be filled in, automatically classify the recognition results and enter them under the corresponding header category; Step 2.3: The system will use the anomaly marking algorithm to visually mark the abnormal data items in the generated Excel table.

10. The method for designing an intelligent handwriting recognition system integrating image modality perception according to claim 9, characterized in that: Step 2.3 specifically includes the following steps: Step 2.3.1: If there is any numerical data outside the predefined threshold range or the two models’ numerical recognition output does not match, it will be marked in the generated Excel table; Step 2.3.2: After completing the recognition processing of the handwritten text scan image dataset, the system automatically extracts the questionnaire names corresponding to the records marked as abnormal or erroneous in the Excel table and creates a dedicated abnormal data archiving folder in the specified storage path to facilitate subsequent manual inspection and error correction.

Images