A method and system for detecting exercise books based on cloud computing and artificial intelligence

Abstract

The present invention proposes a method and system for detecting exercise books based on cloud computing and artificial intelligence, including: obtaining an exercise book image of an exercise book to be detected and uploading it to a cloud server; using a text detection model to detect text area data in the exercise book image; inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result; inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected; determining the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information; based on a predetermined association rule, associating the correction symbol with the corresponding scoring area and the answer area; analyzing and processing the association result to obtain the exercise book detection result. The scheme of the present invention uses cloud computing and artificial intelligence technology to realize intelligent analysis of a large number of exercise book images, thereby improving the efficiency and quality of exercise book correction.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to an exercise book detection method and system based on cloud computing and artificial intelligence. Background Art

[0002] In the smart teaching system, the detection of exercise books is a heavy and important task. With the development of science and technology, the detection of exercise books can be intelligently transformed by using smart terminal technology, image processing technology, Internet technology, etc., but the current exercise book detection scheme is not efficient enough and the detection quality is not high. Summary of the invention

[0003] Based on the above problems, the present invention proposes a workbook detection method and system based on cloud computing and artificial intelligence. Through the scheme of the present invention, the scalability of cloud computing and artificial intelligence technology are utilized to realize intelligent analysis of a large number of workbook images, thereby improving the efficiency and quality of workbook correction.

[0004] In view of this, one aspect of the present invention proposes a workbook detection method based on cloud computing and artificial intelligence, comprising:

[0005] Scan the exercise book to be tested;

[0006] Obtaining a scanned exercise book image, and uploading the exercise book image to a cloud server;

[0007] performing image preprocessing on the workbook image at the cloud server to optimize image quality;

[0008] Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image;

[0009] Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results;

[0010] Input the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be tested;

[0011] Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information;

[0012] Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area;

[0013] The associated results are analyzed and processed to obtain the exercise book detection results.

[0014] Optionally, the method for constructing the text detection model includes:

[0015] Collect image datasets containing a variety of learning materials;

[0016] Annotating text regions in each image in the image dataset as a training dataset for model training;

[0017] Performing data enhancement on the data in the training data set;

[0018] Select the corresponding deep learning basic model;

[0019] Training the basic model on the training data set and performing hyperparameter tuning to optimize the performance of the basic model;

[0020] Collecting a corresponding test data set according to the training data set, evaluating model performance on the test data set, and adjusting the basic model according to the evaluation result to obtain a basic text detection model;

[0021] Analyze the operating status of the basic text detection model in different detection scenarios, determine the points that need to be improved, and make corresponding improvements to obtain the text detection model;

[0022] Using model compression technology to reduce the model size of the text detection model;

[0023] The trained text detection model is published as a callable cloud API service.

[0024] Optionally, the step of using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image comprises:

[0025] According to the input size requirement of the text detection model, scaling the preprocessed exercise book image;

[0026] Converting the scaled exercise book image into a format acceptable to the text detection model to obtain first exercise book image data;

[0027] inputting the first exercise book image data into the text detection model;

[0028] The text detection model outputs a prediction result;

[0029] Decoding the prediction result to obtain first coordinate data of each first text area;

[0030] According to the scaling ratio of the proportional scaling operation, the first coordinate data is de-mapped back to the original image size;

[0031] Optimizing the first text area using an image processing algorithm;

[0032] Draw a first text area frame corresponding to the detected first text area on the original image corresponding to the exercise book image;

[0033] Determine first size data of the first text area frame and first text area image data in the first text area frame;

[0034] The first coordinate data, the first size data and the first text area image data of the first text area are output as the text area data.

[0035] Optionally, the step of inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result includes:

[0036] Selecting a corresponding optical character recognition engine according to a recognition scenario corresponding to the text area data;

[0037] Extracting the first text area image data of each first text area from the text area data;

[0038] Preprocessing the first text area image data to improve OCR recognition accuracy;

[0039] inputting the preprocessed first text area image data into the optical character recognition engine;

[0040] The optical character recognition engine outputs a corresponding text recognition result string;

[0041] Normalizing the text recognition result character string to obtain a first text recognition result;

[0042] Setting a recognition confidence threshold, filtering out recognition results whose confidences are lower than the recognition confidence threshold in the first text recognition results, and obtaining a second text recognition result;

[0043] Correcting the misrecognition in the second text recognition result to obtain a third text recognition result;

[0044] Establishing a mapping relationship between the third text recognition result and the original image according to the coordinates, and generating a structured text recognition result with coordinate information and recognition confidence;

[0045] The text recognition result data is rendered in the original image.

[0046] Optionally, the step of inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected includes:

[0047] Converting the text recognition result into a first input sequence understandable by the language processing model;

[0048] Inputting the first input sequence into the language processing model to obtain a corresponding label sequence, a target string, and hierarchical information;

[0049] According to the tag sequence and the target character string, the corresponding key information and the category, text content and coordinate range corresponding to the key information are parsed;

[0050] Parsing the corresponding topic and / or topic section, and the first coordinate information corresponding to the topic and / or topic section from the key information and the category, text content and coordinate range corresponding to the key information;

[0051] Parsing a hierarchical structure tree corresponding to the question and / or question section according to the hierarchical information;

[0052] Mapping the first coordinate information of each of the questions and / or question sections into the hierarchical structure tree to obtain logical structure data including coordinates and hierarchies;

[0053] The parsed key information and the logical structure data are rendered on the original image.

[0054] Optionally, the method for constructing the language processing model includes:

[0055] Select an initial language model;

[0056] Pre-training the initial language model on a large-scale general corpus to obtain semantic and knowledge understanding capabilities in the corresponding field and obtain a basic language model;

[0057] Acquire multiple reference exercise book images and their corresponding annotation data, wherein the annotation data includes the position and category of key information such as question frame, answer area, scoring area, and correction symbol;

[0058] A first training text is obtained by recognizing the reference exercise book image using the optical character recognition engine;

[0059] Analyze the existing annotated exercise book data to obtain the basic logical structure model of the exercise book;

[0060] According to the basic logical structure mode, in the first training text, each training question / section is marked with its level to obtain a second training text, and the level is encoded to obtain encoding level information;

[0061] Using the second training text to train the basic language model to obtain a first language model;

[0062] Serialize the annotation information into an output format acceptable to the first language model;

[0063] Input the second language model and the serialized annotation data into a sequence annotation or sequence generation framework, and perform supervised fine-tuning on the second language model using the annotation data to obtain a third language model;

[0064] Input the encoded hierarchical information into the serialized word vectors to obtain a hierarchical sequence;

[0065] Use the hierarchical sequence as a supervision label to train the third language model to obtain a fourth language model;

[0066] Obtain a corresponding workbook test image according to the reference workbook image, and use the optical character recognition engine to recognize the workbook test image to obtain a first test text;

[0067] Use the first test text to evaluate the performance of the fourth language model, analyze the performance of the fourth language model on different types of key information, and obtain an evaluation feedback result;

[0068] Optimize the structure of the fourth language model according to the evaluation feedback result to obtain the language processing model.

[0069] Optionally, the step of determining the answer area, scoring area, and correction symbols according to the text area data, the logical structure data, and the key information includes:

[0070] Extract the first coordinate data, the first size data, and the first text area image data from the text area data;

[0071] According to the first coordinate data, the first size data, the first text area image data, the logical structure data, and the key information, identify the corresponding answer area by combining a preset answer area recognition rule, and the answer area recognition rule includes being adjacent to the question area, being an underlined or blank area, and the area size being within a preset range;

[0072] According to the first coordinate data, the first size data, the first text area image data, the logical structure data, and the key information, identify the scoring area by combining a preset scoring area feature pattern, and the scoring area feature pattern includes being adjacent to the answer area, having a score line or a score box, and having words such as "points" and " / ";

[0073] Associate the identified scoring area with the corresponding answer area;

[0074] Detect and identify the correction symbols in the first text area image data, and determine the symbol type and symbol coordinates of each of the correction symbols.

[0075] Optionally, the step of associating the correction symbol with the corresponding scoring area and the answer area based on a predetermined association rule includes:

[0076] Identify a page-level expansion box and a topic-level expansion box at the page level in the scoring area;

[0077] Calculating the intersection-and-joint ratio or the Euclidean distance between each correction symbol and the page-level expansion box or the topic-level expansion box according to the symbol type and the symbol coordinates;

[0078] Associating the first batch of symbols whose intersection-over-union ratio or Euclidean distance meets the association rule to the corresponding page-level expansion box or the topic-level expansion box;

[0079] For the unassociated second correction symbols in the correction symbols, classify them into corresponding big questions according to the big question attribution rule;

[0080] According to the answer blank association rule, the second batch correction symbols are subdivided and associated with specific answer blanks in the answer area.

[0081] Optionally, the association rule includes:

[0082] Page-level extension box association rules for associating full-page level revision symbols with extension boxes;

[0083] The topic-level extension box association rules for associating the correction symbols at the topic level with the excellent box, the division box, and the scoring box;

[0084] Question-level answer blank association rule for associating marking symbol segments to specific answer blanks.

[0085] In this embodiment, the association rule comprehensively considers multiple factors such as the intersection-over-union ratio, Euclidean distance, and relative position relationship between the correction symbol and each frame.

[0086] Another aspect of the present invention provides a workbook detection system based on cloud computing and artificial intelligence, comprising a scanning terminal and a cloud server; wherein:

[0087] The scanning terminal is configured to: scan the exercise book to be inspected, obtain the scanned exercise book image, and upload the exercise book image to the cloud server;

[0088] The cloud server is configured as follows:

[0089] performing image preprocessing on the exercise book image to optimize image quality;

[0090] Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image;

[0091] Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results;

[0092] Input the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be tested;

[0093] Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information;

[0094] Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area;

[0095] The associated results are analyzed and processed to obtain the exercise book detection results.

[0096] Adopting the technical scheme of the present invention, the exercise book detection method based on cloud computing and artificial intelligence includes: scanning the exercise book to be detected; obtaining the scanned exercise book image, and uploading the exercise book image to the cloud server; performing image preprocessing on the exercise book image on the cloud server to optimize the image quality; using a text detection model based on deep learning to detect the text area data in the preprocessed exercise book image; inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result; inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected; determining the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information; based on a predetermined association rule, associating the correction symbol with the corresponding scoring area and the answer area; analyzing and processing the association result to obtain the exercise book detection result. The scheme of the present invention utilizes the scalability of cloud computing and artificial intelligence technology to realize intelligent analysis of a large number of exercise book images, thereby improving the efficiency and quality of exercise book correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 is a flow chart of a method for detecting exercise books based on cloud computing and artificial intelligence provided by one embodiment of the present invention;

[0098] Figure 2 It is a schematic block diagram of an exercise book detection system based on cloud computing and artificial intelligence provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0099] In order to be able to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In addition, although the embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0100] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0101] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are only used for description purposes or to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features (or describing a specific order). Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0102] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0103] Refer to the following Figure 1 to Figure 2 To describe a method and system for detecting exercise books based on cloud computing and artificial intelligence provided according to some embodiments of the present invention.

[0104] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting exercise books based on cloud computing and artificial intelligence, comprising:

[0105] Scan the exercise book to be tested;

[0106] Obtaining a scanned exercise book image, and uploading the exercise book image to a cloud server;

[0107] performing image preprocessing on the workbook image at the cloud server to optimize image quality;

[0108] Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image;

[0109] Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results;

[0110] Input the text recognition result into a preset language processing model to extract the logical structure data and key information (including the question box, answer area, scoring area and correction symbols, etc.) of the exercise book to be tested;

[0111] Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information;

[0112] Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area;

[0113] The associated results are analyzed and processed to obtain the exercise book detection results.

[0114] The technical solution of this embodiment is adopted, by scanning the exercise book to be detected; obtaining the scanned exercise book image, and uploading the exercise book image to the cloud server; performing image preprocessing on the exercise book image on the cloud server to optimize the image quality; using a text detection model based on deep learning to detect the text area data in the preprocessed exercise book image; inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result; inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected; determining the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information; based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area; analyzing and processing the association result to obtain the exercise book detection result. The scheme of the present invention utilizes the scalability of cloud computing and artificial intelligence technology to realize intelligent analysis of a large number of exercise book images, thereby improving the efficiency and quality of exercise book correction.

[0115] In some possible implementations of the present invention, the method for constructing the text detection model includes:

[0116] Collect image datasets containing various learning materials (such as textbooks, exercise books, test papers, forms, etc.);

[0117] In this step, we collect images of learning materials such as exercise books and textbooks in various scenes and layouts to build an image dataset. These images should cover various situations that the model may encounter in practical applications, such as different fonts, sizes, directions, backgrounds, etc.

[0118] Annotating text regions in each image in the image dataset as a training dataset for model training;

[0119] In this step, for each image in the image data set, use the annotation tool to mark all the text areas in the image. The annotation content is usually to use a rectangular box to surround the area where each text line or phrase is located; the annotated result data is processed by necessary cleaning, format conversion, etc. to ensure that the annotated data can be read in parallel by the model training pipeline. The coordinates of the annotated box are usually serialized into a standard format; the processed annotated data and the text area image corresponding to the annotated text area are used as the training data set. After the above steps, a training data set containing a large number of annotated text areas is obtained, which can be used to train the text detection model.

[0120] Performing data enhancement (such as rotation, scaling, adding noise, etc.) on the data in the training data set;

[0121] Select the corresponding deep learning basic model (such as FRCN, EAST, PSENet and other classic text detection networks);

[0122] In this step, you can select a deep learning basic model with corresponding processing capabilities based on the specific fonts, subjects, etc. involved in the image training data set.

[0123] Training the basic model on the training data set and performing hyperparameter tuning to optimize the performance of the basic model;

[0124] Collect a corresponding test data set according to the training data set, evaluate the model performance (common indicators such as precision, recall, F1 score, etc.) on the test data set, and adjust the basic model according to the evaluation results to obtain a basic text detection model;

[0125] In this step, the corresponding test data set is collected according to the type and data volume of the training data set.

[0126] Analyze the operating status of the basic text detection model in different detection scenarios, determine the points that need to be improved, and make corresponding improvements to obtain the text detection model;

[0127] In this step, the operating status (detection efficiency, accuracy of detection results, etc.) of the basic text detection model in different detection scenarios (such as detecting exercise books with different fonts, subjects, or languages) is analyzed, and the problems existing in the basic text detection model (i.e., points that need to be improved) are determined according to the operating status, and corresponding improvements are made to obtain the text detection model.

[0128] Using model compression techniques (such as quantization, pruning, etc.) to reduce the model size of the text detection model to facilitate deployment in a resource-constrained cloud environment;

[0129] The trained text detection model is published as a callable cloud API service (such as through Web Service, etc.) to achieve rapid integration and elastic expansion of the model.

[0130] In this embodiment, during the model training process, technologies such as attention mechanism and context information can also be introduced to improve the robustness of the model to difficult scenes such as distortion, tilt, and occlusion; new data can be collected regularly to continuously improve the text detection model.

[0131] In some possible implementations of the present invention, the step of using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image includes:

[0132] According to the input size requirement of the text detection model, scaling the preprocessed exercise book image;

[0133] In this step, since the deep learning model has certain restrictions on the input image size, it is usually necessary to scale the original image proportionally first. The target size of the scaling is to meet the model input size requirements, such as 224x224 or other sizes.

[0134] Converting the scaled exercise book image into a format acceptable to the text detection model to obtain first exercise book image data;

[0135] inputting the first exercise book image data into the text detection model;

[0136] The text detection model outputs a prediction result (including first coordinate data of a first text area detected in the image);

[0137] Decoding the prediction result to obtain first coordinate data of each first text area;

[0138] According to the scaling ratio of the proportional scaling operation, the first coordinate data is de-mapped back to the original image size;

[0139] In this step, since the coordinate information of the detected text area output after the scaled image is input into the text detection model is based on the scaled image size, the coordinates need to be inversely mapped, that is, obtain the scale factor of the image scaling (such as the length and width are reduced by 0.5 times), divide the text area coordinates output by the model by this scale factor, and enlarge the coordinates to the size of the original image; the final result of the output is the text area coordinates based on the original image size; the detected text area can be rendered on the original image later. This inverse mapping process is necessary because it is ultimately necessary to accurately locate and further process the image at the original resolution. If this step is not performed, the coordinates of the text area will not correspond correctly to the original image. Through scaling and inverse mapping, the model can output detection results that match the high-resolution original image while meeting the input restrictions, ensuring the accuracy and practicality of the results.

[0140] Optimizing the first text area using an image processing algorithm (such as connected domain analysis, etc.);

[0141] In this step, the text detection model outputs preliminary text area detection frame results (i.e., the first text area parsed from the prediction results), but these detection frames may have some inaccuracies, such as overlap, line breaks, etc.; the image is binarized, and the connected domain analysis algorithm is used to obtain all connected text blocks in the image, and each connected domain may correspond to a word or text line; the text area detection frame predicted by the model is associated with the connected domain, and the association is performed based on heuristic rules such as spatial position and size; according to the association results, the text area detection frame is adjusted (such as splitting, merging, and expansion operations) to ensure that the adjusted frame can cover the text area as completely and accurately as possible; in addition to the connected domain analysis, other classic algorithms can be used to assist in optimization (such as projection contour analysis, small rectangle fitting, morphological operations, etc.) to further improve the accuracy of text positioning. By combining deep learning detection and traditional image processing algorithms, the advantages of both can be fully utilized to obtain more accurate text area detection results, laying the foundation for subsequent OCR recognition links.

[0142] Draw a first text area frame corresponding to the detected first text area on the original image corresponding to the exercise book image;

[0143] In this embodiment, after text detection and optimization processing, the text area finally detected is rendered back to the original exercise book image in the form of a rectangular frame (ie, the text area detection frame mentioned above or a rectangular frame generated according to the text area detection frame). Specifically, it can be divided into the following steps: obtain the original high-resolution image of the exercise book currently being processed from the data set; obtain the coordinate data of each detected text area in the output result of the text detection model, and the coordinates are usually in the rectangular box format of (x_min, y_min, x_max, y_max); if the original image has been scaled before, the coordinates of the text area need to be mapped back to the original image size; scale the coordinates with the scaling factor to ensure the accurate position of the rectangular box; on the canvas of the original image, use the drawing function to draw the corresponding rectangular box according to the mapped coordinate data, and set the line color, thickness and other attributes of the box to make it more eye-catching; in addition to the rectangular box, some additional annotation information (such as the OCR recognized text in the area, confidence score, etc.) can also be drawn next to the box; save or output the original image with the rectangular box of the text area drawn, which can be used to visually check the accuracy of the text detection results, and can also be used as the input of the subsequent processing link. Through this process, the results of text detection are presented in a visual form on the original exercise book image, which is helpful for manual inspection and optimization of the model, and also facilitates the subsequent data processing process.

[0144] Determine first size data of the first text area frame and first text area image data in the first text area frame;

[0145] In this step, after obtaining the coordinates of the rectangular frame of the text area output by the text detection model, the specific size of each text area and the corresponding image data slice can be determined based on these coordinate information. Specifically, it can be divided into the following steps: the text detection model outputs the coordinates of the rectangular frame of each detected text area, usually in the format of (xmin, ymin, xmax, ymax) that represents the coordinates of the upper left corner and the lower right corner; according to the coordinates of the rectangular frame, the width and height of the area can be easily calculated; the original high-resolution exercise book image to be processed is read from the data set, and the image data can be in numpy array or other common image formats; the rectangular frame coordinates are used as the start and end indexes of the slice, and the image data slices corresponding to the text area are sliced ​​from the original image data (the numpy index or the slicing function provided by other image processing libraries can be used); the obtained text area size (width and height) and image data slices are stored, and they can be stored together with the corresponding text area coordinates for easy association. Through this process, we not only obtain the precise positioning coordinates of each text area, but also the width and height information of the area, as well as the corresponding original image data slices. These data can provide the required input for subsequent OCR recognition, character area segmentation and other processing links. Accurately obtaining the text area size and image data is a key step in the transition from original image to text recognition, and has an important impact on the final recognition accuracy.

[0146] The first coordinate data, the first size data and the first text area image data of the first text area are output as the text area data.

[0147] In this embodiment, attention mechanism or context information can also be combined to improve the ability to handle difficult situations such as distortion and occlusion.

[0148] In this embodiment, the entire text detection process can be run in the cloud in the form of batch processing or microservices. The input is the uploaded exercise book image, and the output is the detected text area coordinate data, which lays the foundation for subsequent OCR recognition and semantic understanding.

[0149] In some possible implementations of the present invention, the step of inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result includes:

[0150] Selecting a corresponding optical character recognition engine according to a recognition scenario corresponding to the text area data;

[0151] In this step, when selecting an OCR (optical character recognition) engine, it is necessary to select the most suitable engine model according to the specific recognition scenario (such as handwriting, print, subject category, language category, etc.). Different OCR engines have different recognition capabilities and effects for different types of text, so it is very important to choose the right engine according to the actual situation. Specifically: according to the previously acquired text area image data, analyze the text type that needs to be recognized (such as handwritten print, machine print, handwritten curvilinear, subject category, language category and other different scenarios); engine selection (commercial OCR engines and open source engines have different focuses: commercial engines such as ABBYY, LEADTOOLS, etc., usually have good recognition capabilities for print and fixed format documents; open source engines such as Tesseract, CRNN, have good recognition performance for handwriting and irregular scenes; the appropriate engine model can be selected according to the technical solutions and positioning of different engines). After selecting the engine, the engine can be optimized for specific scenarios (such as training or fine-tuning corpus, fonts, etc. to improve the recognition of specific glyphs; adjusting recognition parameters, such as layout analysis, character filtering, etc.; developing or integrating pre-processing modules, such as automatic error correction, etc.); it can also combine multiple different OCR engines to give full play to their respective advantages (such as using handwriting recognition first, and then combining with the print engine to achieve complementary effects in different scenarios and improve overall accuracy). Through the above analysis and optimization, we can select the OCR engine or engine combination that best suits the current recognition scenario, thereby obtaining more accurate text recognition results and laying the foundation for subsequent key information extraction and other links. Reasonable selection and optimization of the OCR engine has an important impact on the effect of the entire exercise book intelligent correction process.

[0152] Extracting the first text area image data of each first text area from the text area data;

[0153] Preprocessing (such as correction, enhancement, etc.) the first text area image data to improve OCR recognition accuracy;

[0154] inputting the preprocessed first text area image data into the optical character recognition engine;

[0155] The optical character recognition engine outputs a corresponding text recognition result string;

[0156] Performing normalization processing (such as space adjustment, uppercase and lowercase conversion, etc.) on the text recognition result character string to obtain a first text recognition result;

[0157] Setting a recognition confidence threshold, filtering out recognition results whose confidences are lower than the recognition confidence threshold in the first text recognition results, and obtaining a second text recognition result;

[0158] In this step, after obtaining the initial text recognition result output by the OCR engine, the recognition result is filtered using the confidence threshold. Specifically, the following steps are included: the OCR engine recognizes the input text area image and outputs the text content of each recognized text area and its confidence score; a confidence threshold (such as 0.8) is set according to actual needs. This threshold is used to filter out results with low confidence and may be misrecognized; each recognition result is traversed, and the results with confidence scores lower than the set threshold are filtered out, and the recognition results with confidence scores higher than the threshold are retained; the filtered text recognition results (including the coordinates of each text area, the recognized text content and the confidence score) are output. By setting the confidence threshold and filtering the low-confidence results, the accuracy of text recognition can be effectively improved. Although the OCR engine is very powerful, some misrecognitions are inevitable, especially for difficult scenarios such as handwriting and illegible handwriting. The confidence score usually reflects the degree of certainty of the OCR engine in the recognition result. A reasonable threshold can only retain the OCR's relatively confident and reliable recognition results and filter out the poorer quality parts. The final output of the filtered high-quality text recognition results will lay the foundation for subsequent key information extraction, correction symbol association and other links.

[0159] Correcting misrecognition in the second text recognition result (in combination with a dictionary, a language model, etc.) to obtain a third text recognition result;

[0160] Establishing a mapping relationship between the third text recognition result and the original image according to the coordinates, and generating a structured text recognition result with coordinate information and recognition confidence;

[0161] In this step, the OCR text recognition result is associated with the original image to form a structured data output, which can be specifically divided into the following steps: obtain the filtered third text recognition result from the previous step (including the coordinates of each text area, the recognized text content and the confidence score), obtain the original high-resolution image data of the exercise book currently being processed, establish a mapping relationship between the text area coordinates in the text recognition result and the original image pixel coordinates, if the image has been scaled before, it is necessary to map the coordinates back to the original size, create an appropriate data structure (such as a list / dictionary, etc.), and store the coordinates of each text area (after mapping), recognized text content, confidence score and other information; further format normalization processing is performed on the structured data (such as storing coordinates in a certain format, using a standardized confidence value range, etc.) to facilitate subsequent data analysis and processing links to read and use; output the final structured text recognition result with coordinates and confidence (can be stored in common formats, such as JSON / XML, etc., or integrated as an API output of upstream services). Through this step, the original OCR recognition results can be associated with the image coordinates and stored and output in a structured form, so that the subsequent data processing and analysis can easily obtain and use these results. This format with coordinate information and confidence scores can also support rendering text recognition results on the original image for manual inspection of recognition quality. At the same time, the confidence also provides a basis for further recognition error correction process.

[0162] (Based on the detection result) rendering the text recognition result data in the original image.

[0163] In this embodiment, the OCR recognition process can be deployed and run in the cloud in the form of batch processing, with the input being the detected text area image and the output being the corresponding structured text data, laying the foundation for subsequent semantic understanding and content analysis; according to actual needs, an OCR model specially trained for scenarios such as handwriting can be used, or multiple different OCR engines can be integrated, which can further improve the recognition accuracy.

[0164] In some possible implementations of the present invention, the step of inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected includes:

[0165] Converting the text recognition result into a first input sequence that can be understood by the language processing model (for example, taking each line of text as a single input sample and including its coordinate information in the original image);

[0166] Inputting the first input sequence into the language processing model to obtain a corresponding label sequence, a target character string and hierarchical information (including a hierarchical structure tree of the entire exercise book to be tested);

[0167] In this embodiment, the OCR text recognition results are serialized and constructed into an input sequence acceptable to the language model (for example, text lines are encoded in sequence as word vectors or subword sequences, and metadata such as image coordinates are also encoded as part of the input); the constructed input sequence is input into a trained language processing model (the model can be a sequence labeling model (such as Transformer models such as BERT) or a sequence generation model; or it can include both a sequence labeling submodel and a sequence generation submodel); different forms of output are obtained according to different model architectures (the sequence labeling model will output a label sequence, and the sequence generation model will output a target string).

[0168] According to the tag sequence and the target character string, the corresponding key information and the category, text content and coordinate range corresponding to the key information are parsed;

[0169] In this step, the label sequence / string contains the hierarchical information corresponding to each text fragment; the label sequence or target string output by the model is parsed to identify the key information represented by each hierarchical label / phrase (such as chapter, main question, sub-question, question box, answer area, scoring area, correction symbol, etc.), as well as the category, text content and coordinate range corresponding to the key information.

[0170] The key information and the category, text content and coordinate range corresponding to the key information are parsed to obtain the corresponding question and / or question section, and the first coordinate information corresponding to the question and / or question section;

[0171] Parsing a hierarchical structure tree corresponding to the question and / or question section according to the hierarchical information;

[0172] Mapping the first coordinate information of each of the questions and / or question sections into the hierarchical structure tree to obtain logical structure data including coordinates and hierarchies;

[0173] In this embodiment, the hierarchical structure tree of the entire exercise book text content is reconstructed according to the parsing results; the parsed hierarchical structure tree is combined with the coordinate information in the OCR text recognition result to construct a structured data output containing coordinates and hierarchical relationships. Through this process, the language processing model can accurately extract and reconstruct its inherent logical hierarchical structure from the exercise book OCR text, laying the foundation for subsequent key information extraction, correction symbol association and other analyses. In the training process of the language processing model, in addition to the encoded text sequence, modal data in the form of image visual features, position prior knowledge, etc. can also be combined to further enhance the model's logical understanding ability.

[0174] The parsed key information and the logical structure data are rendered on the original image.

[0175] In this embodiment, it can be deployed in the cloud in batch processing, with the input being the OCR result text and the output being the extracted exercise book logical structure and key information data, which is efficient and intelligent, and provides a basis for accurate exercise book detection.

[0176] In some possible implementations of the present invention, the method for constructing the language processing model includes:

[0177] Select an initial language model (such as BERT, GPT, T5, RoBERTa, XLNet, etc.);

[0178] Pre-training the initial language model on a large-scale general corpus (such as books, encyclopedias, etc.) to obtain semantic and knowledge understanding capabilities in the corresponding field and obtain a basic language model;

[0179] Acquire multiple reference exercise book images and their corresponding annotation data, wherein the annotation data includes the position and category of key information such as question frame, answer area, scoring area, and correction symbol;

[0180] In this step, a variety of reference workbook images and their corresponding annotation data are obtained to prepare training data for training the language processing model to extract key information from the workbook, specifically including: collecting a large number of workbook images of different types and layouts, covering a variety of scenarios and difficulties, these images can be used as input for language model training; for each workbook image, use annotation tools to manually mark the following key information: question box (including different levels such as big questions, small questions, and sub-questions), answer area (the area where students actually write answers), score area (the area where teachers give scores, such as score boxes, scoring boxes, etc.), correction symbols (symbols used by teachers when correcting, such as check marks, crosses, etc.), the annotation content includes the specific coordinate position and category label of each area; the manual annotation results are processed by necessary cleaning, format conversion, etc. to ensure that the annotation data can be read and used in parallel by the model training pipeline. Through the above steps, a large amount of high-quality manually annotated data can be obtained, including the specific position coordinates and semantic category labels of each key information area. This data containing coordinates and semantic annotations will be used as supervised learning input for the language model, and the model will learn the ability to extract key information under the guidance of these annotated data.

[0181] A first training text is obtained by recognizing the reference exercise book image using the optical character recognition engine;

[0182] Analyze the existing annotated exercise book data to obtain the basic logical structure model of the exercise book (for example: one exercise book → multiple units / chapters → multiple questions in each unit → each question contains multiple small questions / sub-questions);

[0183] In this step, the previously collected and manually annotated exercise book data can be analyzed to obtain the basic logical structure model of the exercise book.

[0184] According to the basic logical structure mode, in the first training text, each training question / section is marked with its level (e.g., section A belongs to the first question of the third question of the second unit), a second training text is obtained, and the level is encoded to obtain the encoding level information;

[0185] Using the second training text to train the basic language model to obtain a first language model;

[0186] Serializing the annotation information into an output format acceptable to the first language model (such as BIO annotation, etc.);

[0187] In this step, the annotated data containing the OCR text and its coordinate positions is serialized, and the text is encoded into an input sequence acceptable to the model, such as a character / word vector sequence. The annotated data is serialized as a supervision label for model learning.

[0188] Input the second language model and the serialized labeled data into a sequence labeling or sequence generation framework, and perform supervised fine-tuning on the second language model on the labeled data (using appropriate loss functions, optimizers, and regularization strategies; paying attention to the alignment relationship between the input text and the annotations) to obtain a third language model;

[0189] In this step, the pre-trained language model is fine-tuned using the annotated data to obtain a fine-tuned model that can extract key information from the exercise book. Specifically, the following steps are included: aligning the model input sequence with the annotated sequence, handling possible alignment issues between text and annotation, and ensuring that the input and label are consistent in position and length; inputting the second language model and serialized data into the sequence annotation or sequence generation framework, determining the appropriate objective function (loss function, such as cross entropy loss, etc.), selecting the optimizer and regularization strategy, and performing several rounds of fine-tuning training iterations on the annotated data. After the training, a third language model fine-tuned on the annotated data is obtained. This new model can extract key information from the exercise book OCR text. Through supervised fine-tuning on a large amount of annotated data, the language model gradually learns how to identify key information such as the question box and answer area from the text, and establishes a mapping relationship with its position coordinates. The fine-tuned model will be integrated into the entire exercise book intelligent analysis processing flow as a key module.

[0190] Input the encoded level information into the serialized word vector to obtain a level sequence;

[0191] In this embodiment, the logical hierarchical relationship reflected in the text content of the exercise book is encoded into serialized data as a training target for the language model, which can be specifically divided into the following steps: annotating each text content in the training data set, annotating the logical level to which each text paragraph belongs (such as the entire exercise book, unit / chapter, major question, minor question / sub-question), and forming annotated corpus with hierarchical labels; designing corresponding label codes for different levels (such as using B-BOOK, I-UNIT, B-QSTN to represent the beginning and continuation of different levels); converting the text content into a sequence (which can be represented by character / word vectors, etc.) according to a certain segmentation granularity (such as characters / words, etc.); traversing the text sequence, assigning the corresponding hierarchical label code to the current word element according to the hierarchical label of the position, scanning all word elements from beginning to end, and constructing a complete hierarchical sequence; ensuring that the text sequence and the hierarchical label sequence are aligned one by one in length (appropriate completion / truncation operations may need to be performed). Through this series of encoding steps, we transformed the hierarchical logical structure of the exercise book text content into a hierarchical label sequence of the same length as the original text sequence. This hierarchical sequence will be used as the training target of the language model. The model will learn to predict the corresponding hierarchical information from the text sequence. During training, the model will try to maximize the fit between the predicted hierarchical sequence and the real hierarchical sequence annotated manually. Through this supervised sequence labeling method, the language model can master the ability to infer the logical hierarchical structure from the text, laying the foundation for the subsequent intelligent analysis of the exercise book.

[0192] The third language model is trained with the hierarchical sequence as a supervision label to obtain a fourth language model (in the training process, metadata such as the title and question number may also be encoded as a model input to improve the accuracy of structural parsing as a supervision signal for the third language model);

[0193] Acquire a corresponding exercise book test image according to the reference exercise book image, and obtain a first test text after recognizing the exercise book test image using the optical character recognition engine;

[0194] Using the first test text to evaluate the performance of the fourth language model (common indicators such as precision, recall, F1 value, etc.), analyzing the performance of the fourth language model on different types of key information, and obtaining evaluation feedback results;

[0195] According to the evaluation feedback result, the structure of the fourth language model is optimized to obtain the language processing model.

[0196] In this embodiment, the model performance can be enhanced by combining technologies such as attention mechanism and multi-task learning; visual features can be integrated to build a multimodal model; and model compression technologies such as quantization, distillation, and pruning can be used to reduce the model size, reduce computing resource consumption, and facilitate cloud deployment; it also includes: publishing the trained model as a callable cloud API service through Web Service and other methods to achieve elastic expansion of the model and meet high concurrent access requirements.

[0197] In this embodiment, the language model constructed through the above steps can more accurately understand and extract the logical hierarchical structure of the exercise book, laying the foundation for subsequent analysis such as question recognition and correction symbol association, and can efficiently and accurately extract key information such as the question box and answer area from the OCR text of the exercise book, laying the foundation for subsequent analysis such as correction symbol association.

[0198] In some possible implementations of the present invention, the step of determining the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information includes:

[0199] extracting the first coordinate data, the first size data and the first text area image data from the text area data;

[0200] In this step, the coordinate data, size data and image data slices of the text area are obtained.

[0201] According to the first coordinate data, the first size data, the first text area image data, the logical structure data and the key information, a corresponding answer area is identified in combination with a preset answer area identification rule, wherein the answer area identification rule includes one or more of being adjacent to the question area, being an underlined or blank area, and having an area size within a preset range;

[0202] In this step, based on the first coordinate data, the first size data, the first text region image data, the logical structure data, and the key information (such as the question box, scoring area, etc.); a set of answer area recognition rules based on heuristic rules are pre-set in advance. These rules include: being adjacent to the question area (within a first preset distance range), being an underlined or blank area, and the area size being within a preset range, or one or several of them; traverse all text regions, and according to the information such as the coordinates, size, and image data of each region, match them against the preset rules; if the rules are met, then this region is considered as the answer area; when matching the rules, combine the logical structure and key information (such as the position of the question box, question level, etc.) extracted by the language model to assist in judging the correspondence between the candidate answer area and the question; output the position coordinates, size, etc. information of the recognized answer area, and perform post-processing (such as merging missing answer blocks, etc.). Through this way of combining rules and structural information, the position of the answer area corresponding to the question can be more accurately recognized from a large number of text regions; these rules can be the experience summary of domain experts or can be learned from labeled data through machine learning methods. The recognized answer area data will provide important position prior information for subsequent steps such as associating marking symbols and automatic marking and scoring.

[0203] Based on the first coordinate data, the first size data, the first text region image data, the logical structure data, and the key information, the scoring area is recognized by combining the preset scoring area feature pattern. The scoring area feature pattern includes one or several of being adjacent to the answer area, having a score line or score box, and having words such as "points" or " / ".

[0204] In this step, based on the first coordinate data, the first size data, the first text region image data, the logical structure data, and the key information (such as the position of the answer area, etc.), a set of feature patterns for identifying the scoring area are preset in advance. These patterns include: adjacent to the answer area (within a second preset distance range), having a score line or a score box, having words such as "points" or " / ". Traverse all text regions. For each region, match the preset scoring area feature patterns according to its coordinates, size, image data, etc. If any pattern is satisfied, it is identified as the scoring area. When performing feature matching, the logical structure information extracted by the language processing model and the answer area identified in the previous step can also be combined to optimize and correct the scoring area recognition result. Output the position coordinates, size, etc. of the identified scoring area, and establish the association relationship between the scoring area and the corresponding answer area. By this way of feature pattern matching and integrating other context clues, the area position related to scoring can be more accurately mined from the exercise book image. These feature patterns can be set manually based on expert experience or learned through machine learning on labeled data. The accurately identified scoring area position will provide a position prior for correctly associating the marking symbols with the scores of the corresponding answer areas in the subsequent steps, which is crucial for the automatic scoring process.

[0205] Associate the identified scoring area with the corresponding answer area;

[0206] Detect and identify the marking symbols in the first text region image data, and determine the symbol type (such as tick, cross, etc.) and symbol coordinates (determined in combination with the first coordinate data) of each marking symbol.

[0207] In this step, a pre-trained deep learning-based marking symbol detection model can be used. Input the text region image data into the model, and the model will output the position box of the detected marking symbols. Apply OCR or symbol recognition technology to the image region of each detected marking symbol to identify the type corresponding to the image pattern in the symbol region (such as different types like tick, cross, number, text, etc.). Obtain the accurate coordinate position of the text region in the original image, and calculate the accurate pixel coordinates of the marking symbol on the original image in combination with the relative position of the marking symbol in the text region image. Structurally output the following information of each detected marking symbol: symbol type code (such as 1 for tick), pixel coordinates of the symbol on the original image (xmin, ymin, xmax, ymax), and an optional confidence score. Through this process, not only the marking symbols existing in the image are detected, but also the accurate type and accurate position coordinate information of each symbol on the original image are obtained, which lays a foundation for accurately spatially associating the detected marking symbols with key information such as the corresponding answer area and scoring area in the subsequent steps.

[0208] In this embodiment, the process can be deployed and run in the cloud, with the input being OCR text data, language processing model output results, and correction symbol detection results, and the output being structured answer areas, scoring areas, and correction symbol data.

[0209] In this embodiment, in the process of rule recognition, more image processing technologies, such as text line construction, layout analysis, etc., can be incorporated according to actual needs to improve the accuracy of answer area and scoring area recognition.

[0210] In some possible implementations of the present invention, the step of associating the correction symbol with the corresponding scoring area and the answer area based on a predetermined association rule includes:

[0211] Identify page-level extension boxes (such as total score boxes, rating boxes, comment boxes, etc.) and topic-level extension boxes (such as excellent boxes, division boxes, scoring boxes, etc.) at the page level in the scoring area;

[0212] In this step, in the entire scoring area, the page-level extension boxes are identified, including: the total score box - recording the total score of the whole page of exercises, the rating box - giving the grade evaluation of the whole page of exercises, and the comment box - the teacher's comment area for the whole page of exercises. In addition to the page level, the question-level extension boxes need to be identified in the scoring area, including: the excellent box - a box for checking the "excellent" or "typical error" option, the division box - a box for dividing a question into several parts according to the score, and the scoring box - the area where the teacher scores the question. Based on the visual features such as the position, size, and shape of the box, combined with the text clues in the OCR recognition results, the type and range of each box are identified and located using pre-designed rules or models based on data training, and the positions of the identified various types of extension boxes are correlated with the logical structure and key information extracted by the language model to establish a corresponding relationship between the extension box and the question and answer area. Through this process, more detailed extension information boxes can be revealed from the scoring area, and they can be logically linked with other elements in the exercise book.

[0213] Calculating the intersection-and-joint ratio or the Euclidean distance between each correction symbol and the page-level expansion box or the topic-level expansion box according to the symbol type and the symbol coordinates;

[0214] In this step, the pixel coordinates (xmin, ymin, xmax, ymax) of each correction symbol are obtained, the pixel coordinates of the page-level and title-level extension boxes are obtained, the intersection and union of the coordinate boxes of each correction symbol and the extension box are calculated, and the area ratio of the intersection and the union is calculated as the IoU value of the two (the IoU value is between 0 and 1, and the larger the value, the higher the overlap degree); the coordinate boxes of the correction symbol and the extension box are regarded as a point respectively, and the Euclidean distance between the two points is calculated based on the coordinates. The smaller the distance value, the higher the spatial proximity between the correction symbol and the extension box; considering the type of extension box and the characteristics of the correction symbol, the Euclidean distance can be appropriately weighted for simple symbols such as "checkmarks" to reduce the degree of influence of distance on such symbols; for each correction symbol, the IoU value and the adjusted distance value with all extension boxes are output, and these values ​​will be used for subsequent symbol association analysis. By calculating IoU and distance, the spatial position relationship between each correction symbol and the extension box can be quantified, laying the foundation for further associating the symbol to a specific frame. Generally speaking, an extended box with a larger IoU value or a closer distance is more likely to be associated with a symbol. This step is the key link to associate the detected correction symbols with the scoring area obtained by semantic understanding, which has an important impact on the subsequent automatic correction quality.

[0215] Associating the first batch of modified symbols whose intersection-over-union ratio or Euclidean distance meets the association rule to the corresponding page-level expansion box or the topic-level expansion box (considering factors such as the intersection, overlap ratio, and aspect ratio between the symbol and the expansion box);

[0216] In this step, based on the previously calculated spatial position relationship indicators such as the intersection over union (IoU) and Euclidean distance between the correction symbol and the expansion box, combined with a series of preset association rules, the correction symbols that meet the rules are associated with the corresponding page-level or title-level expansion box. Specifically, it can be divided into the following steps: A set of rules are pre-set to determine which expansion box to associate a certain correction symbol with. These rules comprehensively consider multiple factors: the IoU value (intersection over union) between the symbol and the box, the Euclidean distance between the symbol and the box, the intersection of the symbol and the box (whether there is overlap), the overlap ratio of the symbol and the box boundary, the length and width of the symbol and the box Compare; traverse each detected correction symbol, match it against the preset association rules according to its spatial relationship index with all extension boxes, and determine which specific extension box the symbol should be associated with; for correction symbols that meet the association rules, associate them with the corresponding page-level or title-level extension boxes, and record the association relationship; a symbol may meet multiple association rules, and you can set priorities for the rules, with higher-level rules taking precedence to avoid a symbol being repeatedly associated with multiple boxes; for special cases that cannot be matched by rules, set exception handling strategies, such as nearest neighbor association, to ensure that each symbol is reasonably associated as much as possible. Through this rule matching and association process, the detected correction symbols can be established with various extension information boxes in the exercise book, preparing for subsequent symbol analysis. These rules can be automatically learned through machine learning methods.

[0217] For the unassociated second correction symbols in the correction symbols, classify them into corresponding big questions according to the big question attribution rule;

[0218] In this step, for the unmatched second correction symbols, it is necessary to further classify and assign points according to the preset big question attribution rules. The factors considered in the big question attribution rules include the degree of intersection and distance between the correction symbols and the big questions and small questions. Whether the degree of intersection and / or the distance meets the preset conditions is used to judge whether the correction symbol belongs to a certain big question. The big question results judged by the big question attribution rules are used as the big questions to which this part of the second correction symbols belong, and they are assigned to the corresponding big questions. The purpose is to further reduce the number of unmatched correction symbols and increase the matching degree. So in general, the second round of classification and allocation is carried out for the unmatched second correction symbols in the first round according to the preset big question attribution standardization rules.

[0219] According to the answer blank association rule, the second batch correction symbols are subdivided and associated with specific answer blanks in the answer area.

[0220] In this step, the answer blank association rule is a rule that further subdivides and matches the correction symbols to specific answer blanks in the answer area (including considerations such as distance and degree of overlap); according to the answer blank association rule, the second batch of correction symbols are subdivided for a second round and matched to specific answer blanks. The "sharing mechanism" of the check mark symbol will be considered during the matching process, that is, whether a check mark symbol can match multiple answer blanks. The purpose is to use the answer blank association rule to further and more carefully match the second batch of correction symbols that have not been successfully matched to the corresponding answer blank, so as to maximize the matching degree. That is, according to the preset answer blank matching rule, the second batch of correction symbols that have not been matched in the first two rounds are subdivided and matched to specific answer blanks, and the sharing of the check mark symbol is handled at the same time.

[0221] In this embodiment, during the association process, it is necessary to pay attention to handling some extreme cases, such as the case where the same symbol partially intersects with multiple elements. A rule-based exception handling strategy can be introduced. In addition, the association rules involve many empirical threshold parameters, such as the intersection over union (IoU) threshold, distance threshold, etc., and a large-scale hyperparameter search can be performed on the annotated validation set to obtain the optimal parameter combination. The entire association process can be deployed and run in the cloud in the form of batch processing or microservices, with the input being the OCR recognition results, semantic understanding results, and correction symbol detection results, and the output being the associated structured data, laying the foundation for the intelligent correction of exercise books.

[0222] In some possible implementations of the present invention, the association rule includes:

[0223] Page-level extension box association rules for associating full-page level revision symbols with extension boxes;

[0224] The page-level extension box association rules specifically include: counting the entire page extension box and the teacher's marking box, correction box and other small question boxes at the small question level; calculating the intersection over union (IOU) of each correction symbol box with the extension box and the small question box; if the correction symbol has the largest intersection with the extension box, the correction symbol is included in the extension box; if there is no intersection with the extension box, the Euclidean distance between the correction symbol and the extension box and the small question box is calculated; if the minimum distance to both types of boxes is less than the threshold, the correction symbol box is removed; the associated correction symbols are removed, and the remaining correction symbols are used for subsequent association; the coordinate conversion is also performed with reference to the scaling ratio of the width and height of the annotation data image; finally, the intersection over union and distance calculation are used to match the correction symbols within the entire page to the extension box to achieve page-level matching; the above rules are mainly used to match the correspondence between correction symbols and extension boxes within the entire page.

[0225] The topic-level extension box association rules for associating the correction symbols at the topic level with the excellent box, the division box, and the scoring box;

[0226] Specifically, the question-level extension box includes the excellent box (excellent, typical wrong box), the division box (score box) and the scoring box under the small question. Among them, the association rules of the excellent box and the division box are the same: calculate the intersection ratio of the correction symbol and the outer box (a large box covering multiple small boxes), calculate the product of the overlap ratio of the correction symbol and the internal small box in the horizontal and vertical directions, and calculate the width ratio of the correction symbol to the small box; if the above three conditions all meet the preset threshold, the correction symbol is classified into the corresponding small box. Regarding the scoring box: classify the correction symbols into digital symbols and candidate symbols; filter the digital symbols and match them to the scoring box with the largest score; filter the candidate symbols, and match the matching content to the score box if the numerical value is not associated with the score box. In this embodiment, the intersection ratio and position relationship are used as a whole to match the correction symbols to different types of question-level extension boxes.

[0227] Question-level answer blank association rule for associating marking symbol segments to specific answer blanks.

[0228] The question-level answer blank association rules specifically include the following: calculate the vertical distance, intersection ratio (IOU), horizontal and vertical position relationship and other features between each correction symbol and the answer blank; select an answer blank with the largest intersection or the smallest distance for matching; if the previous step did not match successfully, determine whether the minimum intersection meets the threshold, and match the answer blank with the minimum intersection; adjust the distance for the unmatched answer blank, and match with the position of the sub-question to which the answer blank belongs. Process the check mark sharing mechanism: determine whether the check mark sharing conditions are met (such as the only check mark in the sub-question and multiple answer blanks), and when matching unassigned answer blanks, select the nearest shareable check mark for matching. In this embodiment, the geometric position relationship and degree of intersection between the answer blank and the correction symbol are considered as a whole to determine the matching relationship; the check mark sharing mechanism supports one check mark matching multiple answer blanks to increase the matching rate.

[0229] In this embodiment, the association rule comprehensively considers multiple factors such as the intersection-over-union ratio, Euclidean distance, and relative position relationship between the correction symbol and each frame.

[0230] See also Figure 2 Another embodiment of the present invention provides a workbook detection system based on cloud computing and artificial intelligence, including a scanning terminal and a cloud server; wherein,

[0231] The scanning terminal is configured to: scan the exercise book to be inspected, obtain the scanned exercise book image, and upload the exercise book image to the cloud server;

[0232] The cloud server is configured as follows:

[0233] performing image preprocessing on the exercise book image to optimize image quality;

[0234] Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image;

[0235] Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results;

[0236] Input the text recognition result into a preset language processing model to extract the logical structure data and key information (including the question box, answer area, scoring area and correction symbols, etc.) of the exercise book to be tested;

[0237] Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information;

[0238] Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area;

[0239] The associated results are analyzed and processed to obtain the exercise book detection results.

[0240] This embodiment uses the scalability of cloud computing and artificial intelligence technology to realize intelligent analysis of a large number of exercise book images, thereby improving the efficiency and quality of exercise book marking.

[0241] It should be known that Figure 2 The block diagram of the exercise book detection system based on cloud computing and artificial intelligence is for illustration only, and the number of modules shown therein does not limit the protection scope of the present invention.

[0242] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0243] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0244] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0245] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0246] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0247] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0248] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0249] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0250] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A method for detecting exercise books based on cloud computing and artificial intelligence, characterized in that: include: Scan the exercise book to be tested; Obtaining a scanned exercise book image, and uploading the exercise book image to a cloud server; performing image preprocessing on the workbook image at the cloud server to optimize image quality; Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image; Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results; Input the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be tested; Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information; Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area; Analyze and process the correlation results to obtain the exercise book detection results; The step of inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected includes: Converting the text recognition result into a first input sequence understandable by the language processing model; Inputting the first input sequence into the language processing model to obtain a corresponding label sequence, a target string, and hierarchical information; According to the tag sequence and the target character string, the corresponding key information and the category, text content and coordinate range corresponding to the key information are parsed; Parsing the corresponding topic and / or topic section, and the first coordinate information corresponding to the topic and / or topic section from the key information and the category, text content and coordinate range corresponding to the key information; Parsing a hierarchical structure tree corresponding to the question and / or question section according to the hierarchical information; Mapping the first coordinate information of each of the questions and / or question sections into the hierarchical structure tree to obtain logical structure data including coordinates and hierarchies; The parsed key information and the logical structure data are rendered on the original image corresponding to the exercise book image.

2. The exercise book detection method based on cloud computing and artificial intelligence according to claim 1, characterized in that: The method for constructing the text detection model includes: Collect image datasets containing a variety of learning materials; Annotating text regions in each image in the image dataset as a training dataset for model training; Performing data enhancement on the data in the training data set; Select the corresponding deep learning basic model; Training the basic model on the training data set and performing hyperparameter tuning to optimize the performance of the basic model; Collecting a corresponding test data set according to the training data set, evaluating model performance on the test data set, and adjusting the basic model according to the evaluation result to obtain a basic text detection model; Analyze the operating status of the basic text detection model in different detection scenarios, determine the points that need to be improved, and make corresponding improvements to obtain the text detection model; Using model compression technology to reduce the model size of the text detection model; The trained text detection model is published as a callable cloud API service.

3. The exercise book detection method based on cloud computing and artificial intelligence according to claim 2 is characterized in that: The step of using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image comprises: According to the input size requirement of the text detection model, scaling the preprocessed exercise book image; Converting the scaled exercise book image into a format that can be processed by the text detection model to obtain first exercise book image data; inputting the first exercise book image data into the text detection model; The text detection model outputs a prediction result; Decoding the prediction result to obtain first coordinate data of each first text area; According to the scaling ratio of the proportional scaling operation, the first coordinate data is de-mapped back to the original image size; Optimizing the first text area using an image processing algorithm; Draw a first text area frame corresponding to the detected first text area on the original image corresponding to the exercise book image; Determine first size data of the first text area frame and first text area image data in the first text area frame; The first coordinate data, the first size data and the first text area image data of the first text area are output as the text area data.

4. The exercise book detection method based on cloud computing and artificial intelligence according to claim 3 is characterized in that: The step of inputting the detected text area data into an optical character recognition engine to generate a corresponding text recognition result includes: Selecting a corresponding optical character recognition engine according to a recognition scenario corresponding to the text area data; Extracting the first text area image data of each first text area from the text area data; Preprocessing the first text area image data to improve OCR recognition accuracy; inputting the preprocessed first text area image data into the optical character recognition engine; The optical character recognition engine outputs a corresponding text recognition result string; Normalizing the text recognition result character string to obtain a first text recognition result; Setting a recognition confidence threshold, filtering out recognition results whose confidences are lower than the recognition confidence threshold in the first text recognition results, and obtaining a second text recognition result; Correcting the misrecognition in the second text recognition result to obtain a third text recognition result; Establishing a mapping relationship between the third text recognition result and the original image according to the coordinates, and generating a structured text recognition result with coordinate information and recognition confidence; The text recognition result data is rendered in the original image.

5. The exercise book detection method based on cloud computing and artificial intelligence according to claim 4 is characterized in that: The method for constructing the language processing model includes: Select an initial language model; Pre-training the initial language model on a large-scale general corpus to obtain semantic and knowledge understanding capabilities in the corresponding field and obtain a basic language model; Acquire multiple reference exercise book images and their corresponding annotation data, wherein the annotation data includes the position and category of key information such as question frame, answer area, scoring area, and correction symbol; Using the optical character recognition engine to recognize the reference exercise book image to obtain a first training text; Analyze the existing annotated exercise book data to obtain the basic logical structure model of the exercise book; According to the basic logical structure pattern, in the first training text, mark the level where each training question / question segment is located to obtain the second training text, and encode the levels to obtain encoded level information; Use the second training text to train the basic language model to obtain the first language model; Serialize the annotation data into an output format acceptable to the first language model; Input the first language model and the serialized annotation data into a sequence annotation or sequence generation framework, and perform supervised fine-tuning on the first language model using the annotation data to obtain the third language model; Input the encoded level information into the serialized word vector to obtain a level sequence; Train the third language model using the level sequence as the supervision label to obtain the fourth language model; Obtain the corresponding workbook test image according to the reference workbook image, and use the optical character recognition engine to recognize the workbook test image to obtain the first test text; Use the first test text to evaluate the performance of the fourth language model, analyze the performance of the fourth language model on different types of key information, and obtain an evaluation feedback result; Optimize the structure of the fourth language model according to the evaluation feedback result to obtain the language processing model.

6. The exercise book detection method based on cloud computing and artificial intelligence according to claim 5, characterized in that: The step of determining the answer area, scoring area, and correction symbol according to the text area data, the logical structure data, and the key information includes: Extract the first coordinate data, the first size data, and the first text area image data from the text area data; Identify the corresponding answer area according to the first coordinate data, the first size data, the first text area image data, the logical structure data, and the key information, in combination with a preset answer area recognition rule, where the answer area recognition rule includes being adjacent to the question area, being an underlined or blank area, and the area size being within a preset range, or a combination of one or more of them; Identify the scoring area according to the first coordinate data, the first size data, the first text area image data, the logical structure data, and the key information, in combination with a preset scoring area feature pattern, where the scoring area feature pattern includes being adjacent to the answer area, having a score line or score box, and having words such as "points" or " / "; Associate the identified scoring area with the corresponding answer area; Detect and identify the correction symbols in the first text area image data, and determine the symbol type and symbol coordinates of each correction symbol.

7. The exercise book detection method based on cloud computing and artificial intelligence according to claim 6, characterized in that: The step of associating the correction symbol with the corresponding scoring area and answer area based on a pre-determined association rule includes: Identify the page-level page-level expansion box and question-level expansion box in the scoring area; Calculate the intersection over union or Euclidean distance between each correction symbol and the page-level expansion box or question-level expansion box according to the symbol type and the symbol coordinates; Associate the first batch of correction symbols whose intersection over union or Euclidean distance meets the association rule with the corresponding page-level expansion box or question-level expansion box; For the unassociated second correction symbols in the correction symbols, classify them into corresponding big questions according to the big question attribution rule; According to the answer blank association rule, the second batch correction symbols are subdivided and associated with specific answer blanks in the answer area.

8. The exercise book detection method based on cloud computing and artificial intelligence according to claim 7, characterized in that: The association rules include: Page-level extension box association rules for associating full-page level revision symbols with extension boxes; The topic-level extension box association rules for associating the correction symbols at the topic level with the excellent box, the division box, and the scoring box; Question-level answer-blank association rules for associating grading symbol segments to specific answer blanks; The association rules comprehensively consider the intersection-and-union ratio, Euclidean distance, and relative position relationship between the correction symbols and each frame.

9. A workbook detection system based on cloud computing and artificial intelligence, characterized in that: Including scanning terminal and cloud server; The scanning terminal is configured to: scan the exercise book to be inspected, obtain the scanned exercise book image, and upload the exercise book image to the cloud server; The cloud server is configured as follows: performing image preprocessing on the exercise book image to optimize image quality; Using a deep learning-based text detection model to detect text area data in the preprocessed exercise book image; Inputting the detected text area data into an optical character recognition engine to generate corresponding text recognition results; Input the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be tested; Determine the answer area, the scoring area and the correction symbol according to the text area data, the logical structure data and the key information; Based on a predetermined association rule, the correction symbol is associated with the corresponding scoring area and the answer area; Analyze and process the correlation results to obtain the exercise book detection results; The step of inputting the text recognition result into a preset language processing model to extract the logical structure data and key information of the exercise book to be detected includes: Converting the text recognition result into a first input sequence understandable by the language processing model; Inputting the first input sequence into the language processing model to obtain a corresponding label sequence, a target string, and hierarchical information; According to the tag sequence and the target character string, the corresponding key information and the category, text content and coordinate range corresponding to the key information are parsed; Parsing the corresponding topic and / or topic section, and the first coordinate information corresponding to the topic and / or topic section from the key information and the category, text content and coordinate range corresponding to the key information; Parsing a hierarchical structure tree corresponding to the question and / or question section according to the hierarchical information; Mapping the first coordinate information of each of the questions and / or question sections into the hierarchical structure tree to obtain logical structure data including coordinates and hierarchies; The parsed key information and the logical structure data are rendered on the original image corresponding to the exercise book image.

Images