A primary school student handwriting composition multi-modal OCR identification method and system

Abstract

The application belongs to the technical field of character recognition, and discloses a primary school student handwriting composition multi-modal OCR recognition method and system, which comprises the following steps: obtaining a primary school student handwriting composition image; obtaining a potential modification area according to stroke connection and interval distribution in the composition image; obtaining a modification tracking trajectory of each potential modification area according to connection change and marking mode of adjacent strokes in each composition image; screening an actual modification area based on the modification tracking trajectory; obtaining an intention association mapping of the corresponding actual modification area according to the modification tracking trajectory of each actual modification area in each composition image; constructing a structured text sequence of the composition image based on the intention association mapping; detecting expression intention and creation thought in the primary school student handwriting composition through the structured text sequence to generate a multi-modal recognition result; and greatly improving the accuracy of handwriting composition recognition.

Description

Technical Field

[0001] This invention relates to the field of character recognition technology, and more specifically, to a multimodal OCR recognition method and system for elementary school students' handwritten compositions. Background Technology

[0002] With the rapid development of digital transformation in education and intelligent grading systems, the automated recognition and understanding of primary school students' handwritten compositions has become an important breakthrough in educational technology innovation. In Chinese language teaching practice, composition grading not only requires recognizing the text content, but also understanding students' expressive intentions and creative ideas, which is of great significance for cultivating students' interest in writing and improving teaching efficiency. In order to achieve intelligent assistance and personalized feedback in composition teaching, in recent years, OCR technology based on deep learning has been widely explored in educational scenarios. Through technologies such as convolutional neural networks and sequence modeling, handwritten text is converted into structured digital content, providing a technical foundation for subsequent semantic analysis, writing assessment, and teaching diagnosis.

[0003] However, existing OCR recognition technology faces a unique contradiction between the irregularities in handwriting and semantic coherence when processing handwritten compositions by elementary school students. This problem is particularly complex in the compositions of lower and middle grade students. Specifically, elementary school students generally exhibit illegible handwriting, irregular stroke connections, and uneven spacing between words in their compositions, accompanied by numerous corrections, markings, and insertions. For example, students frequently use arrows to indicate word and phrase changes, use wavy lines to mark content to be deleted, or densely insert missing text between lines. These corrections not only reflect the students' thought process but also serve as an important indicator of their writing ability development. However, the transmission of... Traditional OCR systems often filter out correction marks and insertion marks as noise when recognizing complex handwriting styles, or produce incorrect text sequences because they cannot correctly interpret the modification logic. The common approach to solving this problem is to ask students to rewrite a clean version or to improve text clarity through image enhancement algorithms. However, this approach not only increases the burden on students but also loses the information about the thought process in the original writing process, failing to truly reflect the evolution of students' writing ideas. More importantly, it ignores the important value of these correction marks for teachers to understand students' writing difficulties and thinking characteristics, ultimately limiting the application effect and educational value of intelligent essay correction systems in real teaching scenarios.

[0004] In view of this, the present invention proposes a multimodal OCR recognition method and system for elementary school students' handwritten compositions to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the existing technology and to achieve the above objectives, the present invention provides the following technical solution: a multimodal OCR recognition method for handwritten compositions by primary school students, comprising:

[0006] Step S1: Obtain images of elementary school students' handwritten compositions, with each composition image corresponding to a student's complete writing page;

[0007] Step S2: Based on the stroke connections and spacing distribution in the essay images, obtain potential modification areas; based on the connection changes and marking patterns of adjacent strokes in each essay image, obtain the modification tracking trajectory for each potential modification area; filter actual modification areas based on the modification tracking trajectory.

[0008] Step S3: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping;

[0009] Step S4: Detect the expressive intent and creative ideas in the handwritten compositions of primary school students through the structured text sequence, and generate multimodal recognition results.

[0010] Furthermore, the method for obtaining the potential modification region includes:

[0011] Endpoint tracing is performed on all strokes in the composition images to identify the start and end positions of each stroke;

[0012] Based on the difference in endpoint positions and the distribution of connection points between any two strokes in the composition image, obtain the structural difference degree between the corresponding two strokes;

[0013] A continuous block of strokes in the composition image whose structural difference is less than a preset structural difference threshold is denoted as a local stroke group.

[0014] The preset merging conditions include: the distance between endpoints of groups is less than a preset threshold and the distribution of connection points is similar; the local stroke groups in the composition image are merged, and adjacent groups that meet the preset merging conditions are included in the merge, until all groups no longer meet the preset merging conditions, and the process stops to obtain the potential modification area.

[0015] Furthermore, the endpoint tracking of strokes in all composition images, identifying the start and end positions of each stroke, includes:

[0016] Each composition image is decomposed into a stroke layer and a background layer through layer-by-layer image segmentation; each stroke in the stroke layer is dynamically tracked, and the continuous pixel sequence of the stroke is detected line by line from the starting scan line of the image, and the first pixel group of the continuous pixel sequence is marked as the potential starting position; according to the pixel density distribution and direction change of the stroke, the expansion path of each potential starting position is calculated until the breakpoint of the continuous pixel sequence is detected and the tracking is paused.

[0017] The breakpoint where tracking is paused is compared with the reconnection point of the subsequent continuous pixel sequence. When the pixel overlap rate between the breakpoint and the connection point is greater than the preset overlap threshold, the breakpoint is adjusted to the termination position. When the pixel overlap rate between the breakpoint and the connection point is less than or equal to the preset overlap threshold, the breakpoint is reconnected through iterative smoothing.

[0018] Repeat the above process until the pixel sequences in all stroke layers have been processed to obtain the start and end positions of each stroke.

[0019] Furthermore, the step of obtaining the structural difference degree between two strokes based on the difference in endpoint positions and the distribution of connection points between any two strokes in the composition image includes:

[0020] Coordinate mapping is performed on the endpoints of any two strokes in the composition image, the start and end positions are represented as two-dimensional coordinate points, and the Euclidean distance between the endpoints of the two strokes is calculated as the initial position difference value.

[0021] Cluster analysis is performed on the distribution of connection points for each stroke to extract the density distribution map of connection points. Based on the density distribution map, the spatial overlap of connection points between any two strokes is evaluated using a weighted kernel function.

[0022] The initial position difference value and spatial overlap are weighted and fused to obtain the structural difference between different strokes.

[0023] Furthermore, obtaining the modification tracking trajectory for each potential modification region includes:

[0024] The strokes of the characters in all the essay images are scanned line by line to identify the endpoints and connection points of each stroke;

[0025] The number of endpoint overlaps and spacing differences of adjacent strokes in each composition image are counted and marked as the connection irregularity value of each stroke;

[0026] In each composition image, the stroke regions with non-standard connection values ​​exceeding the preset standard threshold are used as initial path points;

[0027] The preset connection conditions include that the distance between stroke endpoints is less than a preset distance and the difference in spacing is less than a preset difference threshold. By expanding point by point from the initial path point, adjacent strokes that meet the preset connection conditions are included in the path expansion until all adjacent strokes no longer meet the preset connection conditions, and the expansion stops, thus obtaining the modification tracking trajectory of each potential modification area in each composition image.

[0028] Furthermore, the method for obtaining the actual modified area includes:

[0029] The labeling pattern analysis is performed on the potential modification regions in each essay image to identify the labeling type; the path correlation between the two regions is obtained based on the labeling type similarity and positional association of any two potential modification regions in the essay image.

[0030] Potentially modified regions in the essay image with a path correlation greater than a preset correlation threshold are merged into candidate regions.

[0031] The candidate regions are expanded by including regions containing additional markers within the boundaries until all boundary regions no longer contain additional markers, thus obtaining the actual modified regions in each composition image.

[0032] Furthermore, the method for obtaining the intent association mapping includes:

[0033] Determine the writing structure of each actual modification area in each composition image, including stroke order and marking direction;

[0034] The writing structure is decomposed layer by layer, and the marking logic of each level is extracted; a temporary intent chain is constructed based on the modification tracking trajectory of each actual modification area and the extracted marking logic;

[0035] Associate logical nodes in the temporary intent chain with the writing structure of adjacent regions to generate intent association mappings for each actual modified region.

[0036] Further, the construction of the structured text sequence of the essay image based on the intent association mapping includes:

[0037] Starting from the actual modified area of ​​each essay image, extract the tagging logic from the intent association mapping one by one;

[0038] Reorganize the stroke sequences of adjacent unmodified regions according to the marking logic; integrate the reorganized stroke sequences with the original writing structure to form a preliminary text framework;

[0039] The initial text framework is rearranged in order, and the missing content indicated by the insertion mark is incorporated into the framework according to its positional priority until all the inserted content is integrated, resulting in a structured text sequence of the composition image.

[0040] Furthermore, the step of detecting the expressive intent and creative thinking in elementary school students' handwritten compositions through the structured text sequence includes:

[0041] The system analyzes the marked logical sequence in the structured text sequence; extracts the intent key points based on the association between the marked logical sequence and the original stroke order; integrates the extracted intent key points with the structured text sequence to form the final multimodal recognition result, thereby achieving complete recognition of the intention expressed by primary school students.

[0042] A multimodal OCR recognition system for elementary school students' handwritten compositions includes:

[0043] Data acquisition module: Acquires images of handwritten compositions by elementary school students, with each composition image corresponding to a complete page of the student's writing;

[0044] Region filtering module: Based on the stroke connections and spacing distribution in the essay image, potential modification regions are obtained; based on the connection changes and marking patterns of adjacent strokes in each essay image, modification tracking trajectories of each potential modification region are obtained; actual modification regions are filtered based on the modification tracking trajectories.

[0045] Intent recognition module: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping;

[0046] Result recognition module: Detects the expressive intent and creative ideas in elementary school students' handwritten compositions through the structured text sequence, and generates multimodal recognition results.

[0047] The technical effects and advantages of this invention, a multimodal OCR recognition method and system for elementary school students' handwritten compositions, are as follows:

[0048] This invention identifies potential modification areas through stroke connections and spacing distribution. Changes in the connection of adjacent strokes and marking patterns reflect the characteristics of students' modification behavior. By acquiring modification tracking trajectories and filtering actual modification areas, it avoids the problem of traditional OCR systems filtering modification traces as noise, thus preserving the students' thought process information during writing. The modification tracking trajectory reflects the students' thinking process and modification logic at specific locations. By analyzing the trajectory and constructing an intent association mapping, the originally messy modification marks are transformed into ordered logical relationships, enabling accurate parsing of modification intentions such as word and phrase changes indicated by arrows and deletions marked by wavy lines. Based on the intent association mapping, a structured text sequence is constructed, which not only identifies the text content but, more importantly, reconstructs the evolution of students' creative thinking, achieving a leap from surface text recognition to deep intent understanding. By detecting expressive intent and creative thinking, multimodal recognition results are generated, providing teachers with complete information on students' writing difficulties and thinking characteristics. This overcomes the shortcomings of existing technologies that fail to truly reflect the development of students' writing abilities by ignoring modification traces, thereby enhancing the application value and educational effectiveness of intelligent essay correction systems in real teaching scenarios. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a multimodal OCR recognition method for handwritten compositions by primary school students according to the present invention;

[0050] Figure 2 This is a schematic diagram of a multimodal OCR recognition system for elementary school students' handwritten compositions according to the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1

[0053] Please see Figure 1 As shown in this embodiment, a multimodal OCR recognition method for elementary school students' handwritten compositions includes:

[0054] Step S1: Obtain images of elementary school students' handwritten compositions. Each composition image corresponds to a student's complete writing page.

[0055] In elementary school students' daily learning, handwritten compositions are an important way to train language expression. Students frequently make revisions during the writing process, including erasing, inserting, and rearranging text. These revision marks reflect the students' thought processes and creative ideas. Traditional OCR technology mainly focuses on character recognition and struggles to accurately understand the intentions behind these revisions. Therefore, this solution identifies students' expressive intentions by analyzing these revision marks.

[0056] Handwritten compositions by elementary school students were scanned using a high-resolution scanner to obtain images with a resolution of at least 300 dpi. Each composition image contained a complete page of writing, typically on A4 or B5 size writing paper. The scanning process ensured that the images were clear and accurately reflected stroke details and corrections.

[0057] It should be noted that the image of the composition in this embodiment is a grayscale image; other embodiments may also use a color image. For situations where different colored pens are used for modifications, a color image can provide more information about the modifications.

[0058] Step S2: Based on the stroke connections and spacing distribution in the essay image, obtain potential modification areas; based on the connection changes and marking patterns of adjacent strokes in each essay image, obtain the modification tracking trajectory of each potential modification area; filter the actual modification areas based on the modification tracking trajectory.

[0059] When elementary school students revise their compositions, the strokes in the correction areas often exhibit characteristics different from normal writing. For example, the stroke density in the correction areas is higher, the stroke direction of the insertion marks is different from the main text, and the transposition marks (such as arrows) have specific morphological features. By analyzing these characteristics, potential correction areas can be identified.

[0060] Preferably, the method for obtaining the potential modification region includes: tracing the endpoints of strokes in all essay images to identify the start and end positions of each stroke; obtaining the structural difference degree between any two strokes based on the difference in endpoint positions and the distribution of connection points in the essay image; recording continuous blocks formed by strokes in the essay image whose structural difference degree is less than a preset structural difference threshold as local stroke groups; setting preset merging conditions, including: the endpoint distance between groups is less than a preset threshold and the distribution of connection points is similar; merging local stroke groups in the essay image, including adjacent groups that meet the preset merging conditions, until all groups no longer meet the preset merging conditions, thus obtaining the potential modification region; the similarity of the distribution of connection points between groups is based on the minimum structural difference judgment. For strokes in two groups, the two strokes with the smallest structural difference degree are matched as the minimum difference group, and the mean of the structural difference degree of all minimum difference groups is calculated as the minimum structural difference. When the minimum structural difference is less than the preset group difference, it indicates that the distribution of connection points between groups is similar.

[0061] The specific implementation method of endpoint tracking is as follows: Each composition image is decomposed into a stroke layer and a background layer through layer-by-layer image segmentation. The stroke layer initially isolates the stroke contours based on grayscale threshold and edge gradient analysis. Each stroke in the stroke layer is dynamically tracked. The continuous pixel sequence of the stroke is detected line by line from the starting scan line of the image, and the first pixel group of the continuous pixel sequence is marked as the potential starting position. According to the pixel density distribution and direction change of the stroke, the expansion path of each potential starting position is calculated until a breakpoint of the continuous pixel sequence is detected, and the tracking is paused. The breakpoint of the paused tracking is compared with the reconnection point of the subsequent continuous pixel sequence. When the pixel overlap rate between the breakpoint and the connection point is greater than the preset overlap threshold, the breakpoint is adjusted to the termination position. When the pixel overlap rate between the breakpoint and the connection point is less than or equal to the preset overlap threshold, the breakpoint is reconnected through iterative smoothing. The above process is repeated until the pixel sequence in all stroke layers has been processed to obtain the start and end positions of each stroke.

[0062] In this embodiment, the grayscale threshold is set to 128, the preset overlap threshold is set to 0.7, the preset threshold for the distance between endpoints between groups is set to 5 pixels, and the preset group difference is set to 0.2. These parameters are derived from statistical analysis of a large number of primary school students' handwritten samples.

[0063] The method for calculating structural difference is as follows: Coordinate mapping is performed on the endpoints of any two strokes in the writing image, representing the start and end positions as two-dimensional coordinate points, and the Euclidean distance between the endpoints of the two strokes is calculated as the initial position difference value; cluster analysis is performed on the distribution of connection points for each stroke, extracting the density distribution map of the connection points, and the spatial overlap of the connection points of any two strokes is evaluated based on the density distribution map using a weighted kernel function; the initial position difference value and the spatial overlap are weighted and fused to obtain the structural difference between different strokes. In a specific implementation of this invention, the coordinate mapping is based on the stroke layer image, with the center of each writing square as the origin, the horizontal distance of each pixel in the stroke from the origin as the horizontal axis, and the distance of each pixel in the stroke from the origin as the vertical axis; the density distribution map is an image that only shows the connection points of the strokes, and each connection point also contains coordinate information; the structural difference is expressed by the formula: In the formula, For the first The strokes and the first The structural differences between individual strokes; For the first The strokes and the first Initial positional difference between the endpoints of each stroke; For the first The strokes and the first Spatial overlap of the connection points of each stroke; The weighting coefficient is taken as an empirical value of 0.6 in this embodiment; the formula for calculating spatial overlap is: ;in, Representing the A set of connection points for each stroke. Representing the A set of connection points for each stroke. Representing the The connection points in the set of connection points of each stroke Representing the The connection points in the set of connection points of each stroke Representing connection point and The distance is calculated based on the coordinates of the connecting points, and common calculation methods include Euclidean distance. The bandwidth parameter, representing the kernel function, is used to reflect spatial influence. In this example, it is set to 10, based on the statistical setting of the spacing between handwritten strokes by elementary school students. Representing the The size of the set of connection points of each stroke Representing the The size of the set of connection points of each stroke.

[0064] The preset structural difference threshold is 0.3, which was determined based on statistical analysis of 500 elementary school student essay samples. When the structural difference is less than 0.3, it indicates that the two strokes have a high degree of similarity and may belong to the same modification operation.

[0065] The modification tracking path reflects the path and method by which students perform modification operations. For example, erasures are often represented by dense, repetitive strokes, the insertion mark is usually a "∧" shape, and the swap symbol is usually an arrow shape. By tracing the connections between these characteristic strokes, the complete trajectory of the modification operation can be reconstructed.

[0066] The method for obtaining the modification tracking trajectory of each potential modification region includes: scanning the strokes of the font in all essay images line by line to identify the endpoints and connection points of each stroke; counting the number of endpoint overlaps and spacing differences between adjacent strokes in each essay image and marking them as connection irregularities for each stroke; taking the stroke regions in each essay image whose connection irregularities exceed a preset standard threshold as initial path points; setting preset connection conditions, including that the distance between stroke endpoints is less than a preset distance and the spacing difference is less than a preset difference threshold; and expanding point by point from the initial path point, including adjacent strokes that meet the preset connection conditions in the path expansion, until all adjacent strokes no longer meet the preset connection conditions, thus obtaining the modification tracking trajectory of each potential modification region in each essay image.

[0067] In this embodiment, the preset standard threshold is set to 2.5, the preset distance is set to 5 pixels, and the preset difference threshold is set to 3 pixels. These parameters take into account the writing characteristics of primary school students, which can both identify modification traces and avoid misjudging normal stroke connections as modifications. The spacing difference refers to the degree of difference between the Euclidean distance between the nearest endpoints of two adjacent strokes and the expected normal writing spacing.

[0068] Not all potential modification areas are actual modifications; some may be due to improper handwriting or overlapping strokes. By analyzing marking patterns and path relationships, the true modification areas can be identified.

[0069] The method for obtaining the actual modification region includes: performing label pattern analysis on potential modification regions in each essay image to identify label types, such as arrows or wavy lines; obtaining the path correlation degree between any two potential modification regions in the essay image based on the label type similarity and positional association; merging potential modification regions in the essay image with a path correlation degree greater than a preset correlation threshold into candidate regions; expanding the boundaries of the candidate regions, including regions containing additional labels within the boundaries, until all boundary regions no longer contain additional labels, thus obtaining the actual modification region in each essay image; where label type similarity refers to the degree of similarity in shape and number of modification labels contained in two potential modification regions, obtained by similarity calculation based on statistical vectors constructed from label types, where each value in the statistical vector represents the quantity of a label type; positional association refers to the relative positional relationship between two potential modification regions in the image space, used to determine whether they belong to the same modification operation; path correlation degree is a weighted fusion result of label type similarity and positional association degree, used to comprehensively evaluate whether two potential modification regions belong to the same modification logic path; the preset correlation threshold is set to 0.6, which ensures the accuracy of modification region identification. For example, when students use arrows to indicate word and phrase transpositions, the starting and ending areas of the arrows are identified as areas with high path relevance.

[0070] Step S3: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping.

[0071] After identifying the actual areas to be modified, it's necessary to understand the intent behind these modifications. Different modification markers represent different intentions: wavy lines typically indicate deletion, caret indicates addition, and arrows indicate swapping positions, etc. By analyzing the logical relationships between these markers, an intent-related mapping can be constructed.

[0072] The method for obtaining intent association mapping includes: determining the writing structure of each actual modification area in each composition image, including stroke order and marking direction; decomposing the writing structure layer by layer and extracting the marking logic of each level, such as word and phrase swaps indicated by arrows; constructing a temporary intent chain based on the modification tracking trajectory of each actual modification area and the extracted marking logic; associating the logical nodes in the temporary intent chain with the writing structure of adjacent areas to generate intent association mapping for each actual modification area; decomposing the writing structure layer by layer based on image morphology and stroke tracking, peeling off the structure layer by layer; the first layer: extracting the path from the start to the end of each stroke based on the endpoint tracking results; the second layer: through clustering... The analysis merges adjacent strokes into markers and extracts the marker direction; the third layer: based on the positional relationship between the markers and the main text, it analyzes their semantic roles (such as "delete", "insert", "switch"); the temporary intent chain is an intent sequence composed of the modification tracking trajectory and marker logic in the writing order, reflecting the student's thinking path in the modification process; the construction process includes: 1. Node extraction: extracting key nodes from the modification tracking trajectory (such as the starting point, turning point, and marker position of strokes); extracting operation types from the marker logic (such as "delete word A", "insert word C at word B"); 2. Temporal alignment: arranging logical nodes according to the writing order and the position of the markers, aligning the trajectory with the logical nodes. 3. Chain Connection: Connecting aligned logical nodes into a directed chain; Writing structure association refers to associating logical nodes in the temporary intent chain with the writing structure of adjacent non-modified areas. The association methods include: for each logical node, traversing the stroke structure of its adjacent areas; if the association rules are satisfied, establishing a connection edge with the adjacent stroke structure; recording the type of the connection edge (such as "guided insertion", "triggered deletion"); where the association rules include: position matching rule: the distance between the position of the logical node and the position of the adjacent stroke is less than a preset threshold (set to 5 pixels in this embodiment); logical compatibility rule: the logical node operation is semantically compatible with the adjacent writing structure (such as the "insert" operation requires sufficient blank space). Temporal coherence rule: Logical nodes do not violate the writing logic in terms of the temporal order of text writing; Intent association mapping is a graph structure, where nodes are logical nodes and adjacent writing structures, and edges represent the semantic or spatial relationships between them. The construction method of intent association mapping includes: using the nodes of the temporary intent chain as the initial node set, using the operation sequence in the chain as the initial edge set, adding the associated adjacent writing structures as new nodes to the graph, adding the connection edges established in the writing structure association to the graph, semantically labeling all nodes and edges in the graph, using graph embedding methods (such as Node2Vec) to generate the intent vector of each node, and merging semantically similar nodes based on vector similarity to form the final intent association mapping.

[0073] The method for constructing a structured text sequence of essay images based on intent association mapping includes: starting from the actual modified area of ​​each essay image, extracting the marking logic from the intent association mapping one by one; reorganizing the stroke sequences of adjacent non-modified areas according to the marking logic, for example, applying the swapped positions indicated by arrows to the sequence; integrating the reorganized stroke sequences with the original writing structure to form a preliminary text framework; adjusting the order of the preliminary text framework, incorporating the missing content indicated by the insertion symbol into the framework according to positional priority, until all inserted content is integrated, thus obtaining the structured text sequence of the essay image.

[0074] For example, in a real-world case, a student originally wrote "I was very happy to go to the park today," then inserted the word "park" between "happy" and "go," and used an arrow to move "today" before "I." Through intent association mapping, the system was able to recognize these modifications, and the final structured text sequence generated was "Today I was very happy to go to the park."

[0075] Step S4: Detect the expressive intent and creative ideas in elementary school students' handwritten compositions through structured text sequence detection, and generate multimodal recognition results.

[0076] Structured text sequences not only contain the final text content but also retain information about the revision process. This revision information reflects the student's thought process: frequent word and sentence changes may indicate that the student is considering the expression, while a large amount of inserted content may indicate that the student is enriching the detailed description.

[0077] The method for detecting the expressive intentions and creative ideas in elementary school students' handwritten compositions through structured text sequences includes: analyzing the marked logical sequences in the structured text sequence, such as the deleted content represented by wavy lines; extracting key intention points based on the association between the marked logical sequences and the original stroke order, such as the evolution of thought through word and sentence changes; and integrating the extracted key intention points with the structured text sequence to form the final multimodal recognition result, thereby achieving a complete recognition of the expressive intentions of elementary school students.

[0078] Multimodal recognition results include information from multiple dimensions, such as the original text, the final text, statistics on modification types, and analysis of creative thinking. For example, the system can identify when a student added adjectives when describing scenery or adjusted the chronological order when narrating events. This information is of great value to teachers in understanding students' writing abilities and thinking characteristics.

[0079] This embodiment identifies potential modification areas through stroke connections and spacing distribution. Changes in the connection of adjacent strokes and marking patterns reflect the characteristics of students' modification behavior. By acquiring modification tracking trajectories and filtering actual modification areas, it avoids the problem of traditional OCR systems filtering modification traces as noise, thus preserving the students' thought process information during writing. The modification tracking trajectory reflects the students' thinking process and modification logic at specific locations. By analyzing the trajectory and constructing an intent association mapping, the originally messy modification marks are transformed into ordered logical relationships, enabling accurate parsing of modification intentions such as word and phrase changes indicated by arrows and deletions marked by wavy lines. Based on the intent association mapping, a structured text sequence is constructed, which not only identifies the text content but, more importantly, reconstructs the evolution of students' creative thinking, achieving a leap from surface text recognition to deep intent understanding. By detecting expressive intent and creative thinking to generate multimodal recognition results, it provides teachers with complete information on students' writing difficulties and thinking characteristics, overcoming the shortcomings of existing technologies that fail to truly reflect the development of students' writing abilities due to ignoring modification traces. This enhances the application value and educational effectiveness of intelligent essay correction systems in real teaching scenarios.

[0080] Example 2

[0081] Please see Figure 2 As shown, parts not described in detail in this embodiment are described in Embodiment 1. A multimodal OCR recognition system for elementary school students' handwritten compositions is provided, including:

[0082] Data acquisition module: Acquires images of handwritten compositions by elementary school students, with each composition image corresponding to a complete page of the student's writing;

[0083] Region filtering module: Based on the stroke connections and spacing distribution in the essay image, it obtains potential modification regions; based on the connection changes and marking patterns of adjacent strokes in each essay image, it obtains the modification tracking trajectory of each potential modification region; and filters the actual modification regions based on the modification tracking trajectory.

[0084] Intent recognition module: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping;

[0085] Result recognition module: Detects the expressive intent and creative ideas in elementary school students' handwritten compositions through structured text sequences, and generates multimodal recognition results.

[0086] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0087] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0088] In the description of this invention, it should be understood that the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0089] In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0090] In the description of this invention, "several" means one or more, and "a large number" means two or more.

[0091] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] All formulas in this manual are dimensionless and calculated numerically. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0093] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multimodal OCR recognition method for handwritten compositions by primary school students, characterized in that, include: Step S1: Obtain images of elementary school students' handwritten compositions, with each composition image corresponding to a student's complete writing page; Step S2: Based on the stroke connections and spacing distribution in the essay image, obtain potential modification areas; Based on the connection changes and marking patterns of adjacent strokes in each composition image, obtain the modification tracking trajectory of each potential modification area; The actual modified area is selected based on the modified tracking trajectory; The methods for obtaining potential modification areas include: Endpoint tracing is performed on all strokes in the composition images to identify the start and end positions of each stroke; Based on the difference in endpoint positions and the distribution of connection points between any two strokes in the composition image, obtain the structural difference degree between the corresponding two strokes; A continuous block of strokes in the composition image whose structural difference is less than a preset structural difference threshold is denoted as a local stroke group. The preset merging conditions include: the distance between endpoints of groups is less than a preset threshold and the distribution of connection points is similar; local stroke groups in the composition image are merged, and adjacent groups that meet the preset merging conditions are included in the merge until all groups no longer meet the preset merging conditions, thus obtaining the potential modification area; among them, the similarity of the distribution of connection points between groups is based on the judgment of minimum structural difference. For strokes in two groups, the two strokes with the smallest structural difference are matched as the minimum difference group. The mean of the structural difference of all minimum difference groups is calculated as the minimum structural difference. When the minimum structural difference is less than the preset group difference, it indicates that the distribution of connection points between groups is similar. Obtain the modification tracking trajectory for each potential modification region, including: The strokes of the characters in all the essay images are scanned line by line to identify the endpoints and connection points of each stroke; The number of endpoint overlaps and spacing differences of adjacent strokes in each composition image are counted and marked as the connection irregularity value of each stroke; Use the stroke regions in each composition image whose non-standard connection values ​​exceed the preset standard threshold as the initial path points; The preset connection conditions include that the distance between the stroke endpoints is less than a preset distance and the difference in spacing is less than a preset difference threshold. By expanding point by point from the initial path point, adjacent strokes that meet the preset connection conditions are included in the path expansion until all adjacent strokes no longer meet the preset connection conditions, and the expansion stops, thus obtaining the modification tracking trajectory of each potential modification area in each composition image. Step S3: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping; The methods for obtaining intent association mappings include: Determine the writing structure of each actual modification area in each composition image, including stroke order and marking direction; The writing structure is decomposed layer by layer, and the marking logic of each level is extracted; a temporary intent chain is constructed based on the modification tracking trajectory of each actual modification area and the extracted marking logic; Associate the logical nodes in the temporary intent chain with the writing structure of the adjacent regions to generate an intent association mapping for each actual modified region. Step S4: Detect the expressive intent and creative ideas in the handwritten compositions of primary school students through the structured text sequence, and generate multimodal recognition results.

2. The multimodal OCR recognition method for primary school students' handwritten compositions according to claim 1, characterized in that, The endpoint tracing of strokes in all composition images, identifying the start and end positions of each stroke, includes: Each composition image is decomposed into a stroke layer and a background layer through layer-by-layer image segmentation; each stroke in the stroke layer is dynamically tracked, and the continuous pixel sequence of the stroke is detected line by line from the starting scan line of the image, and the first pixel group of the continuous pixel sequence is marked as the potential starting position; according to the pixel density distribution and direction change of the stroke, the expansion path of each potential starting position is calculated until the breakpoint of the continuous pixel sequence is detected and the tracking is paused. The breakpoint where tracking is paused is compared with the reconnection point of the subsequent continuous pixel sequence. When the pixel overlap rate between the breakpoint and the connection point is greater than the preset overlap threshold, the breakpoint is adjusted to the termination position. When the pixel overlap rate between the breakpoint and the connection point is less than or equal to the preset overlap threshold, the breakpoint is reconnected through iterative smoothing. Repeat the above process until the pixel sequences in all stroke layers have been processed to obtain the start and end positions of each stroke.

3. The multimodal OCR recognition method for primary school students' handwritten compositions according to claim 2, characterized in that, The step of obtaining the structural difference degree between two strokes based on the difference in endpoint positions and the distribution of connection points between any two strokes in the composition image includes: Coordinate mapping is performed on the endpoints of any two strokes in the composition image, the start and end positions are represented as two-dimensional coordinate points, and the Euclidean distance between the endpoints of the two strokes is calculated as the initial position difference value. Cluster analysis is performed on the distribution of connection points for each stroke to extract the density distribution map of connection points. Based on the density distribution map, the spatial overlap of connection points between any two strokes is evaluated using a weighted kernel function. The initial position difference value and spatial overlap are weighted and fused to obtain the structural difference between different strokes.

4. The multimodal OCR recognition method for primary school students' handwritten compositions according to claim 3, characterized in that, The method for obtaining the actual modified area includes: The labeling pattern analysis is performed on the potential modification regions in each essay image to identify the labeling type; the path correlation between the two regions is obtained based on the labeling type similarity and positional association of any two potential modification regions in the essay image. Potentially modified regions in the essay image with a path correlation greater than a preset correlation threshold are merged into candidate regions. The candidate regions are expanded by including regions containing additional markers within the boundaries until all boundary regions no longer contain additional markers, thus obtaining the actual modified regions in each composition image.

5. The multimodal OCR recognition method for primary school students' handwritten compositions according to claim 4, characterized in that, The construction of the structured text sequence of the essay image based on the intent association mapping includes: Starting from the actual modified area of ​​each essay image, extract the tagging logic from the intent association mapping one by one; Reorganize the stroke sequences of adjacent unmodified regions according to the marking logic; integrate the reorganized stroke sequences with the original writing structure to form a preliminary text framework; The initial text framework is rearranged in order, and the missing content indicated by the insertion mark is incorporated into the framework according to its positional priority until all the inserted content is integrated, resulting in a structured text sequence of the composition image.

6. The multimodal OCR recognition method for primary school students' handwritten compositions according to claim 5, characterized in that, The method of detecting the expressive intent and creative ideas in elementary school students' handwritten compositions through the structured text sequence includes: The system analyzes the marked logical sequence in the structured text sequence; extracts the intent key points based on the association between the marked logical sequence and the original stroke order; integrates the extracted intent key points with the structured text sequence to form the final multimodal recognition result, thereby achieving complete recognition of the intention expressed by primary school students.

7. A multimodal OCR recognition system for handwritten compositions by primary school students, used to implement the multimodal OCR recognition method for handwritten compositions by students as described in any one of claims 1 to 6, characterized in that, include: Data acquisition module: Acquires images of handwritten compositions by elementary school students, with each composition image corresponding to a complete page of the student's writing; Region filtering module: Based on the stroke connections and spacing distribution in the essay image, it identifies potential modification areas; Based on the connection changes and marking patterns of adjacent strokes in each composition image, obtain the modification tracking trajectory of each potential modification area; The actual modified area is selected based on the modified tracking trajectory; Intent recognition module: Based on the modification tracking trajectory of each actual modification area in each essay image, obtain the intent association mapping of the corresponding actual modification area; construct a structured text sequence of the essay image based on the intent association mapping; Result recognition module: Detects the expressive intent and creative ideas in elementary school students' handwritten compositions through the structured text sequence, and generates multimodal recognition results.

Images