Picture identification method and system for scanning and reading pen

Abstract

The invention relates to the technical field of computer vision, in particular to a picture recognition method and system for a scanning and reading pen. The method comprises the following steps: acquiring an original image; performing superpixel generation on the original image to obtain a superpixel set; performing semantic segmentation network training based on superpixel perception layer addition on a pre-constructed initial U-Net network model to obtain a semantic segmentation model; performing super-pixel semantic category prediction by using the semantic segmentation model and the super-pixel set, and performing super-pixel semantic graph construction to obtain a super-pixel semantic graph; performing stroke fragment extraction on the superpixel semantic graph, and constructing a stroke connection graph to obtain the stroke connection graph; carrying out stroke constraint on the stroke connection graph, and carrying out stroke reconstruction to obtain a reconstructed stroke set; and generating a stroke structure diagram according to the reconstructed stroke set to obtain the stroke structure diagram. According to the invention, a more accurate and more reliable scanning pen identification result is realized through a computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image recognition method and system for a scanning pen. Background Art

[0002] Initially, optical character recognition (OCR) was mainly based on technologies such as template matching and feature extraction. Early OCR systems could only recognize a limited number of fonts and characters, and had very high requirements for image quality. Later, the introduction of methods such as statistical pattern recognition and structural analysis improved the recognition performance and generalization capabilities of OCR systems. However, traditional OCR technology still faces challenges in some specific scenarios. For example, in the case of complex backgrounds, uneven lighting, distorted text, etc., the recognition rate is still low, and it is difficult to process handwriting and artistic characters. The main reasons for these problems are: Complex background and uneven lighting: These factors will interfere with the extraction and recognition of characters, making feature extraction difficult and thus reducing recognition accuracy.

[0003] Distorted and deformed text: Distorted and deformed text changes the shape and structure of the characters, making traditional recognition methods based on shape features ineffective. The diverse and irregular shapes of handwritten and artistic characters further exacerbate this problem. Summary of the invention

[0004] Based on this, it is necessary to provide an image recognition method and system for a scanning pen to solve at least one of the above technical problems.

[0005] To achieve the above object, a method for image recognition using a scanning pen comprises the following steps: Step S1: obtaining an original image; generating superpixels for the original image to obtain a superpixel set; training a pre-built initial U-Net network model with a semantic segmentation network based on a superpixel perception layer to obtain a semantic segmentation model; using the semantic segmentation model and the superpixel set to predict superpixel semantic categories, and constructing a superpixel semantic map to obtain a superpixel semantic map; Step S2: extracting stroke fragments from the superpixel semantic graph and constructing a stroke connection graph to obtain a stroke connection graph; performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; generating a stroke structure graph based on the reconstructed stroke set to obtain a stroke structure graph; Step S3: obtaining font samples; constructing a multi-scale stroke structure library and a glyph feature library based on the font samples to obtain a multi-scale stroke structure library and a glyph feature library; performing graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; performing glyph matching based on the character similarity matrix to obtain a candidate character set; Step S4: Obtain the preceding text; fuse the vector context information of the candidate character set and the preceding text to obtain a fused feature vector; predict the optimal character on the fused feature vector to obtain a predicted character; splice the predicted character into the preceding text to construct a correction character sequence to obtain a correction character sequence; output the text result of the correction character sequence to obtain a display result.

[0006] The present invention obtains clear original images through the scanning pen camera, and uses the SLIC algorithm to generate superpixels, converting pixel-level processing into superpixel-level processing, effectively reducing the computational complexity while retaining the edge information of the image. Subsequently, a U-Net semantic segmentation model with a superpixel perception layer is constructed and trained. The model integrates the local image features extracted by the convolution layer and the global features (color, texture, position, shape) at the superpixel level, and significantly improves the segmentation accuracy and robustness of the model for text areas through the weighted cross entropy loss function and data enhancement strategy. The superpixel semantic map finally generated can finely present the category (text, background, noise) of each pixel in the image, providing accurate input for subsequent stroke extraction. Superpixels marked as "text" are extracted from the superpixel semantic map, eliminating background and noise interference, and improving the efficiency and accuracy of subsequent processing. Then, the stroke skeleton is extracted by Otsu binarization and Zhang-Suen thinning algorithm, and the thinned pixels are connected into stroke fragments using connected domain analysis. Based on these stroke fragments, a stroke connection graph is constructed, and the possibility of connection between stroke fragments is predicted by a logistic regression model, taking into account multiple features such as directional similarity, distance, and grayscale value difference. Most importantly, stroke constraints (length, angle, and loop detection) are defined, and the Kruskal algorithm is used to solve the weighted minimum spanning tree. Under the premise of meeting the constraints, the stroke fragments are reconstructed into complete strokes, ensuring the rationality and accuracy of the reconstructed strokes. The resulting stroke structure graph fully describes the stroke composition and layout of the text, providing key input for glyph matching. A multi-scale stroke structure library and glyph feature library containing multiple fonts, font sizes, and scales are constructed to ensure the coverage of the font library and its adaptability to input texts of different sizes. When calculating graph similarity, the best matching scale is selected through area similarity calculation, avoiding the error caused by directly comparing stroke structure graphs of different scales. At the same time, the local feature vectors of important stroke structures (horizontal, vertical, left-falling, right-falling, and dot, etc.) are defined and extracted, and the overall structural similarity is calculated in combination with the graph edit distance algorithm, achieving a more refined and accurate similarity calculation. Finally, by setting the similarity threshold and selecting the Top N characters, a candidate character set containing the most likely recognition results is generated, providing a reliable input for subsequent context fusion and final recognition. The recognized preceding text information and candidate character information are fully utilized to improve the accuracy and fluency of recognition. The candidate characters are encoded into feature vectors through pre-trained character embedding models (such as BERT) to capture the semantic information of the characters. At the same time, the pre-trained Transformer language model is used to encode the preceding text to obtain a context vector containing semantic information. Most importantly, the attention mechanism is used to fuse the candidate character feature vector and the context vector, so that the model can focus on the preceding text information most relevant to the current candidate character, thereby more accurately judging which candidate character is most suitable.Finally, the character with the highest probability is selected as the predicted character through the fully connected layer and the softmax layer, and the predicted character is spliced ​​into the previous text to construct a complete correction character sequence, and output in a user-readable format, thereby achieving accurate conversion from image to text. Therefore, the present invention provides a picture recognition method for a scanning pen, which effectively solves the shortcomings of traditional OCR in complex backgrounds, uneven lighting, text distortion, handwriting and artistic character recognition through the combination of semantic pre-segmentation, stroke reconstruction, glyph matching and context correction technology, and further improves the recognition accuracy through context correction. The present invention has great advantages in the application scenario of the scanning pen, and can provide more accurate and reliable recognition results.

[0007] Preferably, step S1 comprises the following steps: Step S11: obtaining an original image through a scanning pen camera; generating superpixels for the original image to obtain a superpixel set; Step S12: extracting superpixel features based on the superpixel set and the original image to obtain a superpixel feature vector; Step S13: obtaining a training data set; adding a superpixel perception layer to the pre-built initial U-Net network model, and performing semantic segmentation network training according to the training data set to obtain a semantic segmentation model; Step S14: inputting the superpixel feature vector into the semantic segmentation model to predict the superpixel semantic category to obtain the superpixel semantic category; Step S15: construct a superpixel semantic map according to the superpixel set and the superpixel semantic category to obtain a superpixel semantic map.

[0008] The present invention obtains the original image of the text to be recognized through the scanning pen camera to provide input data for subsequent processing. The key is that the camera automatically focuses to ensure the image is clear. Then, the SLIC algorithm is used to segment the original image into superpixels. Compared with directly processing pixel-level images, superpixels provide local area information of the image, group pixels with similar colors and spatial positions, reduce the amount of calculation for subsequent processing, and retain the edge information of the image, laying the foundation for subsequent feature extraction and semantic segmentation. Superpixel generation is a good compromise between computational efficiency and retaining image details. Rich feature vectors are extracted for each superpixel, including color, texture, position and shape features. These features describe the visual properties of superpixels from multiple perspectives, providing a basis for distinguishing different categories (text, background, noise) for subsequent semantic segmentation. For example, color features can distinguish text and background of different colors, texture features can distinguish the texture differences between text areas and background areas, position features can help the model learn that text usually appears in specific areas of the image, and shape features can help distinguish text strokes with specific shapes. These features are combined into a high-dimensional feature vector so that the semantic segmentation model can learn and predict based on these features. Prepare annotated training data sets to provide learning samples for the model. Then, pre-build the U-Net model, and use its encoder-decoder structure and jump connection to effectively extract image features and generate pixel-level segmentation results. Most importantly, add a superpixel perception layer to integrate the superpixel features extracted in step S12 into the U-Net model. This enables the model to not only use the local image features extracted by the convolution layer, but also the global features at the superpixel level, improving the model's segmentation accuracy for text areas. The model's recognition ability and robustness for text areas are further improved by the weighted cross entropy loss function (giving higher weights to text categories) and data enhancement. Based on the input superpixel feature vector, the model outputs the probability that the superpixel belongs to the "text", "background" or "noise" category, and takes the category with the highest probability as the prediction result. This step converts the feature information at the superpixel level into semantic information, providing a basis for the subsequent construction of superpixel semantic maps. The semantic category information of the superpixel is mapped back to the original image to construct a superpixel semantic map. The semantic map intuitively shows the category (text, background, noise) of each pixel in the image. Compared with directly outputting the category of each superpixel, the superpixel semantic map provides a more refined segmentation result, can more accurately locate the text area, and provides more accurate input for subsequent stroke extraction. At the same time, it handles the overlapping of multiple superpixel boundaries to ensure the accuracy of the semantic map.

[0009] Preferably, step S13 comprises the following steps: Step S131: performing data enhancement on the training data set to obtain an enhanced training image set; Step S132: integrating the superpixel perception layer of the pre-constructed initial U-Net network model according to the superpixel feature vector to obtain a U-Net model integrating the superpixel perception layer; Step S133: defining a weighted cross entropy loss function according to the enhanced training image set to obtain a weighted cross entropy loss function; Step S134: Perform semantic segmentation network training according to the enhanced training image set, the weighted cross entropy loss function, and the U-Net model with integrated superpixel perception layer to obtain a semantic segmentation model.

[0010] The present invention generates more training samples by applying multiple data enhancement operations (random rotation, scaling, cropping, flipping, color jittering) to the original training data set. This effectively expands the scale and diversity of the training data set and improves the generalization and robustness of the model. Data enhancement can simulate various image changes that may occur in real scenes (for example, texts of different angles, lighting, and sizes), so that the model can adapt to these changes, thereby obtaining better recognition effects in practical applications. The key is that all enhancement operations retain the annotation information of superpixels, ensuring the effectiveness of the enhanced data. Integrating superpixel features into the U-Net model is the key to improving text segmentation accuracy. By adding a superpixel perception layer after each convolutional layer of the U-Net encoder, the model can not only learn the local image features extracted by the convolutional layer, but also utilize the global features (color, texture, position, shape) at the superpixel level. The superpixel perception layer maps the superpixel features to the same dimension as the convolutional layer output through a fully connected layer, and uses the channel attention mechanism to fuse the two features. This enables the model to understand the image information more comprehensively and distinguish text areas from background areas more accurately, thereby improving segmentation accuracy. A weighted cross entropy loss function is defined to measure the difference between the model prediction results and the true annotations. The key is to assign different weights to different categories (text, background, noise). Since text recognition is the main goal, the "text" category is assigned the highest weight (0.6), which makes the model pay more attention to the prediction accuracy of the text area during training, thereby further improving the accuracy of text segmentation. This weighting strategy can effectively solve the problem of category imbalance and improve the model's ability to recognize important categories. The results of the previous steps are integrated to conduct the final training of the model. The enhanced training image set is used to provide rich training samples, the weighted cross entropy loss function guides the model optimization direction, and the U-Net model with integrated superpixel perception layer is used as the learning subject. Through the Adam optimizer and back-propagation algorithm, the model continuously adjusts parameters to minimize the loss function, and finally obtains a trained semantic segmentation model. The validation set evaluation and model preservation strategy during the training process ensure the optimal performance of the final model. This step is the core of the entire semantic segmentation process, combining data, model and optimization strategy to obtain a model that can accurately segment text areas.

[0011] Preferably, step S2 comprises the following steps: Step S21: extracting text regions from the superpixel semantic map to obtain a text region superpixel set; Step S22: extracting stroke segments according to the text area superpixel set and the original image to obtain a candidate stroke segment set; Step S23: constructing a stroke connection graph for the candidate stroke segment set to obtain a stroke connection graph; Step S24: performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; Step S25: generating a stroke structure diagram according to the reconstructed stroke set to obtain a stroke structure diagram.

[0012] The present invention forms a text area superpixel set by extracting superpixels marked as "text". The advantage of this is that the scope of subsequent processing is limited to the text area, the interference of background and noise is eliminated, and the efficiency and accuracy of subsequent stroke extraction and reconstruction are improved. At the same time, the boundary coordinates and pixel coordinate sets of the text area superpixels are recorded to provide an index for the subsequent extraction of the corresponding area from the original image. Stroke fragments are extracted from the original image area corresponding to the text area superpixels. First, the Otsu algorithm is used for binarization to separate the text pixels and the background pixels. Then, the Zhang-Suen thinning algorithm is used to extract the stroke skeleton of a single pixel width, which is a key step in stroke extraction and can effectively extract the shape features of the strokes. Finally, the thinned pixels are connected into stroke fragments through connected domain analysis, and the features of each fragment (center point, main direction, length, pixel coordinates) are calculated. These stroke fragments are the basis for subsequent stroke reconstruction and provide nodes for constructing a stroke connection graph. The stroke fragments are regarded as nodes, and a stroke connection graph is constructed to represent the connection relationship between the stroke fragments. The calculation of connection possibility comprehensively considers multiple features (directional similarity, distance, average gray value difference, endpoint distance ratio) and uses a logistic regression model for prediction. This is more accurate and robust than the judgment of a single feature. By setting the connection possibility threshold, unreasonable connections can be filtered out, reducing the amount of calculation for subsequent stroke reconstruction. The stroke connection graph provides a basis for subsequent stroke merging and is a key data structure for stroke reconstruction. Define stroke constraints (length, angle, loop detection) to avoid generating unreasonable strokes. Then, use the Kruskal algorithm to solve the weighted minimum spanning tree and find the optimal stroke connection method under the premise of satisfying the constraints. According to the results of the minimum spanning tree, the connected stroke fragments are merged into complete strokes and the stroke features are recalculated. This step reconstructs scattered stroke fragments into complete strokes, providing more accurate input for subsequent glyph matching. The combination of the Kruskal algorithm and stroke constraints ensures the rationality and accuracy of the reconstructed strokes. The reconstructed strokes and their spatial relationships are represented as a graph structure. Nodes represent strokes, and edges represent the spatial relationship between strokes (intersection, distance, direction). The stroke structure graph fully describes the stroke composition and layout of the text, and is a key input for glyph matching. Compared with directly using the reconstructed stroke set, the stroke structure graph more clearly expresses the relationship between strokes, providing richer information for subsequent graph similarity calculations.

[0013] Preferably, step S24 comprises the following steps: Step S241: defining stroke constraints according to the candidate stroke segment set to obtain stroke constraint definition data; Step S242: Calculate edge weights according to the stroke connection graph to obtain edge weights; Solve the weighted minimum spanning tree for edge weights to obtain the minimum spanning tree; Step S243: performing multi-stroke segment connection processing on the candidate stroke segment set according to the minimum spanning tree to obtain a reconstructed stroke set.

[0014] The present invention defines the constraints in the stroke reconstruction process, including length constraints, angle constraints, distance constraints and loop constraints. These constraints are based on the prior knowledge of writing rules and character structure, and can effectively avoid the generation of unreasonable strokes. For example, the length constraint prevents the strokes from being too long, the angle constraint prevents the stroke direction from changing suddenly, the distance constraint prevents the distance between the strokes from being too far, and the loop constraint prevents the strokes from forming a closed loop. These constraints provide guidance for subsequent stroke merging, and ensure the rationality and accuracy of reconstructed strokes. These rules are clearly defined as data, which is convenient for subsequent algorithm use and modification. The weight of each edge in the stroke connection graph is calculated according to the connection possibility and the stroke constraint conditions. The weight calculation formula comprehensively considers the connection possibility and various constraint penalty factors, so that the weight can reflect the rationality of the stroke connection. Then, the Kruskal algorithm is used to solve the weighted minimum spanning tree. The Kruskal algorithm can find the stroke connection mode with the minimum total weight under the premise of satisfying the stroke constraints, that is, find the most reasonable stroke connection scheme. The result of the minimum spanning tree provides clear guidance for the subsequent stroke fragment merging. According to the connection relationship of the minimum spanning tree, the connected stroke fragments are merged into a complete stroke. Specifically, the pixel coordinate lists of the stroke fragments are merged and the features of the strokes are recalculated. If a stroke fragment is not connected to other fragments, it is treated as an independent stroke. This step completes the reconstruction from stroke fragments to complete strokes and obtains the final reconstructed stroke set. The reconstructed stroke set is the basis for subsequent glyph matching, and its quality directly affects the accuracy of recognition.

[0015] Preferably, step S3 comprises the following steps: Step S31: obtaining a font sample; extracting a stroke structure graph from the font sample, and constructing a multi-scale stroke structure graph library to obtain a multi-scale stroke structure graph library; Step S32: constructing a font feature library according to the multi-scale stroke structure library to obtain a font feature library; Step S33: performing graph similarity calculation on the multi-scale stroke structure library and the character feature library to obtain a character similarity matrix; Step S34: Screening candidate characters from the character similarity matrix to obtain a list of candidate characters and similarity scores; Step S35: construct a candidate character set for the candidate characters and similarity score lists to obtain a candidate character set.

[0016] The present invention ensures the coverage of the character library by obtaining standard character sample images of multiple fonts and font sizes. Then, the stroke structure diagram of each character is extracted using the same stroke extraction and reconstruction method as step S2, which ensures that the stroke structure diagram in the character library is consistent with the stroke structure diagram of the input image. Most importantly, the stroke structure diagram of each character is scaled at multiple scales to construct a multi-scale stroke structure library. This enables the character library to adapt to input texts of different sizes and improves the robustness of recognition. The multi-scale character library is the key to processing texts of different sizes. A feature vector is constructed for each character in the character library for subsequent similarity calculation. The feature vector contains the number of strokes, the features of each stroke (length, direction, curvature) and the connection relationship between the strokes. These features describe the glyph structure of the character from multiple angles and provide a basis for distinguishing different characters. The feature vector is stored together with the character encoding, font, font size and scaling ratio to construct a glyph feature library, which facilitates subsequent fast similarity calculation. The similarity between the input stroke structure diagram and each character in the character library is calculated. First, according to the size of the input stroke structure graph, a set of stroke structure graphs with the closest scale is selected from the multi-scale character library, which reduces the amount of calculation. Then, the graph edit distance algorithm is used to calculate the similarity. The graph edit distance algorithm can effectively measure the difference between two graph structures and is an effective method for calculating the similarity of stroke structure graphs. The calculated similarity score is stored in the character similarity matrix, which provides a basis for the subsequent screening of candidate characters. Possible candidate characters are screened out according to the similarity score. By setting a similarity threshold, characters with low similarity are filtered out, which reduces the amount of calculation for subsequent processing. Characters with similarity scores higher than the threshold and their scores are stored in a list and sorted by score, which facilitates the subsequent selection of the best candidate characters. The N characters with the highest scores are selected from the candidate characters and similarity score lists to form a candidate character set. The candidate character set contains the most likely recognition results and is the input for subsequent context fusion and final recognition. Selecting Top N instead of only selecting the characters with the highest scores can improve the fault tolerance of recognition and avoid errors in the entire recognition result due to single character recognition errors.

[0017] Preferably, step S33 includes the following steps: Step S331: performing area similarity calculation on the multi-scale stroke structure library, and performing scale matching according to the character feature library to obtain matching scale selection data; Step S332: Obtain important stroke data; define important stroke structures according to the important stroke data to obtain important stroke structure definition data; Step S333: extracting local structural features from the multi-scale stroke structure library according to the important stroke structure definition data to obtain a local structural feature vector; Step S334: Calculate graph similarity based on the multi-scale stroke structure library, the local structure feature vector and the glyph feature library to obtain a character similarity matrix.

[0018] The present invention calculates the area difference between the input stroke structure diagram and the stroke structure diagram of each character in the character library at different scales, and selects the scale with the smallest area difference as the best matching scale of the character. The advantage of this is that it avoids directly comparing the stroke structure diagrams of different scales and improves the accuracy of similarity calculation. Because the stroke structure diagrams of different scales, even if they are the same character, will have large differences in the number of strokes, length and other features, direct comparison will lead to errors. By matching the scale, it is ensured that the subsequent similarity calculation is performed at the same or similar scale. Important stroke structures are usually distinguishing strokes in Chinese characters, such as horizontal, vertical, left-falling, right-falling, dot, hook, etc. By pre-defining the judgment rules of these important stroke structures (for example, according to the direction, length, shape and other features of the strokes), the strokes can be classified, thereby more finely describing the glyph structure of the character. This can better distinguish different characters than using only rough features such as the number and length of strokes. According to the definition of important stroke structures, the number of occurrences, average length, average direction and average curvature of various important stroke structures in each stroke structure diagram are counted. These features describe the local details of the stroke structure diagram and can distinguish different characters more accurately. These statistical information are combined into a local structure feature vector, which provides a more refined feature representation for the subsequent graph similarity calculation. Use the best matching scale obtained in step S331. Then, the overall structural similarity is calculated using the graph edit distance algorithm, and the local feature similarity is calculated using the cosine similarity of the local structure feature vector. Finally, the two similarities are weighted averaged to obtain the final graph similarity score. This similarity calculation method that comprehensively considers overall and local features is more accurate and robust than a single similarity calculation method. The calculated similarity score is stored in the character similarity matrix, which provides a basis for subsequent candidate character screening.

[0019] Preferably, step S333 is specifically: Perform stroke direction judgment on the multi-scale stroke structure library to obtain stroke direction judgment data; perform stroke length judgment on the multi-scale stroke structure library to obtain stroke length judgment data; perform stroke shape judgment on the multi-scale stroke structure library to obtain stroke shape judgment data; Performing important stroke structure marking processing according to the stroke direction judgment data, the stroke length judgment data and the stroke shape judgment data to obtain an important stroke structure mark; According to the important stroke structure marks, the important stroke structure similarity is calculated for the multi-scale stroke structure library and the character feature library, and the relative position is calculated to obtain the local feature calculation result; The feature vector is constructed based on the local feature calculation results to obtain the local structure feature vector.

[0020] The present invention can accurately describe the direction of the strokes by using principal component analysis (PCA) to calculate the main direction angle of the strokes, which is an important basis for distinguishing different stroke types (such as horizontal, vertical, left-falling, and right-falling). Calculating the length of the stroke pixel contour can reflect the thickness and length of the stroke, which is an important basis for distinguishing different stroke types (such as dots, short horizontal, and long horizontal). Calculating the stroke shape complexity (the ratio of the perimeter to the perimeter of the minimum circumscribed rectangle) can reflect the degree of curvature of the stroke, which is an important basis for distinguishing different stroke types (such as left-falling and right-falling). These three features describe the geometric properties of the strokes from different angles and provide basic data for subsequent important stroke structure markings. According to the stroke features extracted previously, each stroke is marked as a predefined important stroke structure (horizontal, vertical, left-falling, right-falling, dot) or others. By setting clear rules (for example, the main direction angle of the horizontal stroke is within a certain range and the length is greater than a certain threshold), the strokes can be classified. This is more semantically informative than directly using the original stroke features and can more effectively describe the glyph structure of the character. Important stroke structure labeling is a key step in local feature extraction, providing more refined features for subsequent similarity calculations. The number of each important stroke structure is counted, and the difference in number is calculated as a similarity score. This reflects the similarity between the two characters in the composition of important strokes. The image is divided into grids, and the difference in the grid area where the center points of the same type of important strokes are located is calculated as a relative position score. This reflects the similarity between the two characters in the spatial layout of important strokes. The important stroke structure similarity score and the relative position score are combined into a feature vector, namely the local structure feature vector. This feature vector concisely summarizes the local features of the stroke structure graph and provides an effective feature representation for subsequent graph similarity calculations. The local structure feature vector is a bridge connecting local feature extraction and overall similarity calculations.

[0021] Preferably, step S4 comprises the following steps: Step S41: obtaining a preceding text; using a pre-trained character embedding model to encode candidate character features of a candidate character set to obtain a candidate character feature vector; Step S42: using the pre-trained Transformer language model to encode the preceding text sequence to obtain a context vector; Step S43: fusing the candidate character feature vector and the context vector with context information to obtain a fused feature vector; Step S44: performing optimal character prediction on the fused feature vector to obtain a predicted character; Step S45: splicing the predicted characters into the preceding text to construct a correction character sequence to obtain a correction character sequence; Step S46: converting the corrected character sequence into a text format to obtain a formatted text; and displaying the formatted text to obtain a display result.

[0022] The present invention utilizes the recognized preceding text information and candidate character information. Obtaining the preceding text provides context information for subsequent context fusion, which is the key to improving recognition accuracy. A pre-trained character embedding model (such as BERT's character-level embedding) is used to encode each character in the candidate character set into a feature vector. The pre-trained model has been trained on large-scale text data and can capture the semantic information of the characters, so that similar characters have similar feature vectors. This is more effective than one-hot encoding or other simple encoding methods. The model can simultaneously consider glyph similarity and semantic information. The Transformer model can capture long-distance dependencies in text sequences and understand the semantic information of the preceding text. The context vector contains the semantic information of the preceding text, which provides an important basis for the subsequent selection of the most suitable candidate character. If the preceding text is empty, a zero vector is used for initialization to ensure the integrity of the process. Fusion using an attention mechanism can effectively capture the correlation between the candidate character and the context. The attention mechanism enables the model to focus on the preceding text information most relevant to the current candidate character, thereby more accurately determining which candidate character is most suitable. The fused feature vector contains comprehensive information about candidate characters and context, providing a more comprehensive basis for subsequent optimal character prediction. Through the fully connected layer and the softmax layer, the fused feature vector is converted into a probability distribution, and the character with the highest probability is selected as the predicted character. This step completes the conversion from feature information to character recognition and is the last step of the entire recognition process. The predicted characters are added to the end of the previous text sequence to construct a new corrected character sequence. The corrected character sequence contains the currently recognized character and all previously recognized characters, and is the accumulation of recognition results. The recognition results are converted into a user-readable text format and displayed.

[0023] Preferably, the present invention further provides an image recognition system for a scanning pen, which is used to execute the image recognition method for a scanning pen as described above. The image recognition system for a scanning pen comprises: The semantic pre-segmentation module is used to obtain the original image; generate superpixels for the original image to obtain a superpixel set; train the semantic segmentation network based on the superpixel perception layer added according to the pre-built initial U-Net network model to obtain a semantic segmentation model; use the semantic segmentation model and the superpixel set to predict the superpixel semantic category, and construct a superpixel semantic map to obtain a superpixel semantic map; The stroke reconstruction module is used to extract stroke fragments from the superpixel semantic map and construct a stroke connection map to obtain a stroke connection map; to impose stroke constraints on the stroke connection map and reconstruct the strokes to obtain a reconstructed stroke set; and to generate a stroke structure map based on the reconstructed stroke set to obtain a stroke structure map; The glyph matching module is used to obtain font samples; construct a multi-scale stroke structure library based on the font samples, and construct a glyph feature library to obtain a multi-scale stroke structure library and a glyph feature library; perform graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; perform glyph matching based on the character similarity matrix to obtain a candidate character set; The context correction module is used to obtain the previous text; fuse the vector context information of the candidate character set and the previous text to obtain a fused feature vector; predict the optimal character of the fused feature vector to obtain a predicted character; splice the predicted character into the previous text to construct a correction character sequence to obtain a correction character sequence; output the text result of the correction character sequence to obtain a display result. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a process flow of a picture recognition method for a scanning pen; Figure 2 It is a schematic diagram of the detailed implementation steps of step S2 in the present invention.

[0025] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0026] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0028] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0029] To achieve this, please refer to Figure 1 to Figure 2 , a method for image recognition using a scanning pen, comprising the following steps: Step S1: obtaining an original image; generating superpixels for the original image to obtain a superpixel set; training a pre-built initial U-Net network model with a semantic segmentation network based on a superpixel perception layer to obtain a semantic segmentation model; using the semantic segmentation model and the superpixel set to predict superpixel semantic categories, and constructing a superpixel semantic map to obtain a superpixel semantic map; Step S2: extracting stroke fragments from the superpixel semantic graph and constructing a stroke connection graph to obtain a stroke connection graph; performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; generating a stroke structure graph based on the reconstructed stroke set to obtain a stroke structure graph; Step S3: obtaining font samples; constructing a multi-scale stroke structure library and a glyph feature library based on the font samples to obtain a multi-scale stroke structure library and a glyph feature library; performing graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; performing glyph matching based on the character similarity matrix to obtain a candidate character set; Step S4: Obtain the preceding text; fuse the vector context information of the candidate character set and the preceding text to obtain a fused feature vector; predict the optimal character on the fused feature vector to obtain a predicted character; splice the predicted character into the preceding text to construct a correction character sequence to obtain a correction character sequence; output the text result of the correction character sequence to obtain a display result.

[0030] In the embodiment of the present invention, reference Figure 1 FIG. 1 is a schematic diagram of the steps of the image recognition method for the scanning pen of the present invention. In this example, the image recognition method for the scanning pen includes the following steps: Step S1: obtaining an original image; generating superpixels for the original image to obtain a superpixel set; training a pre-built initial U-Net network model with a semantic segmentation network based on a superpixel perception layer to obtain a semantic segmentation model; using the semantic segmentation model and the superpixel set to predict superpixel semantic categories, and constructing a superpixel semantic map to obtain a superpixel semantic map; In an embodiment of the present invention, the original image captured by the scanning pen camera is obtained, and then the SLIC algorithm is used to generate a superpixel set to divide the image into multiple small areas. Next, the pre-built initial U-Net network model is used, and a superpixel perception layer is added to its encoder part, and the semantic segmentation network is trained using the training data set to obtain a semantic segmentation model. The superpixel perception layer incorporates the feature information of the superpixel into the U-Net model to improve the model's segmentation accuracy for text areas. After the training is completed, the trained semantic segmentation model is used to predict the semantic categories of the superpixels of the original image, and the prediction results are constructed into a superpixel semantic map, in which each superpixel is labeled as "text", "background" or "noise".

[0031] Step S2: extracting stroke fragments from the superpixel semantic graph and constructing a stroke connection graph to obtain a stroke connection graph; performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; generating a stroke structure graph based on the reconstructed stroke set to obtain a stroke structure graph; In an embodiment of the present invention, a superpixel semantic map is processed to extract superpixels of a text region and decompose them into stroke segments. Then, a stroke connection graph is constructed, in which the nodes represent stroke segments and the edges represent the connection possibilities between segments. The connection possibilities are calculated from features such as the direction, distance, grayscale value difference and endpoint distance ratio of the stroke segments. Next, the stroke connection graph is optimized using stroke constraints (length, angle, distance and loop constraints) and the Kruskal algorithm to obtain a minimum spanning tree, and the stroke segments are merged according to the minimum spanning tree to reconstruct the complete strokes to form a reconstructed stroke set. Finally, a stroke structure graph is generated based on the reconstructed stroke set for subsequent glyph matching.

[0032] Step S3: obtaining font samples; constructing a multi-scale stroke structure library and a glyph feature library based on the font samples to obtain a multi-scale stroke structure library and a glyph feature library; performing graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; performing glyph matching based on the character similarity matrix to obtain a candidate character set; In an embodiment of the present invention, a font sample is obtained, and a multi-scale stroke structure library and a glyph feature library are constructed. The multi-scale stroke structure library contains character stroke structure diagrams of different fonts, font sizes and scaling ratios, and the glyph feature library stores the feature vector of each character. Then, the similarity between the input stroke structure diagram and each character in the glyph feature library is calculated. The similarity calculation includes sub-steps such as scale matching, important stroke structure definition, local structure feature extraction and graph similarity calculation. Scale matching selects the scale closest to the area of ​​the input stroke structure diagram; the important stroke structure definition determines the stroke type according to the characteristics of the stroke direction, length and shape; the local structure feature extraction calculates the similarity and relative position relationship of the important stroke structure; the graph similarity calculation combines the structural similarity and the local feature similarity. Finally, the candidate characters are screened according to the similarity score to obtain a candidate character set.

[0033] Step S4: obtaining the preceding text; fusing the vector context information of the candidate character set and the preceding text to obtain a fused feature vector; performing optimal character prediction on the fused feature vector to obtain a predicted character; splicing the predicted character into the preceding text to construct a correction character sequence to obtain a correction character sequence; outputting the text result of the correction character sequence to obtain a display result; In an embodiment of the present invention, a preceding text that has been recognized by the scanning pen is obtained. Then, a pre-trained character embedding model is used to perform feature encoding on the candidate character set to obtain a candidate character feature vector, and a similarity score is added to the feature vector. At the same time, a pre-trained Transformer language model is used to encode the preceding text to obtain a context vector. Next, an attention mechanism is used to fuse the candidate character feature vector and the context vector to obtain a fused feature vector. Then, the fused feature vector is input into a classifier for prediction, and the character with the highest probability is obtained as the predicted character. The predicted characters are spliced ​​into the preceding text to construct a corrected character sequence. Finally, the corrected character sequence is converted into a text format and displayed on the screen of the scanning pen or transmitted to a connected device.

[0034] Preferably, step S1 comprises the following steps: Step S11: obtaining an original image through a scanning pen camera; generating superpixels for the original image to obtain a superpixel set; Step S12: extracting superpixel features based on the superpixel set and the original image to obtain a superpixel feature vector; Step S13: obtaining a training data set; adding a superpixel perception layer to the pre-built initial U-Net network model, and performing semantic segmentation network training according to the training data set to obtain a semantic segmentation model; Step S14: inputting the superpixel feature vector into the semantic segmentation model to predict the superpixel semantic category to obtain the superpixel semantic category; Step S15: construct a superpixel semantic map according to the superpixel set and the superpixel semantic category to obtain a superpixel semantic map.

[0035] In the embodiment of the present invention, after the scanning pen is started, the built-in camera captures an image of the text to be recognized. The camera is set to autofocus mode to ensure that a clear image is captured. The original image obtained is an RGB color image with a resolution of 1200x800 pixels. The SLIC (Simple Linear Iterative Clustering) algorithm is used to perform superpixel segmentation on the original image. The number of superpixels is set to 500 and the compactness parameter is set to 20. The SLIC algorithm groups image pixels into perceptually uniform superpixels through iterative clustering, and each superpixel represents a small area with similar color and spatial position. The algorithm regards the image as a five-dimensional space (Lab color space and xy coordinate space) and iteratively optimizes the superpixel boundary by minimizing the distance between each pixel and the center of the superpixel to which it belongs. Finally, 500 superpixels are obtained, each of which contains the coordinate information of its boundary pixels.

[0036] For each superpixel generated in step S11, a series of features are extracted to describe its visual properties. First, the mean and standard deviation of the RGB color values ​​of all pixels in each superpixel are calculated to obtain a 6-dimensional color feature. Secondly, the original image is convolved using Gabor filters in 5 different directions and 3 different scales (a total of 15), and then the mean and standard deviation of the Gabor filter response in each superpixel are calculated to obtain a 30-dimensional texture feature. Again, the coordinates of the center point of each superpixel are calculated, and the coordinate values ​​are normalized to the [0,1] interval to obtain a 2-dimensional position feature. Finally, the area, perimeter and shape factor (4π*area / perimeter²) of each superpixel are calculated to obtain a 3-dimensional shape feature. The above color, texture, position and shape features are concatenated into a 41-dimensional feature vector to represent each superpixel. Finally, a 500x41 matrix is ​​obtained, in which each row represents the feature vector of a superpixel.

[0037] Prepare a text image dataset with various fonts, font sizes, backgrounds, and lighting conditions. Each image in the dataset is manually annotated, where each superpixel is labeled as "text", "background", or "noise". Pre-build an initial U-Net network model, which contains an encoder and a decoder part. The encoder is used to extract image features and the decoder is used to generate pixel-level segmentation results. Add a superpixel perception layer after each convolution layer in the encoder part of the U-Net. The superpixel perception layer takes the 41-dimensional superpixel feature vector extracted in step S12 as input and maps it to the same dimension as the number of output channels of the convolution layer through a fully connected layer. Then, the mapped superpixel features are fused with the output features of the convolution layer using an attention mechanism. The attention mechanism generates attention weights by calculating the correlation between superpixel features and convolution layer features, and uses these weights to weight the convolution layer features. Use the weighted cross entropy loss function to train the improved U-Net model. The loss function assigns different weights according to the importance of different categories, and the "text" category is assigned a higher weight to improve the segmentation accuracy of the text area. The model is trained using the augmented training image set, using the Adam optimizer with a learning rate of 0.001 and a batch size of 32. The training process continues until the performance of the model on the validation set stabilizes.

[0038] The superpixel feature vector extracted in step S12 is input into the semantic segmentation model trained in step S13. The model outputs the probability that each superpixel belongs to the category of "text", "background" or "noise". The category with the highest probability is selected as the predicted category of the superpixel. Finally, a vector containing 500 elements is obtained, each element representing the semantic category of a superpixel.

[0039] Create a matrix of the same size as the original image and initialize all elements to 0. Map the superpixel semantic categories predicted in step S14 to the matrix based on the superpixel set generated in step S11. For each superpixel, set the categories of all pixels within its boundary to the predicted category of the superpixel. If a pixel belongs to the boundary of multiple superpixels, select the category of the superpixel with the largest proportion as the category of the pixel. Finally, a superpixel semantic map of the same size as the original image is obtained, in which each pixel is labeled as "text", "background" or "noise".

[0040] Preferably, step S13 comprises the following steps: Step S131: performing data enhancement on the training data set to obtain an enhanced training image set; Step S132: integrating the superpixel perception layer of the pre-constructed initial U-Net network model according to the superpixel feature vector to obtain a U-Net model integrating the superpixel perception layer; Step S133: defining a weighted cross entropy loss function according to the enhanced training image set to obtain a weighted cross entropy loss function; Step S134: Perform semantic segmentation network training according to the enhanced training image set, the weighted cross entropy loss function, and the U-Net model with integrated superpixel perception layer to obtain a semantic segmentation model.

[0041] In an embodiment of the present invention, a text image training data set containing annotation information is obtained, wherein each superpixel of each image is labeled as "text", "background" or "noise". The following data augmentation operations are applied to each image in the training data set: 1) Random rotation: randomly rotate the image by an angle between -15 degrees and +15 degrees; 2) Random scaling: randomly scale the image by 0.8 times to 1.2 times; 3) Random cropping: randomly cropping an area of ​​80% to 100% of the original image from the image; 4) Random flipping: horizontally flipping the image with a probability of 50%; 5) Color jittering: randomly adjusting the brightness, contrast, saturation and hue of the image within a range of ±10%. Five random transformation combinations are performed on each original image to generate five enhanced images, and the enhanced images are added to the training data set. All enhancement operations keep the annotation information of the superpixels unchanged. Finally, an enhanced training image set containing original images and enhanced images is obtained, and its size is 6 times that of the original training data set.

[0042] An initial U-Net network model with an encoder and decoder structure is pre-built for pixel-level image segmentation. The encoder part contains multiple convolutional layers and downsampling layers, and the decoder part contains multiple convolutional layers and upsampling layers. A superpixel perception layer is integrated after each convolutional layer in the U-Net encoder part. The superpixel perception layer receives a 41-dimensional superpixel feature vector as input. First, a fully connected layer is used to map the superpixel feature vector to the same dimension as the number of channels of the corresponding convolutional layer output feature map. Then, the mapped superpixel features are copied and expanded to the same spatial dimension as the convolutional layer output feature map, so that all pixels within each superpixel share the same superpixel features. Finally, the channel attention mechanism is used to fuse the expanded superpixel features with the convolutional layer output features. The channel attention mechanism generates channel weights by performing global average pooling, fully connected layer transformation, and sigmoid activation function operations on superpixel features and convolutional layer features, and then applies the channel weights to the convolutional layer output features to achieve weighted fusion of features.

[0043] Define a weighted cross entropy loss function to measure the difference between the superpixel semantic category predicted by the model and the true annotation. The formula of the loss function is as follows: Loss = -∑_{c=1}^{C} w_c * y_c * log(p_c); Among them, C represents the number of categories ("text", "background" and "noise", C=3), w_c represents the weight of category c, y_c represents the true label of category c (0 or 1), and p_c represents the probability of category c predicted by the model. According to the ratio of the number of pixels in each category in the enhanced training image set, the weights of the "text", "background" and "noise" categories are set to 0.6, 0.3 and 0.1 respectively. This means that the model pays more attention to the prediction accuracy of the "text" category.

[0044] The semantic segmentation network training is performed using the enhanced training image set generated in step S131, the weighted cross entropy loss function defined in step S133, and the U-Net model with integrated superpixel perception layer constructed in step S132. The enhanced training image set is divided into a training set and a validation set with a ratio of 9:1. The Adam optimizer is used for training, the learning rate is set to 0.0001, and the batch size is set to 16. During the training process, the image is input into the U-Net model, and the model outputs the semantic category probability of each superpixel. The weighted cross entropy loss between the model output and the true annotation is calculated, and the model parameters are updated using the back propagation algorithm. The model performance is evaluated on the validation set at regular intervals, and the model with the best performance is saved. The training process continues until the performance of the model on the validation set no longer improves or the preset number of iterations is reached. Finally, a trained semantic segmentation model is obtained.

[0045] Preferably, step S2 comprises the following steps: Step S21: extracting text regions from the superpixel semantic map to obtain a text region superpixel set; Step S22: extracting stroke segments according to the text area superpixel set and the original image to obtain a candidate stroke segment set; Step S23: constructing a stroke connection graph for the candidate stroke segment set to obtain a stroke connection graph; Step S24: performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; Step S25: generating a stroke structure diagram according to the reconstructed stroke set to obtain a stroke structure diagram.

[0046] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes: Step S21: extracting text regions from the superpixel semantic map to obtain a text region superpixel set; In the embodiment of the present invention, the superpixel semantic map generated in step S1 is input. Each superpixel in the superpixel semantic map is traversed. The superpixels with the semantic category marked as "text" are extracted to form a text area superpixel set. The boundary coordinates (e.g., the minimum enclosing rectangle) of each text area superpixel and the corresponding pixel coordinate set in the original image are recorded. The pixel values ​​of the extracted text area superpixels are marked in the original image, and the pixel values ​​of other areas remain unchanged to generate a new image in which the text area is displayed in the original color and the pixel values ​​of the non-text area are set to 0 for subsequent processing.

[0047] Step S22: extracting stroke segments according to the text area superpixel set and the original image to obtain a candidate stroke segment set; In an embodiment of the present invention, the text area superpixel set extracted in step S21 and the original image are input. For each text area superpixel, the corresponding region of interest (ROI) is first extracted from the original image according to its boundary coordinates. Then, the ROI is binarized using the Otsu algorithm to convert text pixels to 1 and background pixels to 0. Next, the binarized image is thinned using the Zhang-Suen thinning algorithm to obtain a stroke skeleton with a single pixel width. The thinned image is subjected to a connected domain analysis, and the 8-connected pixel set is marked as a stroke segment. The center point coordinates, main direction (calculated by principal component analysis PCA), length, and pixel coordinate list of each stroke segment are calculated. The information of all stroke segments is stored in the candidate stroke segment set.

[0048] Step S23: constructing a stroke connection graph for the candidate stroke segment set to obtain a stroke connection graph; In an embodiment of the present invention, the set of candidate stroke segments extracted in step S22 is input. A graph structure is created in which each node represents a stroke segment. For each pair of stroke segments, the connection possibility between them is calculated. The connection possibility is calculated based on the following four features: 1) Direction similarity: Calculate the cosine similarity of the main direction vectors of the two stroke segments; 2) Distance: Calculate the minimum Euclidean distance between the endpoints of the two stroke segments; 3) Average gray value difference: Calculate the difference in the average gray value of the corresponding pixels of the two stroke segments in the original image; 4) Endpoint distance ratio: The ratio of the minimum distance between the endpoints of the two stroke segments to the sum of the lengths of the two stroke segments. These four feature values ​​are normalized to the interval [0,1], and then the connection possibility score is calculated using a pre-trained logistic regression model. The connection possibility score is used as the weight of the edge in the graph. If the connection possibility score between the two stroke segments is greater than a preset threshold (for example, 0.5), an edge connecting the two nodes is added to the graph, and the weight of the edge is the connection possibility score.

[0049] Step S24: performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; In an embodiment of the present invention, the stroke connection graph constructed in step S23 is input. Stroke constraints are defined, including: 1) length constraint: the length of the merged strokes shall not exceed a preset maximum length threshold (for example, half the width of the image); 2) angle constraint: the angle between adjacent stroke segments shall not exceed a preset maximum angle threshold (for example, 150 degrees); 3) loop detection: avoid forming loop strokes. According to the weights of the edges of the stroke connection graph and the stroke constraints, the Kruskal algorithm is used to solve the weighted minimum spanning tree. During the construction of the minimum spanning tree, if adding an edge will cause a violation of the stroke constraints, the edge is skipped. According to the results of the minimum spanning tree, the connected stroke segments are merged into complete strokes. After merging, the center point coordinates, main direction, length, curvature and other features of each stroke are recalculated, and this information is stored in the reconstructed stroke set.

[0050] Step S25: generating a stroke structure diagram according to the reconstructed stroke set to obtain a stroke structure diagram; In an embodiment of the present invention, the reconstructed stroke set generated in step S24 is input. A graph structure is created in which each node represents a reconstructed stroke. For each pair of strokes, the spatial relationship between them is calculated. If two strokes intersect, an edge is added to the graph to connect the corresponding two nodes, and the angle and position of the intersection are recorded. If the two strokes do not intersect but the distance between the endpoints is less than a preset threshold, an edge is also added to the graph, and the distance and direction between the two endpoints are recorded. The stroke structure graph finally generated contains nodes (representing strokes) and edges (representing the spatial relationship between strokes), and the attributes of the edges include information such as distance, direction, and intersection angle.

[0051] Preferably, step S24 comprises the following steps: Step S241: defining stroke constraints according to the candidate stroke segment set to obtain stroke constraint definition data; Step S242: Calculate edge weights according to the stroke connection graph to obtain edge weights; Solve the weighted minimum spanning tree for edge weights to obtain the minimum spanning tree; Step S243: performing multi-stroke segment connection processing on the candidate stroke segment set according to the minimum spanning tree to obtain a reconstructed stroke set.

[0052] In the embodiment of the present invention, the candidate stroke segment set is analyzed and the following stroke constraints are defined to guide the stroke reconstruction process: 1. Length constraint: The length of the merged stroke must not exceed half of the image width. Calculate the length of each candidate stroke segment and set the length threshold to image width * 0.5. In the subsequent stroke merging process, if the stroke length obtained after merging two stroke segments exceeds the threshold, merging is prohibited.

[0053] 2. Angle constraint: The angle between two connected stroke segments must not exceed 150 degrees. Calculate the main direction of each candidate stroke segment. In the subsequent stroke merging process, if the angle between the main directions of two stroke segments is greater than 150 degrees, reduce their connection priority. The angle calculation uses the vector dot product formula: cos(theta) = dot(v1, v2) / (norm(v1) * norm(v2)), where v1 and v2 are the main direction vectors of the two stroke segments.

[0054] 3. Distance constraint: The minimum Euclidean distance between the endpoints of two connected stroke segments must not exceed 3 times the length of the shorter stroke segment. Calculate the minimum endpoint distance between each pair of stroke segments. If the distance exceeds 3 times the length of the shorter stroke segment, reduce its connection priority.

[0055] 4. Loop constraint: The connection relationship of strokes must not form a loop. During the stroke merging process, a union-find data structure is used to detect and avoid the formation of loops. If merging two stroke segments will form a loop, the merging is prohibited.

[0056] Input the stroke connection graph constructed in step S23 and the stroke constraints defined in step S241. For each edge in the connection graph, calculate its weight. The weight calculation formula is as follows: weight = connection_probability * penalty_length * penalty_angle *penalty_distance; in: connection_probability is the connection probability score calculated in step S23; penalty_length is the length constraint penalty factor. If the length of two stroke segments after merging exceeds the threshold, penalty_length = 0, otherwise penalty_length = 1; penalty_angle is the angle constraint penalty factor, penalty_angle = 1 - (angle - 150) / (180 - 150), where angle is the angle between the main directions of the two stroke segments. If the angle is less than 150 degrees, penalty_angle = 1; penalty_distance is the distance constraint penalty factor, penalty_distance = 1 - distance / (3 * min_length), where distance is the minimum distance between the endpoints of two stroke segments, and min_length is the shorter length of the two stroke segments. If the distance is less than 3 times the shorter stroke length, penalty_distance = 1.

[0057] After calculating the weights of all edges, the Kruskal algorithm is used to solve the weighted minimum spanning tree. The Kruskal algorithm sorts all edges from smallest to largest weights and then tries to add the edges to the spanning tree one by one. If adding an edge would result in a loop (detected using a union-find), the edge is skipped. This process is repeated until all nodes are connected to the spanning tree.

[0058] Input the minimum spanning tree and candidate stroke fragment set generated in step S242. According to the connection relationship of the minimum spanning tree, merge the connected stroke fragments into a complete stroke. Specifically, traverse each edge in the minimum spanning tree, and merge the pixel coordinate lists of the two stroke fragments connected by the edge into a new list. Then, based on the merged pixel coordinate list, recalculate the center point coordinates, main direction, length, curvature and other features of the stroke. Store all reconstructed stroke information in the reconstructed stroke set. If a stroke fragment is not connected to any other stroke fragments in the minimum spanning tree, add it as an independent stroke to the reconstructed stroke set.

[0059] Preferably, step S3 comprises the following steps: Step S31: obtaining a font sample; extracting a stroke structure graph from the font sample, and constructing a multi-scale stroke structure graph library to obtain a multi-scale stroke structure graph library; Step S32: constructing a font feature library according to the multi-scale stroke structure library to obtain a font feature library; Step S33: performing graph similarity calculation on the multi-scale stroke structure library and the character feature library to obtain a character similarity matrix; Step S34: Screening candidate characters from the character similarity matrix to obtain a list of candidate characters and similarity scores; Step S35: construct a candidate character set for the candidate characters and similarity score lists to obtain a candidate character set.

[0060] In an embodiment of the present invention, a standard character font sample image containing various fonts (e.g., Songti, Heiti, Kaiti, Times New Roman, Arial) and font sizes (e.g., 12pt, 16pt, 24pt) is obtained. For each character image of each font and font size, a binarization process is first performed to convert the character pixels to 1 and the background pixels to 0. Then, the same stroke extraction and reconstruction method as step S2 is used to obtain the stroke structure diagram of the character. In order to construct a multi-scale stroke structure library, the stroke structure diagram of each character is scaled at different ratios, for example, the scaling ratios are 0.5, 0.75, 1.0, 1.25 and 1.5. The scaled stroke structure diagram is stored in the multi-scale stroke structure library, and the character encoding (e.g., Unicode encoding), font, font size and scaling ratio corresponding to each stroke structure diagram are recorded.

[0061] The glyph feature library is constructed using the multi-scale stroke structure library constructed in step S31. The glyph feature library stores the feature vector of each character. For each stroke structure diagram in the multi-scale stroke structure library, the following features are extracted: 1) the number of strokes; 2) the length, direction and curvature of each stroke; 3) the connection relationship between the strokes (including the connection type, distance, direction and angle). These features are combined into a feature vector of fixed length. The feature vector and its corresponding character encoding, font, font size and scaling ratio are stored in the glyph feature library.

[0062] Input the stroke structure graph generated in step S2, the multi-scale stroke structure graph library constructed in step S31, and the glyph feature library constructed in step S32. First, according to the overall size of the stroke structure graph (for example, the area of ​​the stroke bounding box), select a set of stroke structure graphs with the closest scale from the multi-scale stroke structure graph library. Then, for each selected stroke structure graph, use the graph edit distance algorithm to calculate its similarity with the input stroke structure graph. The graph edit distance algorithm calculates the minimum number of editing operations (for example, adding nodes, deleting nodes, adding edges, deleting edges) required to convert one graph into another. The fewer the number of editing operations, the higher the similarity. The calculated similarity scores are stored in the character similarity matrix, where the rows of the matrix correspond to the input stroke structure graphs, the columns correspond to the characters in the glyph feature library, and the matrix element values ​​are the corresponding similarity scores.

[0063] Input the character similarity matrix calculated in step S33. Set a similarity threshold (e.g., 0.8). Filter out characters with similarity scores higher than the threshold, and store these characters and their corresponding similarity scores in a candidate character and similarity score list. Sort the list from high to low according to the similarity score.

[0064] Input the list of candidate characters and similarity scores obtained by screening in step S34. Select the N characters with the highest scores (for example, N=5) and their corresponding similarity scores from the list to form a candidate character set. The candidate character set will be used as the input of step S4.

[0065] Preferably, step S33 includes the following steps: Step S331: performing area similarity calculation on the multi-scale stroke structure library, and performing scale matching according to the character feature library to obtain matching scale selection data; Step S332: Obtain important stroke data; define important stroke structures according to the important stroke data to obtain important stroke structure definition data; Step S333: extracting local structural features from the multi-scale stroke structure library according to the important stroke structure definition data to obtain a local structural feature vector; Step S334: Calculate graph similarity based on the multi-scale stroke structure library, the local structure feature vector and the glyph feature library to obtain a character similarity matrix.

[0066] In an embodiment of the present invention, the input is a multi-scale stroke structure library and a glyph feature library. First, the area of ​​the input stroke structure diagram is calculated. The area calculation method is: add the minimum circumscribed rectangle areas of all strokes in the stroke structure diagram. Then, for each character in the glyph feature library, calculate the area of ​​its stroke structure diagram at different scales. The method for calculating the area is the same as that of the input stroke structure diagram. Next, calculate the absolute value of the difference between the area of ​​the input stroke structure diagram and the area of ​​the stroke structure diagram of each character in the glyph feature library at different scales. For each character, select the scale with the smallest absolute value of the area difference as the best matching scale for the character. The best matching scale of each character and the corresponding absolute value of the area difference are stored in the matching scale selection data.

[0067] Get predefined important stroke data, which contains description information of various important stroke structures, such as horizontal, vertical, left-falling stroke, right-falling stroke, dot, hook, etc. Define the judgment rules of important stroke structures according to the important stroke data. For example, the judgment rule of horizontal strokes is: the main direction angle is between -15 degrees and 15 degrees, and the length is greater than 1.5 times the average stroke length; the judgment rule of vertical strokes is: the main direction angle is between 75 degrees and 105 degrees, and the length is greater than 1.5 times the average stroke length; the judgment rule of dots is: the length is less than 0.5 times the average stroke length. Store these judgment rules as important stroke structure definition data.

[0068] The input is a multi-scale stroke structure library and important stroke structure definition data. For each stroke structure diagram in the multi-scale stroke structure library, according to the rules defined in step S332, determine which important stroke structure each stroke belongs to (for example, horizontal, vertical, left-falling stroke, right-falling stroke, dot, etc.), and record the result. Then, count the number of times each important stroke structure appears in the stroke structure diagram. In addition, calculate the average length, average direction and average curvature of each important stroke structure. Combine these statistical information into a feature vector, namely the local structure feature vector.

[0069] The input is a multi-scale stroke structure library, a local structure feature vector and a glyph feature library. First, according to the matching scale selection data obtained in step S331, the stroke structure diagram of the best matching scale of each character is selected from the multi-scale stroke structure library. Then, for the input stroke structure diagram and the stroke structure diagram of the best matching scale of each character in the glyph feature library, their graph similarity is calculated. The graph similarity calculation method is as follows: 1. Structural similarity calculation: The structural similarity between two stroke structure graphs is calculated using the graph edit distance algorithm. The graph edit distance algorithm calculates the minimum number of edit operations (e.g., adding nodes, deleting nodes, adding edges, deleting edges) required to transform one graph into another. The fewer the number of edit operations, the higher the structural similarity.

[0070] 2. Local feature similarity calculation: Calculate the cosine similarity of the local structural feature vectors of two stroke structure graphs.

[0071] 3. Final similarity calculation: Take the weighted average of the structural similarity and local feature similarity to get the final graph similarity score. The weights can be set based on experience, for example, the structural similarity weight is 0.7 and the local feature similarity weight is 0.3.

[0072] The calculated similarity scores are stored in a character similarity matrix. The rows of the matrix correspond to the input stroke structure diagram, the columns correspond to the characters in the glyph feature library, and the matrix element values ​​are the corresponding similarity scores.

[0073] Preferably, step S333 is specifically: Perform stroke direction judgment on the multi-scale stroke structure library to obtain stroke direction judgment data; perform stroke length judgment on the multi-scale stroke structure library to obtain stroke length judgment data; perform stroke shape judgment on the multi-scale stroke structure library to obtain stroke shape judgment data; Performing important stroke structure marking processing according to the stroke direction judgment data, the stroke length judgment data and the stroke shape judgment data to obtain an important stroke structure mark; Calculate the similarity of important stroke structures for the multi-scale stroke structure library and the glyph feature library according to the important stroke structure markers, and perform relative position calculations to obtain the local feature calculation results; Construct feature vectors from the local feature calculation results to obtain local structure feature vectors.

[0074] In the embodiments of the present invention, each stroke structure diagram in the multi-scale stroke structure library is traversed. For each stroke, calculate its main direction angle. The main direction angle calculation method is principal component analysis (PCA). Treat the stroke pixel coordinates as two-dimensional data points, calculate the covariance matrix, and the eigenvector corresponding to the largest eigenvalue of the covariance matrix is the main direction vector. Convert the main direction vector into an angle value. Store the angle value in the stroke direction judgment data. Traverse each stroke structure diagram in the multi-scale stroke structure library. For each stroke, calculate its length. The stroke length calculation method is to calculate the length of the stroke pixel contour. Store the length value in the stroke length judgment data. Traverse each stroke structure diagram in the multi-scale stroke structure library. For each stroke, calculate its shape complexity. The shape complexity calculation method is the ratio of the perimeter of the stroke pixel contour to the perimeter of its minimum bounding rectangle. Store the shape complexity value in the stroke shape judgment data.

[0075] Important stroke structure markers: Based on the stroke direction judgment data, stroke length judgment data, and stroke shape judgment data, perform important stroke structure markers on each stroke. The rules are as follows: Horizontal: The main direction angle is between -15 degrees and 15 degrees, and the length is greater than 1.5 times the average stroke length.

[0076] Vertical: The main direction angle is between 75 degrees and 105 degrees, and the length is greater than 1.5 times the average stroke length.

[0077] Left-falling stroke: The main direction angle is between -60 degrees and -30 degrees, and the shape complexity is greater than 1.2.

[0078] Right-falling stroke: The main direction angle is between 30 degrees and 60 degrees, and the shape complexity is greater than 1.2.

[0079] Dot: The length is less than 0.5 times the average stroke length.

[0080] Mark each stroke as "horizontal", "vertical", "left-falling stroke", "right-falling stroke", "dot", or "other".

[0081] Calculation of important stroke structure similarity: For the input stroke structure diagram and the stroke structure diagram of each character in the glyph feature library, calculate the similarity of the important stroke structures between them. First, count the number of each important stroke structure. Then, for each important stroke structure, calculate the absolute value of the difference in the number of this structure in the two stroke structure diagrams. Add up all the differences and take the average value as the important stroke structure similarity score.

[0082] Relative position calculation: For the input stroke structure diagram and the stroke structure diagram of each character in the glyph feature library, calculate the relative position relationship between the important stroke structures. Divide the image plane into 3x3 grids. Record the grid area where the center point of each important stroke is located. Calculate the difference in the grid area where the center points of the same type of important strokes in the two stroke structure diagrams are located. Add all the differences and take the average value as the relative position score.

[0083] Feature vector construction: The important stroke structure similarity scores and relative position scores are combined into a feature vector, namely the local structure feature vector.

[0084] Preferably, step S4 comprises the following steps: Step S41: obtaining a preceding text; using a pre-trained character embedding model to encode candidate character features of a candidate character set to obtain a candidate character feature vector; Step S42: using the pre-trained Transformer language model to encode the preceding text sequence to obtain a context vector; Step S43: fusing the candidate character feature vector and the context vector with context information to obtain a fused feature vector; Step S44: performing optimal character prediction on the fused feature vector to obtain a predicted character; Step S45: splicing the predicted characters into the preceding text to construct a correction character sequence to obtain a correction character sequence; Step S46: converting the corrected character sequence into a text format to obtain a formatted text; and displaying the formatted text to obtain a display result.

[0085] In an embodiment of the present invention, the recognized preceding text sequence is read from the cache of the scanning pen. If the scanning pen has just started working, the preceding text sequence is empty. A pre-trained character embedding model (e.g., BERT's character-level embedding) is used to encode each character in the candidate character set to obtain a candidate character feature vector. The feature vector dimension of each candidate character is the hidden layer dimension of the pre-trained model (e.g., 768). The similarity score of each candidate character calculated in step S3 is added to the corresponding feature vector to expand the feature vector dimension.

[0086] Using a pre-trained Transformer language model (e.g., BERT), encode the preceding text sequence obtained in step S41 into a context vector. Specifically, input the preceding text sequence into the BERT model and obtain the hidden state of its output. The output of the last time step of the hidden state is used as the context vector. The dimension of the context vector is the same as the hidden layer dimension of the BERT model (e.g., 768). If the preceding text sequence is empty, initialize the context vector to an all-zero vector.

[0087] The candidate character feature vector obtained in step S41 and the context vector obtained in step S42 are fused. The fusion method adopts the attention mechanism. Specifically, the context vector is used as the query, the candidate character feature vector is used as the key and value, and the attention weight is calculated. Then, the candidate character feature vector is weighted and summed using the attention weight to obtain a fused feature vector. The dimension of the fused feature vector is the same as that of the candidate character feature vector.

[0088] The fused feature vector obtained in step S43 is input into a classifier consisting of a fully connected layer and a softmax layer. The output dimension of the fully connected layer is equal to the size of the candidate character set. The softmax layer converts the output of the fully connected layer into a probability distribution. The character with the highest probability is selected as the predicted character.

[0089] The predicted characters obtained in step S44 are added to the end of the preceding text sequence obtained in step S41 to construct a new corrected character sequence.

[0090] The corrected character sequence obtained in step S45 is converted into a user-readable text format. For example, if the corrected character sequence contains Unicode encoding, it is converted into corresponding characters. The formatted text result is displayed on the screen of the scanning pen, or transmitted to a device connected thereto.

[0091] Preferably, the present invention further provides an image recognition system for a scanning pen, which is used to execute the image recognition method for a scanning pen as described above. The image recognition system for a scanning pen comprises: The semantic pre-segmentation module is used to obtain the original image; generate superpixels for the original image to obtain a superpixel set; train the semantic segmentation network based on the superpixel perception layer added according to the pre-built initial U-Net network model to obtain a semantic segmentation model; use the semantic segmentation model and the superpixel set to predict the superpixel semantic category, and construct a superpixel semantic map to obtain a superpixel semantic map; The stroke reconstruction module is used to extract stroke fragments from the superpixel semantic map and construct a stroke connection map to obtain a stroke connection map; to impose stroke constraints on the stroke connection map and reconstruct the strokes to obtain a reconstructed stroke set; and to generate a stroke structure map based on the reconstructed stroke set to obtain a stroke structure map; The glyph matching module is used to obtain font samples; construct a multi-scale stroke structure library based on the font samples, and construct a glyph feature library to obtain a multi-scale stroke structure library and a glyph feature library; perform graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; perform glyph matching based on the character similarity matrix to obtain a candidate character set; The context correction module is used to obtain the previous text; fuse the vector context information of the candidate character set and the previous text to obtain a fused feature vector; predict the optimal character of the fused feature vector to obtain a predicted character; splice the predicted character into the previous text to construct a correction character sequence to obtain a correction character sequence; output the text result of the correction character sequence to obtain a display result.

[0092] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0093] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A method for image recognition using a scanning pen, characterized in that: The following steps are involved: Step S1: obtaining an original image; generating superpixels for the original image to obtain a superpixel set; The pre-built initial U-Net network model is trained with a semantic segmentation network based on the superpixel perception layer to obtain a semantic segmentation model; the semantic segmentation model and the superpixel set are used to predict the superpixel semantic category and construct a superpixel semantic map to obtain a superpixel semantic map; Step S2: extracting stroke fragments from the superpixel semantic graph and constructing a stroke connection graph to obtain a stroke connection graph; performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; generating a stroke structure graph based on the reconstructed stroke set to obtain a stroke structure graph; Step S3: obtaining font samples; constructing a multi-scale stroke structure library and a glyph feature library based on the font samples to obtain a multi-scale stroke structure library and a glyph feature library; performing graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; performing glyph matching based on the character similarity matrix to obtain a candidate character set; Step S4: Obtain the preceding text; The candidate character set and the preceding text are vector-context-fused to obtain a fused feature vector. The feature vectors are fused to perform optimal character prediction to obtain predicted characters; the predicted characters are concatenated into the preceding text to construct a correction character sequence to obtain a correction character sequence; and the correction character sequence is output as a text result to obtain a display result.

2. The image recognition method for a scanning pen according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: obtaining an original image through a scanning pen camera; generating superpixels for the original image to obtain a superpixel set; Step S12: extracting superpixel features based on the superpixel set and the original image to obtain a superpixel feature vector; Step S13: obtaining a training data set; adding a superpixel perception layer to the pre-built initial U-Net network model, and performing semantic segmentation network training according to the training data set to obtain a semantic segmentation model; Step S14: inputting the superpixel feature vector into the semantic segmentation model to predict the superpixel semantic category to obtain the superpixel semantic category; Step S15: construct a superpixel semantic map according to the superpixel set and the superpixel semantic category to obtain a superpixel semantic map.

3. The image recognition method for a scanning pen according to claim 2, characterized in that: Step S13 includes the following steps: Step S131: performing data enhancement on the training data set to obtain an enhanced training image set; Step S132: integrating the superpixel perception layer of the pre-constructed initial U-Net network model according to the superpixel feature vector to obtain a U-Net model integrating the superpixel perception layer; Step S133: defining a weighted cross entropy loss function according to the enhanced training image set to obtain a weighted cross entropy loss function; Step S134: Perform semantic segmentation network training according to the enhanced training image set, the weighted cross entropy loss function, and the U-Net model with integrated superpixel perception layer to obtain a semantic segmentation model.

4. The image recognition method for a scanning pen according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: extracting text regions from the superpixel semantic map to obtain a text region superpixel set; Step S22: extracting stroke segments according to the text area superpixel set and the original image to obtain a candidate stroke segment set; Step S23: constructing a stroke connection graph for the candidate stroke segment set to obtain a stroke connection graph; Step S24: performing stroke constraints on the stroke connection graph and reconstructing the strokes to obtain a reconstructed stroke set; Step S25: generating a stroke structure diagram according to the reconstructed stroke set to obtain a stroke structure diagram.

5. The image recognition method for a scanning pen according to claim 4, characterized in that: Step S24 includes the following steps: Step S241: defining stroke constraints according to the candidate stroke segment set to obtain stroke constraint definition data; Step S242: Calculate edge weights according to the stroke connection graph to obtain edge weights; Solve the weighted minimum spanning tree for the edge weights to obtain the minimum spanning tree; Step S243: performing multi-stroke segment connection processing on the candidate stroke segment set according to the minimum spanning tree to obtain a reconstructed stroke set.

6. The image recognition method for a scanning pen according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: obtaining a font sample; extracting a stroke structure graph from the font sample, and constructing a multi-scale stroke structure graph library to obtain a multi-scale stroke structure graph library; Step S32: constructing a font feature library according to the multi-scale stroke structure library to obtain a font feature library; Step S33: performing graph similarity calculation on the multi-scale stroke structure graph library and the character feature database to obtain a character similarity matrix; Step S34: Screening candidate characters from the character similarity matrix to obtain a list of candidate characters and similarity scores; Step S35: construct a candidate character set for the candidate characters and similarity score lists to obtain a candidate character set.

7. The image recognition method for a scanning pen according to claim 6, characterized in that: Step S33 includes the following steps: Step S331: performing area similarity calculation on the multi-scale stroke structure library, and performing scale matching according to the character feature library to obtain matching scale selection data; Step S332: Obtain important stroke data; define important stroke structures according to the important stroke data to obtain important stroke structure definition data; Step S333: extracting local structural features from the multi-scale stroke structure library according to the important stroke structure definition data to obtain a local structural feature vector; Step S334: Calculate graph similarity based on the multi-scale stroke structure library, the local structure feature vector and the glyph feature library to obtain a character similarity matrix.

8. The image recognition method for a scanning pen according to claim 7, characterized in that: Step S333 is specifically as follows: Perform stroke direction judgment on the multi-scale stroke structure library to obtain stroke direction judgment data; perform stroke length judgment on the multi-scale stroke structure library to obtain stroke length judgment data; perform stroke shape judgment on the multi-scale stroke structure library to obtain stroke shape judgment data; Performing important stroke structure marking processing according to the stroke direction judgment data, the stroke length judgment data and the stroke shape judgment data to obtain an important stroke structure mark; According to the important stroke structure marks, the important stroke structure similarity is calculated for the multi-scale stroke structure library and the character feature library, and the relative position is calculated to obtain the local feature calculation result; The feature vector is constructed based on the local feature calculation results to obtain the local structure feature vector.

9. The image recognition method for a scanning pen according to claim 1, characterized in that: Step S4 includes the following steps: Step S41: obtaining a preceding text; using a pre-trained character embedding model to encode candidate character features of a candidate character set to obtain a candidate character feature vector; Step S42: using the pre-trained Transformer language model to encode the preceding text sequence to obtain a context vector; Step S43: fusing the candidate character feature vector and the context vector with context information to obtain a fused feature vector; Step S44: performing optimal character prediction on the fused feature vector to obtain a predicted character; Step S45: splicing the predicted characters into the preceding text to construct a correction character sequence to obtain a correction character sequence; Step S46: converting the corrected character sequence into a text format to obtain a formatted text; and displaying the formatted text to obtain a display result.

10. An image recognition system for a scanning pen, characterized in that: The image recognition system for the scanning pen is used to perform the image recognition method for the scanning pen as claimed in claim 1, and the image recognition system for the scanning pen comprises: The semantic pre-segmentation module is used to obtain the original image; generate superpixels for the original image to obtain a superpixel set; train the semantic segmentation network based on the superpixel perception layer added according to the pre-built initial U-Net network model to obtain a semantic segmentation model; use the semantic segmentation model and the superpixel set to predict the superpixel semantic category, and construct a superpixel semantic map to obtain a superpixel semantic map; The stroke reconstruction module is used to extract stroke fragments from the superpixel semantic map and construct a stroke connection map to obtain a stroke connection map; to impose stroke constraints on the stroke connection map and reconstruct the strokes to obtain a reconstructed stroke set; and to generate a stroke structure map based on the reconstructed stroke set to obtain a stroke structure map; The glyph matching module is used to obtain font samples; construct a multi-scale stroke structure library based on the font samples, and construct a glyph feature library to obtain a multi-scale stroke structure library and a glyph feature library; perform graph similarity calculation on the multi-scale stroke structure library and the glyph feature library to obtain a character similarity matrix; perform glyph matching based on the character similarity matrix to obtain a candidate character set; The context correction module is used to obtain the previous text; fuse the vector context information of the candidate character set and the previous text to obtain a fused feature vector; predict the optimal character of the fused feature vector to obtain a predicted character; splice the predicted character into the previous text to construct a correction character sequence to obtain a correction character sequence; output the text result of the correction character sequence to obtain a display result.

Images