Character recognition method and device for handwritten image, terminal equipment and storage medium

Abstract

The invention discloses a character recognition method and device for a handwritten image, terminal equipment and a storage medium, and the method comprises the steps: obtaining the handwritten image, and converting the handwritten image into a single-channel gray-scale map; wherein the single-channel grey-scale map is an image with a white background and black characters; performing reverse binarization processing on the single-channel grey-scale map to generate a reverse binarization image with black-matrix white characters; all contours in the reverse binarization image are recognized, and the contour with the largest area serves as a character core area; calculating an amplification coefficient according to the width and height of the reverse binarization image; performing boundary expansion on the character core region according to the amplification coefficient, and performing cutting based on the expanded boundary to obtain a cut image; and performing character recognition according to the clipped image. According to the invention, the character recognition accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the field of image analysis and processing, and in particular to a method, apparatus, terminal device, and storage medium for recognizing characters in handwritten images. Background Technology

[0002] With the widespread adoption of mobile internet and smart terminals, handwriting input, as an interaction method that conforms to users' natural writing habits, is widely used in text input scenarios on devices such as mobile phones, tablets, and handwriting tablets. It is especially suitable for the elderly, children, and users who are not familiar with Pinyin / Wubi input methods, or for scenarios that require quick input of rare characters and professional symbols (such as educational notes, medical records, and ancient book collation).

[0003] However, handwritten images are prone to blurring in low-light environments, making it easy to lose key stroke features in cursive or messy handwriting, resulting in lower accuracy in character recognition. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal device, and storage medium for recognizing characters in handwritten images, which can solve the problem of low accuracy in character recognition in the prior art.

[0005] An embodiment of the present invention discloses a method, apparatus, terminal device and storage medium for character recognition of handwritten images, comprising: acquiring a handwritten image and converting the handwritten image into a single-channel grayscale image; wherein the single-channel grayscale image is an image with black text on a white background; The single-channel grayscale image is subjected to inverse binarization to generate an inverse binarized image with black background and white text. Identify all contours in the inverse binarized image and use the contour with the largest area as the character core region; Calculate the amplification coefficients based on the width and height of the inverse binarized image; The character core region is expanded according to the amplification coefficient, and cropped based on the expanded boundary to obtain the cropped image. Text recognition based on cropped images.

[0006] Further reverse binarization processing is performed on the single-channel grayscale image to generate a reverse binarized image with black background and white text, including: The single-channel grayscale image is divided into several image blocks. For each image block, the mean and standard deviation of grayscale values ​​of all pixels in the image block are calculated. The dynamic segmentation threshold of the image block is calculated based on the mean and standard deviation of grayscale values. Linear interpolation is performed on the dynamic segmentation thresholds of all image patches to generate a dynamic continuous threshold matrix covering the entire image; For each pixel, if the gray value of the pixel is less than or equal to the dynamic threshold at the corresponding position in the dynamic continuous threshold matrix, then the gray value of the pixel is set to 255; otherwise, the gray value of the pixel is set to 0, generating a reverse binarized image with black background and white text.

[0007] Furthermore, the text recognition based on the cropped image includes: Scale the cropped image to obtain the scaled image; Connected components are extracted from the scaled image, and the total number of pixels and minimum bounding rectangle of each connected component are recorded; the pixel density of each connected component is calculated based on the total number of pixels and minimum bounding rectangle of each connected component. Skeletonize each connected component to obtain the skeleton line corresponding to each connected component; For each connected component, iterate through all pixels on the skeleton line and count the number of branches for each pixel. Connected regions with a number of branches greater than the preset number of branches, a pixel density greater than the preset density, and an aspect ratio of the minimum bounding rectangle within the preset ratio are considered as intersecting regions, while the remaining connected regions are considered as non-intersecting regions. For non-intersecting regions, morphological opening is performed using a 3×3 square structuring element. For intersecting regions, erosion is performed once using a 1×1 structuring element. Then, dilation is performed along the corresponding direction of the skeleton lines using a long strip structuring element to obtain the morphologically processed image. The morphologically processed image is converted into an RGB image and then input into a character recognition model to identify the characters corresponding to the handwritten image.

[0008] Furthermore, the text recognition model includes: an input layer, several convolutional modules, several residual modules, and a classification head; except for the initial convolutional module, all other convolutional modules are depthwise classifiable convolutional modules, and there is a preset number of depthwise classifiable convolutional modules connected to the residual modules; The step of converting the morphologically processed image into an RGB image and then inputting it into a character recognition model to identify the characters corresponding to the handwritten image includes: The input layer receives the RGB image, extracts the feature map corresponding to the RGB image based on each convolution module, and generates the text recognition result based on the final feature map through the classification head.

[0009] Furthermore, the training of the character recognition model includes: Acquire several handwritten image samples and set a label for each handwritten image sample; The handwritten image samples are divided into several batches of training samples. The preset neural network model is iteratively trained according to each batch of training samples until the preset number of rounds or the accuracy convergence is reached, and the text recognition model is obtained. In each training session, a neural network model to be trained is acquired. If it is the first training session, the neural network model to be trained is an initialized neural network model. If it is not the first training session, the model parameters of the already stored optimal neural network model are extracted according to a preset storage path, and the neural network model to be trained is constructed. A batch of training samples is input into the neural network model to be trained so that the model can recognize the corresponding text categories; The identified text categories are compared with their corresponding labels, and the loss function is calculated based on the comparison results. The model parameters of the neural network model to be trained are adjusted according to the loss function to obtain the updated neural network model; The updated neural network model is validated using a validation set to obtain its accuracy. The accuracy of the updated neural network model is then compared with the accuracy of the stored optimal neural network model. The neural network model with the higher accuracy is selected as the updated optimal neural network model, and the model parameters of the updated optimal neural network model are stored according to the storage path.

[0010] Furthermore, the step of acquiring the handwritten image includes: receiving a text recognition request for the handwritten image, determining whether the request has an independent image data field, and if so, calling a first data interface to extract the handwritten image from the image data field through the first data interface; if not, calling a second data interface to extract the imeParam nested object through the second data interface, and extracting the handwritten image from the imeParam nested object.

[0011] Furthermore, before receiving a text recognition request for a handwritten image, the process also includes: The encrypted string parameter is obtained from the user based on the token interface; the encrypted string parameter includes: application ID, username, password and timestamp; Decrypt the encrypted string parameters to extract the application ID, username, password, and timestamp; The timestamp data is verified based on the timestamp, and if the timestamp data verification passes, the username and password are entered into the database for matching. If the same username and password are matched in the database, a token is generated based on the application ID, username, and password. The token is stored and sent back to the user's corresponding front-end device, so that the front-end device can generate a text recognition request carrying the token based on the token and the user's handwritten image.

[0012] Another embodiment of the present invention provides a character recognition device for handwritten images, including: a grayscale image conversion module, a reverse binarization module, a core region recognition module, an amplification coefficient calculation module, a boundary expansion module, and a recognition module; The grayscale image conversion module is used to acquire a handwritten image and convert the handwritten image into a single-channel grayscale image; wherein, the single-channel grayscale image is an image with black text on a white background; The reverse binarization module is used to perform reverse binarization processing on the single-channel grayscale image to generate a reverse binarized image with black background and white text. The core region recognition module is used to recognize all contours in the reverse binarized image and take the contour with the largest area as the character core region. The amplification coefficient calculation module is used to calculate the amplification coefficient based on the width and height of the inverse binarized image; The boundary expansion module is used to expand the boundary of the character core region according to the amplification coefficient, and to crop the image based on the expanded boundary. The recognition module is used to perform text recognition based on the cropped image.

[0013] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the handwritten image text recognition method of the present invention.

[0014] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform steps such as the character recognition method for handwritten images of the present invention.

[0015] The following benefits can be obtained by implementing the present invention: The present invention provides a method, an apparatus, a terminal device and a storage medium for recognizing text in a handwritten image. The method first converts the handwritten image input by the user into a single-channel grayscale image, and then performs reverse binarization processing on the single-channel grayscale image to generate a reverse binarized image with a black background and white characters. Through the reverse binarization processing, the white background and black characters can be reversed to a black background and white characters, strengthening the contrast between the character edges and the background, and solving the problem of blurred handwriting in low-light scenarios. Immediately afterwards, all the contours in the reverse binarized image are recognized, and the contour with the largest area is used as the character core area; thus solving the problem of regional positioning deviation caused by connected handwriting and overlapping handwriting contours. Then, according to the width and height of the reverse binarized image, an amplification coefficient is calculated; the character core area is expanded based on the amplification coefficient, and cropping is performed based on the expanded boundary to obtain a cropped image; by amplifying the character core area according to the amplification coefficient calculated based on the width and height of the reverse binarized image, it is possible to avoid losing the edge strokes of connected handwriting during cropping (such as the extended part of "辶" or "走之底"). Finally, text recognition is performed based on the cropped image, thereby improving the accuracy of text recognition. Brief Description of the Drawings

[0016] In order to more clearly illustrate the technical solutions of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 It is a flowchart of the method for recognizing text in a handwritten image provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of the apparatus for recognizing text in a handwritten image provided by an embodiment of the present invention. Detailed Embodiments

[0018] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0020] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0023] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0024] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0025] See Figure 1 An embodiment of the present invention provides a method for recognizing characters in a handwritten image, comprising: S1. Obtain a handwritten image and convert the handwritten image into a single-channel grayscale image; wherein the single-channel grayscale image is an image with black text on a white background.

[0026] Specifically, the user performs handwriting operations based on a front-end device (such as a mobile phone, tablet, etc.) to generate a handwritten image in RGB format. After obtaining the handwritten image, it is converted to grayscale to generate the aforementioned single-channel grayscale image.

[0027] In a preferred embodiment, obtaining a handwritten image includes: receiving a text recognition request for the handwritten image; determining whether the request has an independent image data field; if so, calling a first data interface to extract the handwritten image from the image data field; if not, calling a second data interface to extract an imeParam nested object and extracting the handwritten image from the imeParam nested object.

[0028] In a preferred embodiment, before receiving a text recognition request for a handwritten image, the method further includes: The encrypted string parameter is obtained from the user based on the token interface; the encrypted string parameter includes: application ID, username, password and timestamp; Decrypt the encrypted string parameters to extract the application ID, username, password, and timestamp; The timestamp data is verified based on the timestamp, and if the timestamp data verification passes, the username and password are entered into the database for matching. If the same username and password are matched in the database, a token is generated based on the application ID, username, and password. The token is stored and sent back to the user's corresponding front-end device, so that the front-end device can generate a text recognition request carrying the token based on the token and the user's handwritten image.

[0029] Existing handwritten image text recognition technologies suffer from limited service interfaces, making them incompatible with different front-end parameter formats and requiring custom front-end development. Furthermore, they lack cross-domain support and fault tolerance, with invalid images or Base64 errors easily causing service crashes and increasing the difficulty of multi-terminal (APP / webpage / mini-program) integration. To address this issue, this invention obtains encrypted string parameters from the user's front-end device via a token interface. These encrypted string parameters primarily consist of the application ID, username, password, and timestamp (e.g., 14406043100-US001-US00889-1669799637114), encrypted and decrypted using an agreed-upon unified encryption / decryption key via RSA algorithm. Upon receiving the encrypted string parameter, the RSA algorithm is used for decryption to extract the application ID, username, password, and timestamp. Next, the timestamp is verified to determine if it is within the validity period. If not, the timestamp verification fails, and the process terminates. If it is, the timestamp verification passes, and the username and password are matched against those pre-stored in the database. If a match is found, verification succeeds. A token is then created based on the application ID, username, and password, stored in the Redis cache, and returned to the user's front-end device. If no match is found, authentication fails, and the process terminates.

[0030] After the user inputs a handwritten image through the front-end device, the front-end device generates a text recognition request based on the previously created token and the user's handwritten image.

[0031] Upon receiving a text recognition request, the request is parsed. If the request contains an independent image field (Base64 image string), i.e., a request for the aforementioned image data field, the basic data interface, i.e., the first data interface, is called. The first data interface adopts a flat parameter structure, which is suitable for lightweight scenarios (such as personal web pages). The Token and image fields are directly extracted through the first data interface. The base64 decoding library is called to convert the image into a binary byte stream, and then it is converted into a uint8 type two-dimensional array (pixel data), which is temporarily stored in memory to obtain the aforementioned handwritten image.

[0032] If the text recognition request does not have an independent image field, the open interface, namely the second data interface mentioned above, is called. The second data interface adopts a nested imeParam structure to adapt to complex systems (such as enterprise apps), supports transparent transmission of device information and pagination, and meets the standardized integration requirements of external systems. The nested imeParam object is extracted through the second data interface, and the imgBase64 (Base64 image) is obtained from it and temporarily stored in memory, thereby obtaining the handwritten image mentioned above. In addition, data such as device model and page number can also be extracted from the nested imeParam object and temporarily stored in memory.

[0033] S2. Perform reverse binarization on the single-channel grayscale image to generate a reverse binarized image with black background and white text; In a preferred embodiment, the single-channel grayscale image is subjected to inverse binarization processing to generate an inverse binarized image with black background and white text, including: The single-channel grayscale image is divided into several image blocks. For each image block, the mean and standard deviation of grayscale values ​​of all pixels in the image block are calculated. The dynamic segmentation threshold of the image block is calculated based on the mean and standard deviation of grayscale values. Linear interpolation is performed on the dynamic segmentation thresholds of all image patches to generate a dynamic continuous threshold matrix covering the entire image; For each pixel, if the gray value of the pixel is less than or equal to the dynamic threshold at the corresponding position in the dynamic continuous threshold matrix, then the gray value of the pixel is set to 255; otherwise, the gray value of the pixel is set to 0, generating a reverse binarized image with black background and white text.

[0034] Specifically, in this embodiment, the single-channel grayscale image is first divided into blocks, each block being 31*31 pixels in size (this balances local illumination characteristics with computational efficiency). Then, the average grayscale value of all pixels within each image block is calculated block by block. and standard deviation Then, the dynamic segmentation threshold for each image block is calculated using the following formula: Where T is the dynamic segmentation threshold corresponding to the image block; K is an empirical coefficient, which can be illustratively set to 1.2. It is negatively correlated with light intensity; that is, the stronger the light intensity, the smaller the K value, and the weaker the light intensity, the larger the K value. The specific value can be fine-tuned according to different lighting scenarios. When the light is strong, the overall grayscale of the image is high, and the difference between the grayscale of the text and the background is obvious. If the K value is too large, it will easily lead to a low T value, causing bright background pixels to be misjudged as text, resulting in noise. Therefore, the K value needs to be reduced. When the light is weak, the overall grayscale of the image is low, and the difference between the grayscale of the text and the background is blurred. If the K value is too small, it will easily lead to a high T value, causing dark pixels in the text to be misjudged as background, resulting in broken strokes. Therefore, the K value needs to be increased. Next, linear interpolation is performed on the dynamic segmentation threshold between each image block to generate a dynamic continuous threshold matrix covering the entire image, avoiding text breakage caused by abrupt threshold changes at block boundaries. Subsequently, pixel-by-pixel judgment and conversion are performed on the single-channel grayscale image based on the dynamic continuous threshold matrix. ---If the gray value of a pixel is less than or equal to the dynamic segmentation threshold T at the corresponding position, it is determined to be a handwritten character pixel, and its gray value is set to 255 (white). ---If the gray value of a pixel is greater than the dynamic segmentation threshold T at the corresponding position, it is determined to be a background pixel, and its gray value is set to 0 (black); thus completing the reverse conversion from "white background with black text" to "black background with white text", so that the blurred text outline under low light forms a strong contrast with the background.

[0035] S3. Identify all contours in the reverse binarized image and take the contour with the largest area as the character core region.

[0036] Specifically, using existing contour recognition algorithms, all contours in the inverse binarized image are identified, the area of ​​each contour is calculated, and the contour with the largest area is selected as the character core region. The boundary coordinates (min_x, max_x, min_y, max_y) of this region are recorded. Here, min_x is the minimum X-axis coordinate (horizontal coordinate) of all points in the character core region, corresponding to the leftmost position of the character core region and the left edge of the bounding rectangle corresponding to the character core region. max_x is the maximum X-axis coordinate of all points in the character core region, corresponding to the rightmost position of the character core region and the right edge of the bounding rectangle corresponding to the character core region. min_y is the minimum y-axis coordinate of all points in the character core region, corresponding to the topmost position of the character core region and the top edge of the bounding rectangle corresponding to the character core region. max_y is the maximum y-axis coordinate of all points in the character core region, corresponding to the bottommost position of the character core region and the bottom edge of the bounding rectangle corresponding to the character core region. This embodiment can solve the problem of region positioning deviation caused by overlapping contours in cursive handwriting. S4. Calculate the amplification coefficient based on the width and height of the inverse binarized image; As an illustration, the expansion factor can be calculated as 1 / 32 of the width and height of the inverse binarized image: cw = width / 32, ch = height / 32; where cw is the expansion factor in the width direction, ch is the expansion factor in the height direction, width is the width of the inverse binarized image, and height is the height of the inverse binarized image.

[0037] S5. Expand the boundary of the character core region according to the amplification coefficient, and crop the image based on the expanded boundary to obtain the cropped image. Schematic representation: The boundary of the character core region is extended based on the following formula: min_x 2= max(0, min_x - cw), max_x 2= min(width, max_x + cw); min_y2= max(0, min_y - ch), max_x 2= min(width, max_y + ch); In the formula, min_x2 is the minimum X-axis coordinate of all points in the expanded character core region, and max_x2 is the maximum X-axis coordinate of all points in the expanded character core region. min_y2 is the minimum y-axis coordinate of all points in the expanded character core region, and max_y2 is the maximum y-axis coordinate of all points in the expanded character core region.

[0038] Crop out the character core area containing complete connected strokes according to the expanded boundary above. If the aspect ratio is unbalanced, adjust it to an approximate square by filling the edges with black to avoid subsequent scaling deformation. Through the above boundary expansion, it is possible to avoid losing the edge strokes of connected handwriting during cropping (such as the extended part of "辶" or "走之底").

[0039] S6. Perform character recognition based on the cropped image.

[0040] In a preferred embodiment, the performing character recognition based on the cropped image includes: Scale the cropped image to obtain a scaled image; Extract connected components from the scaled image, and record the total number of pixels and the minimum bounding rectangle of each connected component; calculate the pixel density corresponding to each connected component based on the total number of pixels and the minimum bounding rectangle of each connected component. Skeletonize each connected component to obtain the skeleton line corresponding to each connected component. For the skeleton line corresponding to each connected component, traverse all pixel points on the skeleton line and count the number of branches of each pixel point; Regard the connected components with the number of branches greater than the preset number of branches, pixel density greater than the preset density, and aspect ratio of the minimum bounding rectangle within the preset ratio as cross regions, and regard the remaining connected components as non-cross regions; Perform morphological opening operation with a 3×3 square structuring element on the non-cross regions, and perform 1 erosion on the cross regions using a 1×1 structuring element, and then perform dilation along the corresponding direction through a long strip structuring element according to the direction of the skeleton line to obtain a morphologically processed image. Convert the morphologically processed image to an RGB image and input it into the character recognition model to recognize the characters corresponding to the handwritten image.

[0041] Specifically, use the bicubic interpolation algorithm to scale the cropped image to 64×64 pixels to adapt to the input of the ConvNet model (i.e., the subsequent character recognition model), and at the same time translate it to the center of the image according to the character center of gravity to ensure consistent character poses. Immediately extract connected components from the scaled image, and record the total number of pixels and the minimum bounding rectangle of each connected component; calculate the ratio of the total number of pixels of each connected component to the area of the corresponding minimum bounding rectangle to obtain the pixel density corresponding to each connected component. Immediately, perform skeletonization on each connected component separately (such as the Zhang-Suen fast skeleton algorithm) to obtain the central skeleton line of the stroke, reduce contour redundancy, and highlight cross features. Next, iterate through all pixels on the skeleton line and count the number of 8-neighbor branches for each pixel; Connected regions with a number of branches greater than a preset number of branches (the preset number of branches can be 2 for illustration), a pixel density greater than a preset density (the preset density can be 0.6 for illustration, which can ensure that the inner strokes are compact and not sparse lines), and the aspect ratio of the smallest bounding rectangle is within a preset ratio (the above preset ratio can be an aspect ratio between 0.8 and 1.2 for illustration) are considered as intersecting regions; otherwise, they are considered as non-intersecting regions. For non-intersecting regions, a morphological opening operation (erosion followed by dilation) is performed using a 3×3 square structuring element to remove isolated noise (such as ink dots) with an area < 8 pixels, while preserving the connectivity of the main stroke. The 3×3 structuring element is suitable for processing fine strokes, and the 8-pixel threshold is determined by statistically analyzing the area of ​​common noise to avoid accidentally deleting stroke details. For intersecting regions, a 1×1 structuring element is first used to perform one erosion operation, removing only completely isolated 1-pixel noise without destroying the branch structure at the intersection. Then, through directional analysis of the skeleton lines (such as calculating the angle of each branch at the intersection), dilation is performed using long strip structuring elements in the corresponding direction (such as 1×3 horizontal structuring elements for horizontal branches) to fill in the tiny gap pixels in the intersecting region (such as the blank spaces at the overlapping of strokes), avoiding stroke breaks caused by the opening operation, and without expanding non-target areas, thus obtaining the final morphologically processed image. The final morphologically processed image is copied to RGB three channels and standardized to uint8 type to meet the model input format requirements. This implementation, which differs from conventional techniques such as fixed-ratio cropping, linear interpolation scaling, and large kernel erosion, ensures that the output image conforms to model standards while fully preserving the key features of handwritten strokes through dynamic adaptation and detail preservation design.

[0042] In a preferred embodiment, the text recognition model includes: an input layer, several convolutional modules, several residual modules, and a classification head; except for the initial convolutional module, all other convolutional modules are depthwise classifiable convolutional modules, and a preset number of depthwise classifiable convolutional modules are connected to the residual modules. The step of converting the morphologically processed image into an RGB image and then inputting it into a character recognition model to identify the characters corresponding to the handwritten image includes: The input layer receives the RGB image, extracts the feature map corresponding to the RGB image based on each convolution module, and generates the text recognition result based on the final feature map through the classification head.

[0043] The training of the character recognition model includes: Acquire several handwritten image samples and set a label for each handwritten image sample; The handwritten image samples are divided into several batches of training samples. The preset neural network model is iteratively trained according to each batch of training samples until the preset number of rounds or the accuracy convergence is reached, and the text recognition model is obtained. In each training session, a neural network model to be trained is acquired. If it is the first training session, the neural network model to be trained is an initialized neural network model. If it is not the first training session, the model parameters of the already stored optimal neural network model are extracted according to a preset storage path, and the neural network model to be trained is constructed. A batch of training samples is input into the neural network model to be trained so that the model can recognize the corresponding text categories; The identified text categories are compared with their corresponding labels, and the loss function is calculated based on the comparison results. The model parameters of the neural network model to be trained are adjusted according to the loss function to obtain the updated neural network model; The updated neural network model is validated using a validation set to obtain its accuracy. The accuracy of the updated neural network model is then compared with the accuracy of the stored optimal neural network model. The neural network model with the higher accuracy is selected as the updated optimal neural network model, and the model parameters of the updated optimal neural network model are stored according to the storage path.

[0044] In existing technologies, the number of parameters in current models is mostly over 100MB, resulting in problems such as long startup latency (over 3 seconds), lag, and rapid battery drain on low- and mid-range mobile devices. While some lightweight solutions reduce the number of parameters, they excessively sacrifice feature extraction capabilities, leading to decreased accuracy in recognizing rare characters and similar-structured Chinese characters, making it difficult to balance "performance" and "accuracy." Furthermore, the current training process relies on manual monitoring: after interruptions such as power outages or equipment failures, retraining from scratch is required (training large-scale datasets can take several days per session), and there is a lack of automated optimal model tracking mechanisms, requiring manual selection of results, which can easily lead to suboptimal model deployment due to subjective bias. At the same time, fixed learning rate designs are prone to slow convergence or overfitting in the later stages of training, further increasing iteration costs.

[0045] To solve the above problems, in this embodiment of the present invention, a lightweight ConvNet model architecture is adopted: to solve the contradiction problem of "high-precision - lightweight". Specifically, in the text recognition model, except for the initial convolution module using the traditional 3×3 convolution, the remaining convolution modules use the conv_dw module (depthwise separable convolution module). In the depth convolution stage, convolution is grouped according to the number of input channels, and only single-channel features are extracted; in the pointwise convolution stage, multi-channel features are fused through 1×1 convolution, and the parameter quantity is reduced by 73.2% compared with the traditional convolution; in addition, feature residual stacking is added in the middle layers of the network (conv9, conv11, conv14), and the output features of the previous layer are directly stacked with the output features of the current layer to solve the gradient disappearance problem of the deep network (16-layer convolution), and the recognition accuracy of rare words (such as "龘" " ") is increased to 95.7%; in addition, the classification head adopts a linear structure of "512×4×4→1024→num_classes", and the overall parameter quantity of the model is controlled within 48.6MB, and the volume is reduced by 18.5% compared with the existing lightweight models (such as MobileNetV1).

[0046] More specifically, the overall model can be divided into: an input layer, an initial convolution module (conv1), a first feature processing module, a second feature processing module, a third feature processing module, and a classification head; among them, the first feature processing module includes: 8 depthwise separable convolution modules conv2-conv8 without feature residual stacking, and one depthwise separable convolution module conv9 with feature residual stacking; the second feature processing module includes: one depthwise separable convolution module conv10 without feature residual stacking and one depthwise separable convolution module conv11 with feature residual stacking; the third feature processing module includes: 4 depthwise separable convolution modules conv12, conv13, conv15, conv16 without feature residual stacking and one depthwise separable convolution module conv14 with feature residual stacking; the classification head includes: one pooling layer and two fully connected layers; When an RGB image is input into the text recognition model, it is first collected by the input layer and then transmitted to conv1. Conv1 performs traditional 3×3 convolution (input 3 channels → output 32 channels, stride = 1, padding = 1) to extract basic features and obtain a 32×64×64 feature map.

[0047] Next, the 8-layer depthwise separable convolutional module from conv2 to conv8 (the depthwise separable convolutional module consists of depthwise convolution (grouped by input channel) + BN + ReLU + 1×1 pointwise convolution) processes the input 32×64×64 feature map, doubling the number of channels from 32 to 64, and reducing the feature map size to 32×32 due to stride=2; then, conv9 expands the channels to 128 and reduces the size to 16×16. At the same time, based on the residual module, the feature map output by conv8 is introduced and superimposed with the current layer result, and then activated by ReLU to obtain a 128×16×16 feature map.

[0048] conv10 first refines the 128×16×16 feature map (keeping 128 channels and stride=1), outputting the refined 128×16×16 feature map; then conv11 expands the channels to 256 and shrinks the size to 8×8, and at the same time, based on the residual module, the feature map output by conv10 (after adaptation) is introduced and superimposed with the current layer result, and after ReLU activation, a 256×8×8 feature map is obtained; conv12-conv13 first refine the 256×8×8 feature map (keeping 256 channels and stride=1); then conv14 expands the channels to 512 and shrinks the size to 4×4. At the same time, based on the residual module, the feature map output by conv13 is introduced and superimposed with the current layer result. After ReLU activation, a 512×4×4 feature map is obtained; then conv15-conv16 enhance the deep features (keeping 512 channels and stride=1). After enhancement, the features are processed by a pooling layer, using global average pooling to compress the 512×4×4 feature map into a 512×1×1 feature vector. This vector is then fed into the first fully connected layer, where a "512→1024" fully connected transformation is performed to prevent overfitting, outputting a 1024-dimensional vector. Finally, the vector passes through a second fully connected layer ("1024→num_classes", where num_classes is the number of commonly used Chinese characters and symbols), and a character probability distribution is generated using Softmax. The final output is the Top-100 characters and their corresponding confidence scores, with the highest confidence score being taken as the recognition result.

[0049] The specific training of the model is as follows: I. Training Data and Label Settings: 1. Training Samples: Preprocessed 64×64×3 RGB handwritten character images are used as training samples (covering common Chinese characters, symbols, etc.). The preprocessing includes the specific processing flow of the present invention, including inverse binarization, character core region recognition, amplification coefficient calculation, boundary expansion, cropping, scaling, and morphological processing. The images are grouped according to batch_size (e.g., 64 / 128) to form a tensor of shape [batch_size, 3, 64, 64], which is used as the model input.

[0050] 2. Label format: Uses category index + one-hot encoding. Assuming the total number of categories is num_classes, each character corresponds to a unique index, and the label is represented as a "one-hot vector of length num_classes" (the target category bit is 1, and the rest are 0).

[0051] II. Loss Function 2. Loss function: Cross-entropy loss is used. It combines softmax activation and negative log-likelihood loss, directly using the model's output logits and class index labels as input to calculate the difference between the prediction and the true distribution.

[0052] III. Model Training: The training process adopts the "breakpoint continuation training - automatic optimization - dynamic parameter tuning" approach. Initialization phase: Initial training: Initialize the lightweight ConvNet model, optimizer (e.g., Adam, initial learning rate 0.001), learning rate scheduler (StepLR, step_size=10, gamma=0.5), set the current epoch=0, and the best_valid_acc=0.

[0053] Breakpoint recovery: The system scans the preset file directory, parses the latest model weight file and the latest complete training status file (epoch, model parameters, optimizer parameters, training accuracy history, etc.), loads the model / optimizer parameters, and continues training from that epoch.

[0054] 2. Single-round training cycle (executed once per epoch) Forward propagation: Input the training batch of images into the model to obtain the predicted output logits of [batch_size, num_classes].

[0055] Loss calculation: The loss between logits and the true class index is calculated using cross-entropy loss.

[0056] Backpropagation and parameter update: Loss is backpropagated, and the optimizer updates the model parameters.

[0057] Validation set evaluation: Calculate the updated model's accuracy, valid_acc, on the validation set. If valid_acc > best_valid_acc, then update the current valid_acc to best_valid_acc, and use the current model as the latest optimal neural network model, best_model.pth. Record the model's epoch, loss, learning rate, and model parameters, and generate the latest complete training state file and the latest model weight file.

[0058] In addition, during training, StepLR decays the learning rate to 0.5 times every 10 rounds and updates the optimizer learning rate to alleviate the problem of slow convergence in the later stages. It also adopts a multi-level backup method, which automatically switches to the backup path if the main path fails to save, so as to avoid the loss of training results.

[0059] Training terminates when the preset number of rounds (e.g., 100 rounds) is reached or the accuracy on the validation set no longer improves. The best_model.pth model is then selected as the optimal model for subsequent handwritten character recognition inference.

[0060] In this embodiment, after each training round, the test set accuracy is calculated and compared with the historical best accuracy. The best accuracy is automatically updated, and the complete training state file and model weight file of the best model are saved to avoid the bias of manual selection. During each training round, the complete training state file and model weight file of the best model are read according to the storage path. In this way, even after the training terminal is closed, the system only needs to automatically scan the checkpoints directory, parse the epoch value of the latest state file, load the corresponding parameters and continue training from the breakpoint, without having to start from the beginning.

[0061] After the model training is completed, during the model inference phase, the image to be recognized is input into the text recognition model. The text recognition model converts the image into a float32 tensor and adjusts the dimensions to [1, 3, 64, 64]. If GPU is supported, the tensor is transferred to GPU memory; otherwise, the CPU is used. The model weights are loaded, and inference is performed to obtain the output tensor. The output tensor is converted into a NumPy array, and the first 100 indices are taken by sorting the probability values ​​in descending order. The character dictionary is read and non-Chinese characters are filtered out to generate a character + confidence list. The character with the highest confidence is taken as the final text recognition result.

[0062] The results are encapsulated into a JSON response and returned to the front-end device via the HTTP protocol; the recognition time is calculated and recorded in the log, and the process ends.

[0063] like Figure 2As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; An embodiment of the present invention provides a character recognition device for handwritten images, comprising: The system includes a grayscale image conversion module, a reverse binarization module, a core region recognition module, an amplification coefficient calculation module, a boundary expansion module, and a recognition module. The grayscale image conversion module is used to acquire a handwritten image and convert the handwritten image into a single-channel grayscale image; wherein, the single-channel grayscale image is an image with black text on a white background; The reverse binarization module is used to perform reverse binarization processing on the single-channel grayscale image to generate a reverse binarized image with black background and white text. The core region recognition module is used to recognize all contours in the reverse binarized image and take the contour with the largest area as the character core region. The amplification coefficient calculation module is used to calculate the amplification coefficient based on the width and height of the inverse binarized image; The boundary expansion module is used to expand the boundary of the character core region according to the amplification coefficient, and to crop the image based on the expanded boundary. The recognition module is used to perform text recognition based on the cropped image. It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the text recognition method for handwritten images provided by any of the above-described method embodiments of the present invention.

[0064] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0065] Based on the above embodiments of the handwritten image text recognition method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the handwritten image text recognition method of any embodiment of the present invention.

[0066] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0067] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0068] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0069] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the handwritten image text recognition method described in any of the above-described method embodiments of the present invention.

[0070] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0071] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for recognizing characters in handwritten images, characterized in that, include: Acquire a handwritten image and convert the handwritten image into a single-channel grayscale image; wherein the single-channel grayscale image is an image with black text on a white background; The single-channel grayscale image is subjected to inverse binarization to generate an inverse binarized image with black background and white text. Identify all contours in the inverse binarized image and use the contour with the largest area as the character core region; Calculate the amplification coefficients based on the width and height of the inverse binarized image; The character core region is expanded according to the amplification coefficient, and cropped based on the expanded boundary to obtain the cropped image. Text recognition based on cropped images.

2. The method for recognizing characters from handwritten images as described in claim 1, characterized in that, The single-channel grayscale image is subjected to inverse binarization processing to generate an inverse binarized image with black background and white text, including: The single-channel grayscale image is divided into several image blocks. For each image block, the mean and standard deviation of grayscale values ​​of all pixels in the image block are calculated. The dynamic segmentation threshold of the image block is calculated based on the mean and standard deviation of grayscale values. Linear interpolation is performed on the dynamic segmentation thresholds of all image patches to generate a dynamic continuous threshold matrix covering the entire image; For each pixel, if the gray value of the pixel is less than or equal to the dynamic segmentation threshold at the corresponding position in the dynamic continuous threshold matrix, then the gray value of the pixel is set to 255; otherwise, the gray value of the pixel is set to 0, generating a reverse binarized image with black background and white text.

3. The method for recognizing characters from handwritten images as described in claim 1, characterized in that, The text recognition based on the cropped image includes: Scale the cropped image to obtain the scaled image; Connected components are extracted from the scaled image, and the total number of pixels and minimum bounding rectangle of each connected component are recorded; the pixel density of each connected component is calculated based on the total number of pixels and minimum bounding rectangle of each connected component. Skeletonize each connected component to obtain the skeleton line corresponding to each connected component; For each connected component, iterate through all pixels on the skeleton line and count the number of branches for each pixel. Connected regions with a number of branches greater than the preset number of branches, a pixel density greater than the preset density, and an aspect ratio of the minimum bounding rectangle within the preset ratio are considered as intersecting regions, while the remaining connected regions are considered as non-intersecting regions. For non-intersecting regions, morphological opening operations are performed using 3×3 square structuring elements. For intersecting regions, erosion is performed once using 1×1 structuring elements. Then, dilation is performed along the corresponding direction of the skeleton lines using long strip structuring elements to obtain the morphologically processed image. The morphologically processed image is converted into an RGB image and then input into a character recognition model to identify the characters corresponding to the handwritten image.

4. The method for recognizing characters from handwritten images as described in claim 3, characterized in that, The text recognition model includes: an input layer, several convolutional modules, several residual modules, and a classification head; except for the initial convolutional module, all other convolutional modules are depthwise separable convolutional modules, and there is a preset number of depthwise separable convolutional modules connected to the residual modules. The step of converting the morphologically processed image into an RGB image and then inputting it into a character recognition model to identify the characters corresponding to the handwritten image includes: The input layer receives the RGB image, extracts the feature map corresponding to the RGB image based on each convolution module, and generates the text recognition result based on the final feature map through the classification head.

5. The method for recognizing characters from handwritten images as described in claim 4, characterized in that, The training of the character recognition model includes: Acquire several handwritten image samples and set a label for each handwritten image sample; The handwritten image samples are divided into several batches of training samples. The preset neural network model is iteratively trained according to each batch of training samples until the preset number of rounds or the accuracy convergence is reached, and the text recognition model is obtained. In each training session, a neural network model to be trained is acquired. If it is the first training session, the neural network model to be trained is an initialized neural network model. If it is not the first training session, the model parameters of the already stored optimal neural network model are extracted according to a preset storage path, and the neural network model to be trained is constructed. A batch of training samples is input into the neural network model to be trained so that the model can recognize the corresponding text categories; The identified text categories are compared with their corresponding labels, and the loss function is calculated based on the comparison results. The model parameters of the neural network model to be trained are adjusted according to the loss function to obtain the updated neural network model; The updated neural network model is validated using a validation set to obtain its accuracy. The accuracy of the updated neural network model is then compared with the accuracy of the stored optimal neural network model. The neural network model with the higher accuracy is selected as the updated optimal neural network model, and the model parameters of the updated optimal neural network model are stored according to the storage path.

6. The method for recognizing characters from handwritten images as described in claim 1, characterized in that, The acquisition of the handwritten image includes: The system receives a text recognition request for a handwritten image, determines whether the request has an independent image data field, and if so, calls the first data interface to extract the handwritten image from the image data field; otherwise, it calls the second data interface to extract the nested imeParam object and extracts the handwritten image from the nested imeParam object.

7. The method for recognizing characters from handwritten images as described in claim 6, characterized in that, Before receiving a text recognition request for a handwritten image, the process also includes: The encrypted string parameter is obtained from the user based on the token interface; the encrypted string parameter includes: application ID, username, password and timestamp; Decrypt the encrypted string parameters to extract the application ID, username, password, and timestamp; The timestamp data is verified based on the timestamp, and if the timestamp data verification passes, the username and password are entered into the database for matching. If the same username and password are matched in the database, a token is generated based on the application ID, username, and password. The token is stored and sent back to the user's corresponding front-end device, so that the front-end device can generate a text recognition request carrying the token based on the token and the user's handwritten image.

8. A character recognition device for handwritten images, characterized in that, include: The system includes a grayscale image conversion module, a reverse binarization module, a core region recognition module, an amplification coefficient calculation module, a boundary expansion module, and a recognition module. The grayscale image conversion module is used to acquire a handwritten image and convert the handwritten image into a single-channel grayscale image; wherein, the single-channel grayscale image is an image with black text on a white background; The reverse binarization module is used to perform reverse binarization processing on the single-channel grayscale image to generate a reverse binarized image with black background and white text. The core region recognition module is used to recognize all contours in the reverse binarized image and take the contour with the largest area as the character core region. The amplification coefficient calculation module is used to calculate the amplification coefficient based on the width and height of the inverse binarized image; The boundary expansion module is used to expand the boundary of the character core region according to the amplification coefficient, and to crop the image based on the expanded boundary. The recognition module is used to perform text recognition based on the cropped image.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the character recognition method for handwritten images as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the character recognition method for handwritten images as described in any one of claims 1-7.

Images