Method, device and equipment for processing handwritten character image and readable medium

Abstract

The application relates to a handwriting character image processing method and device, electronic equipment and a computer readable medium. The method comprises the following steps: inputting a handwriting character image into a text line segmentation model to generate a text line segmentation image, wherein a backbone feature extraction network in the text line segmentation model is generated through an MIM pre-training method; regarding each text line region in the text line segmentation image as an instance boundary; calculating the real value of pixels in each instance boundary through a KD-Tree method; generating a text line boundary image based on the real value of the pixels in each boundary instance; and extracting the text line region in the handwriting character image through the text line boundary image to perform character analysis processing. The handwriting character image processing method can overcome the technical difficulties in the prior art, efficiently and accurately perform text line segmentation on handwriting text, and then perform character analysis and recognition processing, thereby improving the processing efficiency and accuracy.

Description

Technical Field

[0001] This application relates to the field of computer information processing, and more specifically, to a method, apparatus, electronic device, and computer-readable medium for processing handwritten text images. Background Technology

[0002] With the advancement of information and network technologies, handwritten Chinese character recognition has enormous potential demand in various applications where Chinese characters are used for information transmission, thus becoming a hot research topic for many scholars. Currently, Chinese character recognition technology has been applied in multiple fields such as license plate recognition and check signature recognition, significantly improving work efficiency. However, due to the difficulty in recognizing handwritten Chinese characters, the recognition rate has not yet reached the level required for practical application. The reasons why handwritten Chinese character recognition is widely recognized as a challenge in pattern recognition are as follows:

[0003] (1) There are many commonly used Chinese characters. In most cases in daily life, there are more than 3,700 Chinese characters in the first-level character set, which is a huge number;

[0004] (2) The writing of handwritten characters is more arbitrary, with multiple sub-images, and each person has their own writing habits and style when writing Chinese characters by hand;

[0005] (3) There are many similar characters in Chinese with similar font structures. These similar characters have similar fonts, but there are subtle differences, which increases the difficulty of computer recognition and feature extraction.

[0006] Although handwritten text recognition has always been an important research direction in the field of computer vision, many text detection and recognition algorithms still cannot handle these problems well. Traditional deep learning-based handwritten line segmentation methods usually rely on pixel-level prediction, using powerful deep learning models to obtain better line segmentation results. However, for handwritten lines with strong prior information, relying solely on more powerful deep learning models for segmentation is not feasible.

[0007] Therefore, there is a need for a new method, apparatus, electronic device, and computer-readable medium for processing handwritten text images.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] In view of this, this application provides a method, apparatus, electronic device and computer-readable medium for processing handwritten text images, which can overcome the technical difficulties in the prior art, and can efficiently and accurately segment handwritten text lines, and then perform text analysis and recognition, thereby improving processing efficiency and accuracy.

[0010] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0011] According to one aspect of this application, a method for processing handwritten text images is proposed. The method includes: inputting the handwritten text image into a text line segmentation model to generate a text line segmentation image, wherein the backbone feature extraction network in the text line segmentation model is generated using a MIM pre-training method; using each text line region in the text line segmentation image as an instance boundary; calculating the ground truth value of pixels in each instance boundary using a KD-Tree method; generating a text line boundary image based on the ground truth value of pixels in each boundary instance; and extracting text line regions from the handwritten text image using the text line boundary image for text analysis processing.

[0012] In one exemplary embodiment of this application, the method further includes: randomly replacing characters in a handwritten text image to generate a training image; inputting the training image into a convolutional neural network model, the convolutional neural network model including: an initial backbone feature extraction network; training the convolutional neural network model using a MIM pre-training method; and extracting the backbone feature extraction network from the trained convolutional neural network model at the end of training.

[0013] In one exemplary embodiment of this application, a training image is generated by randomly replacing characters in a handwritten text image, including: performing character recognition on the handwritten text image to extract multiple character regions; randomly extracting a preset proportion of character regions from the multiple character regions; replacing pixels in the preset proportion of character regions with a mask token; and generating the training image from the handwritten text image after the replacement process.

[0014] In one exemplary embodiment of this application, training the convolutional neural network model using the MIM pre-training method includes: extracting features from the training image using sub-manifold sparse convolutional layers to generate a feature map; extracting character regions from the feature map using RoI Align; performing pixel prediction on the character regions using a decoder to generate predicted features; mapping the predicted features back to the training image; and reconstructing the image.

[0015] In one exemplary embodiment of this application, the method further includes: generating an initial text line segmentation model through the trained backbone feature extraction network; training the initial text line segmentation model using handwritten character images; and generating the text line segmentation model at the end of training.

[0016] In one exemplary embodiment of this application, calculating the true value of pixels in each instance boundary using a KD-Tree method includes: modeling each instance boundary in the text line segmentation image using a KD-Tree; modeling pixels within each instance boundary in the text line segmentation image using a KD-Tree; calculating the distance between the modeled instance boundary and the pixels within the instance boundary; and generating the true value of pixels in each instance boundary using the distance.

[0017] In one exemplary embodiment of this application, generating a text line boundary image based on the real value of a pixel in each boundary instance includes: comparing the real value of a pixel in each boundary instance with a boundary threshold to generate a generated text line boundary; performing dilation processing on the text line boundary to generate an expanded text line boundary; and generating a text line boundary image by expanding the text line boundary.

[0018] According to one aspect of this application, a processing apparatus for handwritten text images is proposed. The apparatus includes: an input module for inputting the handwritten text image into a text line segmentation model to generate a text line segmentation image, wherein the backbone feature extraction network in the text line segmentation model is generated using a MIM pre-training method; an instance module for using each text line region in the text line segmentation image as an instance boundary; a calculation module for calculating the ground truth value of pixels in each instance boundary using a KD-Tree method; a boundary module for generating a text line boundary image based on the ground truth value of pixels in each boundary instance; and an extraction module for extracting text line regions from the handwritten text image using the text line boundary image for processing.

[0019] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method as described above.

[0020] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0021] According to the handwritten text image processing method, apparatus, electronic device, and computer-readable medium of this application, a text line segmentation image is generated by inputting the handwritten text image into a text line segmentation model, wherein the backbone feature extraction network in the text line segmentation model is generated through MIM pre-training; each text line region in the text line segmentation image is used as an instance boundary; the true value of the pixel in each instance boundary is calculated using the KD-Tree method; a text line boundary image is generated based on the true value of the pixel in each boundary instance; and the text line region in the handwritten text image is extracted through the text line boundary image for text analysis processing. This method overcomes the technical difficulties in the prior art, can efficiently and accurately segment handwritten text into lines, and then perform text analysis and recognition, thereby improving processing efficiency and accuracy.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0023] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This is a flowchart illustrating a method for processing handwritten text images according to an exemplary embodiment.

[0025] Figures 2 to 7 This is a schematic diagram illustrating a method for processing handwritten text images according to an exemplary embodiment.

[0026] Figure 8 This is a flowchart illustrating a method for processing handwritten text images according to another exemplary embodiment.

[0027] Figures 9 to 12 This is a schematic diagram illustrating a method for processing handwritten text images according to another exemplary embodiment.

[0028] Figure 13 This is a block diagram illustrating a handwritten text image processing apparatus according to an exemplary embodiment.

[0029] Figure 14 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0030] Figure 15 This is a block diagram illustrating a computer-readable medium according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0032] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0036] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0037] The technical abbreviations used in this application are explained as follows:

[0038] Handwritten text images refer to Chinese character text images written with writing tools such as brushes or pens. Their characteristic is that each Chinese character is composed of strokes, and there are connections between the strokes.

[0039] Deep learning algorithms are machine learning algorithms based on neural network models. They simulate the interconnections between neurons in the human brain, extracting high-level, abstract feature representations from massive amounts of data through multi-level nonlinear transformations. Deep learning algorithms can be used for data analysis and processing in various fields such as image processing, speech processing, and natural language processing. The core of deep learning algorithms is the deep neural network, composed of multiple layers of neurons. Each neuron contains elements such as input, output, and activation function. The weights between neurons can be adjusted using the backpropagation algorithm to optimize the performance of the neural network. Deep learning algorithms have high automation and generalization capabilities, enabling them to autonomously learn and discover patterns in data, thereby achieving accurate prediction and classification in unknown data.

[0040] RoIAlign is a crucial operation in two-stage detection algorithms in deep learning. This operation connects the two stages, turning them into an end-to-end network, and also improves the performance of the entire detection method.

[0041] Ground truth (GT) values ​​are used as the training target for deep learning models. When training a deep learning model, a labeled dataset is required, consisting of the input data and corresponding output for each data sample.

[0042] MIM (Masked Image Modeling) demonstrates impressive results in self-supervised representation learning by reconstructing damaged image patches. It uses semantically rich visual tokenizers as reconstruction targets for mask prediction; specifically, it introduces vector quantization knowledge distillation to train the tokenizer, which discretizes the continuous semantic space into compact codes. Then, it pre-trains a visual Transformer by predicting the original visual tokens of the masked image patches.

[0043] A KD-Tree (k-dimensional tree) is a tree-like data structure that stores instance points in a k-dimensional space for fast retrieval. It is primarily used for searching key data in multidimensional space, and is a special case of a binary space partitioning tree.

[0044] Figure 1 This is a flowchart illustrating a method for processing handwritten text images according to an exemplary embodiment. The method 10 for processing handwritten text images includes at least steps S102 to S110.

[0045] like Figure 1 As shown, in S102, the handwritten text image is input into the text line segmentation model to generate a text line segmentation image. The backbone feature extraction network in the text line segmentation model is generated by MIM pre-training.

[0046] like Figure 2 The handwritten text image shown, after image preprocessing, is input into a text line segmentation model to obtain the following result: Figure 3 The text line segmentation image shown is an example of image preprocessing. Image preprocessing may include image resizing, image enhancement, and other processing techniques. The purpose of image preprocessing is to perform simple processing on input images of different resolutions and sizes from the user, so that they can be input into the text line segmentation model for segmentation.

[0047] The “text line segmentation model and backbone feature extraction network” will be described in detail in subsequent embodiments.

[0048] In S104, each text line region in the text line segmentation image is used as an instance boundary. For example... Figure 3 In the example image shown, each segmented text line in the text line segmentation image is a white area, while other areas are black. Based on the color differences in the image, each text line in the text line segmentation image can be extracted, and each text line can be treated as an instance. The boundary of the text line is used as the instance boundary.

[0049] In S106, the true value of the pixel in each instance boundary is calculated using a KD-Tree method. For example, the boundary of each instance in the text line segmentation image is modeled using a KD-Tree; the pixels within each instance boundary in the text line segmentation image are modeled using a KD-Tree; the distance between the modeled instance boundary and the pixels within the instance boundary is calculated; and the true value of the pixel in each instance boundary is generated using the distance.

[0050] In the embodiments of this application, the KD-Tree treats each pixel node in the instance as a binary tree of k-dimensional points. All non-leaf nodes can be viewed as acting as a hyperplane that divides the space into two half-spaces. The subtree to the left of a node represents the pixels to the left of the hyperplane, and the subtree to the right of a node represents the pixels to the right of the hyperplane.

[0051] One approach is to use the instance boundary as a hyperplane, where each pixel node is associated with the k-dimensional dimension perpendicular to the hyperplane. Therefore, if the partitioning is done along the x-axis, all pixel nodes with x-values ​​less than a specified value will appear in the left subtree, and all pixel nodes with x-values ​​greater than a specified value will appear in the right subtree.

[0052] After calculating the instance boundary and the distance between each pixel within the instance boundary, within each text line instance boundary, each pixel is divided by the maximum distance from the current instance's internal point to the current line boundary, thereby locally normalizing the pixels within that instance. The true value of each pixel within the instance boundary is generated based on the normalized value, which is the GT value.

[0053] In S108, a text line boundary image is generated based on the real values ​​of pixels in each boundary instance. For example, the real values ​​of pixels in each boundary instance are compared with a boundary threshold to generate a text line boundary; the text line boundary is dilated to generate an expanded text line boundary; and a text line boundary image is generated by expanding the text line boundary.

[0054] Setting a boundary threshold involves comparing the actual value of each pixel in each boundary instance with the boundary threshold. This can be understood as... Figure 3 In the text line image, the closer the image is to the center of the text line, the higher the GT value; the closer the pixel is to the boundary, the lower the GT value. The GT value of each pixel is compared with a set threshold, and pixels with values ​​less than the boundary threshold are removed as non-text line nodes, resulting in the image shown below. Figure 4 The text line boundaries are shown.

[0055] right Figure 4 The text lines in the image are expanded to obtain the following result: Figure 5 The image shown depicts the text line boundaries. Dilation is a process that adds pixel values ​​to the edges of an image, causing the overall pixel values ​​to expand, thus achieving the effect of image dilation.

[0056] In S110, the text line region in the handwritten text image is extracted using the text line boundary image for processing. Image processing is performed on the text line boundary image and the input image to accurately extract the text lines from the input image.

[0057] More specifically, the input image can be as follows: Figure 6 The extracted text lines from the text image shown can be as follows: Figure 7 As shown, from Figure 6 ,and Figure 7 The comparison shows that the method in this application can accurately extract text lines from the input image. Even if the text in the original input image is not written in a standardized way and some parts exceed the width of the text in the input image, the method in this application can still accurately extract the text lines.

[0058] After the text region is extracted, text recognition, text writing scoring, semantic recognition, and other processing can be performed on the text within the text region. This application is not limited to these.

[0059] According to the handwritten text image processing method of this application, the handwritten text image is input into a text line segmentation model to generate a text line segmentation image. The backbone feature extraction network in the text line segmentation model is generated by MIM pre-training. Each text line region in the text line segmentation image is used as an instance boundary. The true value of the pixel in each instance boundary is calculated using the KD-Tree method. A text line boundary image is generated based on the true value of the pixel in each boundary instance. The text line boundary image is used to extract the text line region in the handwritten text image for text analysis processing. This method can overcome the technical difficulties in the prior art, and can efficiently and accurately segment handwritten text into lines, and then perform text analysis and recognition, thereby improving processing efficiency and accuracy.

[0060] Figure 8 This is a flowchart illustrating a method for processing handwritten text images according to an exemplary embodiment. The method 80 for processing handwritten text images may further include steps S802 to S808.

[0061] like Figure 8 As shown, in S802, characters in the handwritten text image are randomly replaced to generate a training image. For example, character recognition is performed on the handwritten text image to extract multiple character regions; a preset proportion of character regions is randomly extracted from the multiple character regions; pixels in the preset proportion of character regions are replaced using a mask token; and the training image is generated from the replaced handwritten text image.

[0062] Character regions of up to 25% can be extracted and replaced with mask tokens. More specifically, the parameters of the mask token are initialized to 0. The mask token can also be a set of tensors learned through model iteration.

[0063] In one embodiment, the input image may be Figure 2 The training image generated after the replacement can be Figure 9 .like Figure 9 As shown, some areas in 9 have been replaced by mask tokens.

[0064] In S804, the training image is input into a convolutional neural network model, which includes an initial backbone feature extraction network.

[0065] In S806, the convolutional neural network model is trained using the MIM pre-training method. For example, feature extraction is performed on the training image through sub-manifold sparse convolutional layers to generate a feature map; character regions in the feature map are extracted using RoI Align; pixel prediction is performed on the character regions using a decoder to generate predicted features; the predicted features are mapped back to the training image, and image reconstruction is performed.

[0066] More specifically, in this application, the MIM pre-trained model uses sub-manifold sparse convolutional layers. After feature extraction by the model, the feature map of the last layer is obtained. RoIAlign is used to extract the masked regions in the input image, and the extracted RoI features are fed into a lightweight decoder for pixel-level prediction. The decoder learns to align the masked parts of the ROI features with the corresponding normalized pixels of the original image. Through this pre-training process, the MIM pre-trained model can learn the intrinsic feature representation of the font.

[0067] In S808, at the end of training, the backbone feature extraction network is extracted from the trained convolutional neural network model. A threshold for the number of training iterations can be set. After reaching the threshold, the model training is considered complete, and the parameters in the current backbone feature network are saved as optimized values ​​for use in the text segmentation model.

[0068] In this application, a certain character region before the mask token replacement can be as follows: Figure 10 As shown, the effect of the replaced characters is as follows: Figure 11 As shown, the image reconstruction results obtained after model training are as follows: Figure 12 As shown.

[0069] In one embodiment, the method further includes: generating an initial text line segmentation model using the trained backbone feature extraction network; training the initial text line segmentation model using handwritten text images; and generating the text line segmentation model at the end of training.

[0070] More specifically, during the initial training of the text line segmentation model, L1 loss and SSIM loss can be calculated using the model prediction graph and ground truth (GT). During the inference phase, the model prediction graph can be thresholded to obtain the final line segmentation result.

[0071] It is worth mentioning that in this application, the text line region in the handwritten text line image can also be randomly masked, instead of only masking certain characters as in the example above. This application is not limited to this.

[0072] In this application, during the pre-training process using the MIM method, ordinary convolutional layers can be used instead of submanifold sparse convolutional layers for computation, and this application is not limited to this.

[0073] In this application, L2 Loss can be used instead of L1 loss and SSIMloss during the training of the text line segmentation model. More specifically, the text line segmentation model can be the UNet model, HRNet model, Swin-transformer model, etc., but this application is not limited to these.

[0074] It should be clearly understood that this application describes how specific examples are formed and used, but the principles of this application are not limited to any details of these examples. Rather, based on the teachings of the disclosure of this application, these principles can be applied to many other embodiments.

[0075] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, it performs the functions defined by the method provided in this application. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0076] Furthermore, it should be noted that the above figures are merely illustrative representations of the processes included in the method according to exemplary embodiments of this application, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0077] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0078] Figure 13 This is a block diagram illustrating a handwritten text image processing apparatus according to an exemplary embodiment. Figure 13 As shown, the handwritten text image processing device 130 includes: an input module 1302, an instance module 1304, a calculation module 1306, a boundary module 1308, and an extraction module 1310.

[0079] The input module 1302 is used to input handwritten text images into the text line segmentation model to generate text line segmentation images. The backbone feature extraction network in the text line segmentation model is generated by MIM pre-training.

[0080] Instance module 1304 is used to use each text line region in the text line segmentation image as an instance boundary;

[0081] Calculation module 1306 is used to calculate the true value of the pixels in the boundary of each instance using the KD-Tree method;

[0082] Boundary module 1308 is used to generate text line boundary images based on the true values ​​of pixels in each boundary instance;

[0083] The extraction module 1310 is used to extract the text line region in the handwritten text image through the text line boundary image for processing.

[0084] The handwritten text image processing apparatus of this application generates a text line segmentation image by inputting the handwritten text image into a text line segmentation model, wherein the backbone feature extraction network in the text line segmentation model is generated by MIM pre-training; each text line region in the text line segmentation image is used as an instance boundary; the true value of the pixel in each instance boundary is calculated using the KD-Tree method; a text line boundary image is generated based on the true value of the pixel in each boundary instance; and the text line region in the handwritten text image is extracted from the text line boundary image for text analysis processing. This method overcomes the technical difficulties in the prior art, can efficiently and accurately segment handwritten text into text lines, and then perform text analysis and recognition, thereby improving processing efficiency and accuracy.

[0085] Figure 14 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0086] The following reference Figure 14 To describe an electronic device 1400 according to this embodiment of the present application. Figure 14 The electronic device 1400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0087] like Figure 14 As shown, the electronic device 1400 is presented in the form of a general-purpose computing device. The components of the electronic device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, a bus 1430 connecting different system components (including storage unit 1420 and processing unit 1410), a display unit 1440, etc.

[0088] The storage unit stores program code that can be executed by the processing unit 1410, causing the processing unit 1410 to perform the steps described in this specification according to various exemplary embodiments of this application. For example, the processing unit 1410 can perform actions such as... Figure 1 , Figure 8 The steps are shown in the figure.

[0089] The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 14201 and / or a cache storage unit 14202, and may further include a read-only memory unit (ROM) 14203.

[0090] The storage unit 1420 may also include a program / utility 14204 having a set (at least one) of program modules 14205, such program modules 14205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0091] Bus 1430 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0092] Electronic device 1400 can also communicate with one or more external devices 1400' (e.g., keyboard, pointing device, Bluetooth device, etc.), enabling users to communicate with devices that interact with electronic device 1400, and / or any device that allows electronic device 1400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1450. Furthermore, electronic device 1400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1460. Network adapter 1460 can communicate with other modules of electronic device 1400 via bus 1430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0093] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software, or by combining software with necessary hardware. Therefore, as... Figure 15 As shown, the technical solution according to the embodiments of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described method according to the embodiments of this application.

[0094] The software product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0095] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0096] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0097] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the following functions: inputting a handwritten text image into a text line segmentation model to generate a text line segmentation image, wherein the backbone feature extraction network in the text line segmentation model is generated using a MIM pre-training method; using each text line region in the text line segmentation image as an instance boundary; calculating the ground truth value of pixels in each instance boundary using a KD-Tree method; generating a text line boundary image based on the ground truth value of pixels in each boundary instance; and extracting text line regions from the handwritten text image using the text line boundary image for text analysis processing.

[0098] The computer-readable medium can also perform the following functions: randomly replace characters in a handwritten text image to generate a training image; input the training image into a convolutional neural network model, the convolutional neural network model including: an initial backbone feature extraction network; train the convolutional neural network model through MIM pre-training; and at the end of training, extract the backbone feature extraction network from the trained convolutional neural network model.

[0099] The computer-readable medium can also perform the following functions: generate an initial text line segmentation model through the trained backbone feature extraction network; train the initial text line segmentation model through handwritten text images; and generate the text line segmentation model at the end of training.

[0100] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0101] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0102] Exemplary embodiments of this application have been specifically shown and described above. It should be understood that this application is not limited to the detailed structures, arrangements, or implementation methods described herein; rather, this application is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for processing handwritten text images, characterized in that, include: The handwritten text image is input into the text line segmentation model to generate a text line segmentation image. The backbone feature extraction network in the text line segmentation model is generated by MIM pre-training. Each text line region in the text line segmentation image is used as the instance boundary; The true value of pixels at the boundary of each instance is calculated using the KD-Tree method, including: The boundary of each instance in the text line segmentation image is modeled using a KD-Tree; The pixels within the boundary of each instance in the text line segmentation image are modeled using a KD-Tree; Calculate the distance between the instance boundary after modeling and the pixels within the instance boundary; The distance is used to generate the true value of the pixel in the boundary of each instance; Generate a text line boundary image based on the ground truth values ​​of pixels in each boundary instance, including: The true value of the pixel in each boundary instance is compared with the boundary threshold to generate the text line boundary; The text line boundaries are expanded to generate extended text line boundaries; Generate a text line boundary image by extending the text line boundaries; The text line regions in the handwritten text image are extracted using the text line boundary image for further processing.

2. The method as described in claim 1, characterized in that, Also includes: Randomly replace characters in handwritten text images to generate training images; The training images are input into a convolutional neural network model, which includes an initial backbone feature extraction network. The convolutional neural network model is trained using the MIM pre-training method; At the end of training, the backbone feature extraction network is extracted from the trained convolutional neural network model.

3. The method as described in claim 2, characterized in that, Randomly replace characters in a handwritten text image to generate training images, including: The handwritten text image is subjected to character recognition to extract multiple character regions; A preset proportion of character regions are randomly extracted from the multiple character regions; The pixel in the character area is replaced by a mask token at a preset ratio. The training image is generated by replacing the processed handwritten text image.

4. The method as described in claim 3, characterized in that, The convolutional neural network model is trained using the MIM pre-training method, including: Feature maps are generated by extracting features from the training images using sub-manifold sparse convolutional layers. Character regions are extracted from the feature map using RoI Align; The character region is pixel-wise predicted using a decoder to generate predicted features; The predicted features are mapped back to the training images, and image reconstruction is performed.

5. The method as described in claim 2, characterized in that, Also includes: The trained backbone feature extraction network is used to generate an initial text line segmentation model. The initial text line segmentation model is trained using handwritten text images; At the end of training, the text line segmentation model is generated.

6. A device for processing handwritten text images, characterized in that, include: The input module is used to input handwritten text images into the text line segmentation model to generate text line segmentation images. The backbone feature extraction network in the text line segmentation model is generated through MIM pre-training. The instance module is used to use each text line region in the text line segmentation image as an instance boundary; The calculation module is used to calculate the true value of pixels in the boundary of each instance using the KD-Tree method, including: The boundary of each instance in the text line segmentation image is modeled using a KD-Tree; The pixels within the boundary of each instance in the text line segmentation image are modeled using a KD-Tree; Calculate the distance between the instance boundary after modeling and the pixels within the instance boundary; The distance is used to generate the true value of the pixel in the boundary of each instance; The boundary module, used to generate text line boundary images based on the ground truth values ​​of pixels in each boundary instance, includes: The true value of the pixel in each boundary instance is compared with the boundary threshold to generate the text line boundary; The text line boundaries are expanded to generate extended text line boundaries; Generate a text line boundary image by extending the text line boundaries; The extraction module is used to extract the text line region in the handwritten text image from the text line boundary image for processing.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images