A handwritten character recognition model training method and a handwritten character recognition method

Abstract

The present application relates to a handwritten text recognition model training method and a handwritten text recognition method, wherein the handwritten text recognition model training method includes constructing a handwritten text recognition model to be trained, including a first feature extraction network, an image matching network and a text recognition network; obtaining a text image sample set; wherein the text image sample includes a printed text image and a handwritten text image with the same text content; inputting the text image sample after fusion processing into the first feature extraction network to obtain a first text image feature vector; inputting the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result; training the parameters of the handwritten text recognition model according to the model training target loss function, so as to obtain a trained handwritten text recognition model. The present application can combine local and global features to improve the accuracy of handwritten text recognition.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a handwritten text recognition model training method and a handwritten text recognition method. Background Art

[0002] At present, the text recognition model mainly regards the text recognition problem as a non-segmented sequence recognition problem, and most of the existing models are based on the following framework: first, extract image features through a backbone network such as CNN; second, build a network to fuse the features; and finally use a CTC decoder or an Attention decoder for decoding. However, feature extraction based on CNN is usually used to extract local character features, and pays insufficient attention to the global features of the image, and performs poorly when extracting global features. Handwritten text mostly comes from text images. Although the background information of the image is relatively simple, the text shape transformation is relatively diverse. The handwriting style, as a special recognition information, has not yet been extracted as a global feature. In addition, handwriting style differences also exist within a single text and between different texts. For example, the same strokes within a text have similar lengths and twists, and different texts have similar spacing, connected strokes, and broken strokes. These have not yet been used as auxiliary features to improve the accuracy of text recognition. Therefore, how to overcome the above problems and improve the accuracy of handwritten text recognition is a technical problem that needs to be solved. Summary of the invention

[0003] The present application intends to provide a handwritten text recognition model training method and a handwritten text recognition method as well as corresponding devices, electronic devices, and storage media. The technical problems to be solved are achieved through the following technical solutions.

[0004] In a first aspect of this embodiment, a handwritten character recognition model training method is proposed, comprising:

[0005] Constructing a handwritten text recognition model to be trained; the handwritten text recognition model includes a first feature extraction network, an image matching network and a text recognition network;

[0006] Acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content;

[0007] Inputting the fused text image sample into the first feature extraction network to obtain a first text image feature vector;

[0008] Inputting the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result;

[0009] The parameters of the handwriting recognition model are trained according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwriting recognition model.

[0010] Further, the first feature extraction network is constructed based on a ResNet network to obtain a first text image feature vector; and / or obtaining the first text image feature vector includes:

[0011] Determining a preset step size of the ResNet network and a preset size of the first text image feature vector;

[0012] According to the preset step size, the text image sample is passed through multiple preset convolutional layers of the ResNet network so as to adjust the vertical size of the image based on the horizontal set size of the image to obtain the first text image feature vector that meets the preset size.

[0013] Furthermore, the text image samples are fused, including: according to a preset probability and a preset order, the printed text image and the handwritten text image are superimposed to obtain a text image fusion sample; wherein the preset probability includes that the printed text image and the handwritten text image are each 50%.

[0014] Further, the image matching network includes a second feature extraction network and a feature matching module; and / or, the first text image feature vector is input into the image matching network to obtain an image matching result, including:

[0015] Inputting the printed text image into the second feature extraction network to obtain a printed text supervision feature vector, and using the printed text supervision feature vector as a label;

[0016] The first text image feature vector and the printed text supervision feature vector are input into the feature matching module to obtain the image matching result; the feature matching module includes calculating the similarity between feature vectors, and the similarity includes a mean square error.

[0017] Further, the text recognition network includes a feature encoding module and a feature decoding module; and / or, the first text image feature vector is input into the text recognition network to obtain a text recognition result, including:

[0018] Inputting the first text image feature vector into the feature encoding module to obtain a second text image feature vector; the feature encoding module includes a BiLSTM network-based construction;

[0019] The second text image feature vector is input into the feature decoding module to obtain the text recognition result; the feature decoding module is constructed based on a CTC decoder.

[0020] Furthermore, the model training target loss function is determined by a first loss function corresponding to the text recognition network and a second loss function corresponding to the image matching network.

[0021] Furthermore, the first loss function is determined based on the text recognition result; and the second loss function is determined based on the image matching result.

[0022] In a second aspect of this embodiment, a handwritten text recognition method is proposed, comprising:

[0023] Based on the handwritten text recognition model training method described in the first aspect, a trained handwritten text recognition model is obtained, wherein the handwritten text recognition model includes a first feature extraction network and a text recognition network;

[0024] Obtain the handwritten text image to be recognized;

[0025] Inputting the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector;

[0026] The text image feature vector is input into the text recognition network to obtain the text recognition result.

[0027] In a third aspect of the present embodiment, a handwritten character recognition model training device is provided, comprising:

[0028] A model building module, configured to build a handwritten text recognition model to be trained; the handwritten text recognition model includes a first feature extraction network, an image matching network and a text recognition network;

[0029] A text image sample acquisition module is configured to acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content;

[0030] A first feature extraction module is configured to input the text image sample after fusion processing into the first feature extraction network to obtain a first text image feature vector;

[0031] A matching and recognition module, configured to input the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result;

[0032] The model training module is configured to train the parameters of the handwriting recognition model according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwriting recognition model.

[0033] In a fourth aspect of the present embodiment, a handwritten text recognition device is provided, comprising:

[0034] A model acquisition module, configured to obtain a trained handwritten text recognition model based on the handwritten text recognition model training method described in the first aspect, wherein the handwritten text recognition model includes a first feature extraction network and a text recognition network;

[0035] A module for acquiring a text image to be recognized, configured to acquire a handwritten text image to be recognized;

[0036] A feature extraction module is configured to input the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector;

[0037] The recognition result acquisition module is configured to input the text image feature vector into the text recognition network to obtain the text recognition result.

[0038] In a fifth aspect of the present embodiment, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect or the second aspect are implemented.

[0039] In a sixth aspect of the present embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0040] The embodiments of the present application include the following advantages:

[0041] The handwritten text recognition model training method provided in the embodiment of the present application is to construct a handwritten text recognition model to be trained, including a first feature extraction network, an image matching network and a text recognition network; obtain a text image sample set; wherein the text image sample includes a printed text image and a handwritten text image with the same text content; input the text image sample after fusion processing into the first feature extraction network to obtain a first text image feature vector; input the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result; train the parameters of the handwritten text recognition model according to the model training target loss function until the preset iteration termination condition is reached, so as to obtain a trained handwritten text recognition model. The present application adopts a multi-task model training framework including text recognition tasks and image matching tasks, and constructs a model target loss function based on the loss function of the text recognition task and the loss function of the image matching task; at the same time, the image matching task of the multi-task model training uses the global feature vector extracted from the printed text image as auxiliary supervision information, so that the recognition accuracy is effectively improved when predicting the text recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Attached Figure 1 This is one of the flowcharts of the image generation and fusion method for handwritten text recognition in an embodiment of the present application;

[0043] Attached Figure 2 This is the second flow chart of the handwriting recognition model training method according to the embodiment of the present application;

[0044] Attached Figure 3 This is the third flow chart of the handwriting recognition model training method of the embodiment of the present application;

[0045] Attached Figure 4 This is the fourth flowchart of the handwriting recognition model training method of the embodiment of the present application;

[0046] Attached Figure 5 is a flow chart of a handwritten text recognition method according to an embodiment of the present application;

[0047] Attached Figure 6 It is a schematic diagram of improving the ResNet network according to an embodiment of the present application;

[0048] Attached Figure 7 is a schematic diagram of the training process of the handwritten text recognition model of the embodiment of the present application;

[0049] Attached Figure 8 It is a schematic diagram of the structure of a handwritten character recognition model training device according to an embodiment of the present application;

[0050] Attached Fig. 9is a schematic diagram of the structure of a handwritten text recognition device according to an embodiment of the present application;

[0051] Attached Fig.10 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0053] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0054] As described in the background technology, the current text recognition model mainly regards the text recognition problem as a non-segmented sequence recognition problem, and most of the existing models are based on the following framework: first, extract image features through a backbone network such as CNN; second, build a network to fuse the features; finally, use a CTC decoder or an Attention decoder for decoding. However, CNN-based feature extraction is usually used to extract local character features, and pays insufficient attention to the global features of the image, and performs poorly when extracting global features. Most handwritten text comes from text images. Although the background information of the picture is relatively simple, the text shape transformation is relatively diverse. The handwriting style as a special recognition information has not yet been extracted as a global feature. In addition, differences in handwriting style also exist within a single text and between different texts. For example, the same strokes within the text have similar lengths and twists, and different texts have similar spacing, connected strokes, and broken strokes. These have not yet been used as auxiliary features to improve the accuracy of text recognition.

[0055] Therefore, in response to the above problems, the technical idea of ​​the present application is: to adopt a multi-task model training framework including text recognition tasks and image matching tasks, and to construct a model target loss function based on the loss function of the text recognition task and the loss function of the image matching task; at the same time, the image matching task of the multi-task model training uses printed text images to extract global feature vectors as auxiliary supervision information, so that the recognition accuracy is effectively improved when predicting text recognition results.

[0056] Embodiment 1:

[0057] As attached Figure 1FIG. 1 is a flow chart of a handwriting recognition model training method according to an embodiment of the present application. The method comprises:

[0058] S101: Constructing a handwritten text recognition model to be trained; the handwritten text recognition model includes a first feature extraction network, an image matching network and a text recognition network.

[0059] S102: Acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content.

[0060] S103: Input the text image sample that has undergone fusion processing into the first feature extraction network to obtain a first text image feature vector.

[0061] S104: Input the first text image feature vector into the image matching network and the text recognition network to obtain image matching results and text recognition results.

[0062] S105: Training the parameters of the handwritten text recognition model according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwritten text recognition model.

[0063] In some embodiments, the text image samples are fused, including: according to a preset probability and a preset order, the printed text image and the handwritten text image are superimposed to obtain a text image fusion sample; wherein the preset probability includes that the printed text image and the handwritten text image are each 50%.

[0064] In some embodiments, the first feature extraction network is constructed based on a ResNet network to obtain a first text image feature vector.

[0065] In some embodiments, as shown in the attached Figure 2 As shown, the obtaining of the first text image feature vector includes:

[0066] S211: Determine a preset step size of the ResNet network and a preset size of the first text image feature vector.

[0067] S212: According to the preset step size, the text image sample is passed through multiple preset convolutional layers of the ResNet network, so as to adjust the vertical size of the image based on the set horizontal size of the image, and obtain the first text image feature vector that meets the preset size.

[0068] In some embodiments, the image matching network includes a second feature extraction network and a feature matching module.

[0069] In some embodiments, the first text image feature vector is input into the image matching network to obtain an image matching result, as shown in the attached figure. Figure 3 As shown, including:

[0070] S311: Input the printed text image into the second feature extraction network to obtain a printed text supervision feature vector, and use the printed text supervision feature vector as a label.

[0071] S312: Input the first text image feature vector and the printed text supervision feature vector into the feature matching module to obtain the image matching result; the feature matching module includes calculating the similarity between feature vectors, and the similarity includes the mean square error.

[0072] In some embodiments, the text recognition network includes a feature encoding module and a feature decoding module.

[0073] In some embodiments, the first text image feature vector is input into the text recognition network to obtain a text recognition result, as shown in the attached figure. Figure 4 As shown, including:

[0074] S411: Input the first text image feature vector into the feature encoding module to obtain a second text image feature vector; the feature encoding module includes a BiLSTM network-based construction.

[0075] S412: Input the second text image feature vector to the feature decoding module to obtain the text recognition result; the feature decoding module includes a module constructed based on a CTC decoder.

[0076] In some embodiments, the model training objective loss function is determined by a first loss function corresponding to the text recognition network and a second loss function corresponding to the image matching network. Further, the first loss function is determined based on the text recognition result; and the second loss function is determined based on the image matching result.

[0077] In a second aspect of this embodiment, a handwritten text recognition method is proposed, as shown in the attached Figure 5 Said, including:

[0078] S501: Obtain a trained handwritten text recognition model, wherein the handwritten text recognition model includes a first feature extraction network and a text recognition network.

[0079] S502: Obtain a handwritten text image to be recognized.

[0080] S503: Input the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector.

[0081] S504: Input the text image feature vector into the text recognition network to obtain a text recognition result.

[0082] The following uses Uyghur as an example to further illustrate the scheme of the embodiment of the present invention. The handwritten text recognition model structure based on printed supervision is mainly composed of five stages, namely: image acquisition, image fusion, feature extraction, feature encoding and feature decoding. Based on feature extraction, the model training includes two task network branches: a text recognition network and an image matching network, and the two network branches participate in the training at the same time; when performing text recognition prediction, only the text recognition network is used.

[0083] In the image acquisition stage, the text image samples include printed text images and handwritten text images with the same text content. That is, when the model is trained, the input is a pair of images with the same content IMG (IMG h ,IMG p )IMG h and IMG p Represent handwritten text images and printed text images respectively. In the image fusion stage. In order to improve the generalization of the model, the parallel input images will be superimposed in different orders with a probability of 50% to obtain the text image fusion sample X=Concat(IMG h ,IMG p ).

[0084] For the text recognition network branch. First, extract the local features of the character shape. Since handwritten characters are complicated, the ResNet network can be improved to more accurately extract the features of handwritten characters. In the specific implementation method, the ResNet network can use ResNet18. When reducing the dimension of the image, the horizontal information should be retained as much as possible. The improved ResNet structure and output vector size are shown in the attached figure. Figure 6 As shown in the figure, S represents the number of steps in each horizontal and vertical movement in the CNN network, and BatchNorm con2d represents the convolutional neural network after batch normalization. At the same time, in order to adapt to the parallel input mode of images, the number of ResNet input channels is increased from 3 to 6. The text image fusion sample X is extracted through the ResNet network, the vertical resolution is gradually reduced, and the original resolution is retained in the horizontal direction, and the composite feature is finally output. As the local features of the glyphs of the text image. Secondly, extract the global features between characters, that is, Input into the BiLSTM encoder to extract the character sequence features in the image The vector is used as the global feature between characters of the text image. Finally, the feature vector is decoded. The CTC decoder can be selected for decoding to obtain the text recognition prediction result.

[0085] For the image matching network branch. Specifically, the image matching network extracts the feature vectors of the printed text image sample and the text image fusion sample respectively. and As a global feature, the mean squared error (MSE) of the two image feature vectors is used to measure the similarity of the two images. as labels for supervised model training.

[0086] When training the model, the network branches of two tasks are used simultaneously, as shown in the attached Figure 7 As shown. At the entrance of the text recognition network branch, the printed text image samples and the handwritten text image samples are simultaneously fused to obtain text image fusion samples, which are then passed through the first feature extraction network to obtain the first text image feature vector, and then the text recognition result is output based on the first text image feature vector; at the entrance of the image matching network branch, the printed text image is input, and the printed supervision feature vector obtained by the second feature extraction network is matched with the first text image feature vector to obtain the image matching result. As for the model target loss function, it is obtained by weighting the CTC loss function of the text recognition network and the MSE loss function of the image matching network, that is, .

[0087] In the text recognition prediction stage, only the text recognition network branch is used, that is, the handwritten text image to be predicted is input into the first feature extraction network for local feature extraction, then the BiLSTM is used for global feature extraction, and then the text recognition result is output through the CTC decoder.

[0088] The handwritten text recognition model training method provided in the embodiment of the present application is to construct a handwritten text recognition model to be trained, including a first feature extraction network, an image matching network and a text recognition network; obtain a text image sample set; wherein the text image sample includes a printed text image and a handwritten text image with the same text content; input the text image sample after fusion processing into the first feature extraction network to obtain a first text image feature vector; input the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result; train the parameters of the handwritten text recognition model according to the model training target loss function until the preset iteration termination condition is reached, so as to obtain a trained handwritten text recognition model. The present application adopts a multi-task model training framework including text recognition tasks and image matching tasks, and constructs a model target loss function based on the loss function of the text recognition task and the loss function of the image matching task; at the same time, the image matching task of the multi-task model training uses the global feature vector extracted from the printed text image as auxiliary supervision information, so that the recognition accuracy is effectively improved when predicting the text recognition results.

[0089] Embodiment 2:

[0090] Attached Figure 8 Schematic diagram of a handwriting recognition model training device provided in an embodiment of the present application. Figure 8 As shown, the handwritten character recognition model training device includes:

[0091] The model building module 801 is configured to build a handwritten text recognition model to be trained; the handwritten text recognition model includes a first feature extraction network, an image matching network and a text recognition network.

[0092] The text image sample acquisition module 802 is configured to acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content.

[0093] The first feature extraction module 803 is configured to input the text image sample after fusion processing into the first feature extraction network to obtain a first text image feature vector.

[0094] The matching and recognition module 804 is configured to input the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result.

[0095] The model training module 805 is configured to train the parameters of the handwriting recognition model according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwriting recognition model.

[0096] It should be understood that the handwriting recognition model training device of the embodiment of this specification can also execute the attached Figure 1 To Attachment Figure 4 And attached Figures 6 to 7 The method for executing the handwritten character recognition model training device in the embodiment of the present invention is realized by the handwritten character recognition model training device in the embodiment of the present invention. Figure 1 To Attachment Figure 4 And attached Figures 6 to 7 The functions of the example shown will not be described in detail here.

[0097] Attached Fig. 9 Schematic diagram of a handwritten text recognition device provided in an embodiment of the present application. Fig. 9 As shown, the handwritten text recognition device includes:

[0098] The model acquisition module 901 is configured to obtain a trained handwritten text recognition model based on the handwritten text recognition model training method described in the first embodiment, wherein the handwritten text recognition model includes a first feature extraction network and a text recognition network.

[0099] The to-be-recognized text image acquisition module 902 is configured to acquire the to-be-recognized handwritten text image.

[0100] The feature extraction module 903 is configured to input the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector.

[0101] The recognition result acquisition module 904 is configured to input the text image feature vector into the text recognition network to obtain the text recognition result.

[0102] It should be understood that a handwritten text recognition device according to the embodiment of the present specification can also execute the attached Figure 5 The method executed by the handwritten character recognition device in the embodiment of the present invention is realized by the handwritten character recognition device in the embodiment of the present invention. Figure 5 The functions of the example shown will not be described in detail here.

[0103] At the same time, the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0104] Embodiment three:

[0105] Attached Fig.10 1 is a schematic diagram of an electronic device 10 provided in an embodiment of the present application. Fig.10As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0106] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that the attached Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or different components.

[0107] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.

[0108] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a shared resource access control device at the logical level. The processor executes the program stored in the memory, and is specifically used to perform the following operations:

[0109] Constructing a handwritten text recognition model to be trained; the handwritten text recognition model to be trained includes a first feature extraction network, an image matching network and a text recognition network;

[0110] Acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content;

[0111] Inputting the fused text image sample into the first feature extraction network to obtain a first text image feature vector;

[0112] Inputting the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result;

[0113] The parameters of the handwritten character recognition model to be trained are trained according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwritten character recognition model.

[0114] or,

[0115] Based on the handwriting recognition model training method described in Example 1, a trained handwriting recognition model is obtained, wherein the handwriting recognition model includes a first feature extraction network and a character recognition network;

[0116] Obtain the handwritten text image to be recognized;

[0117] Inputting the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector;

[0118] The text image feature vector is input into the text recognition network to obtain the text recognition result.

[0119] The above is as attached to this manual Figure 1 To Attachment Figure 7 The method disclosed in the illustrated embodiment can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of this specification. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of this specification can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0120] Of course, in addition to software implementation, the electronic device of the embodiments of this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0121] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0123] Embodiment 4:

[0124] The embodiment of the present specification also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a portable electronic device including a plurality of application programs, enable the portable electronic device to execute an attached program. Figure 1 To Attachment Figure 7 The method disclosed in the illustrated embodiment is specifically used to perform the following method:

[0125] Constructing a handwritten text recognition model to be trained; the handwritten text recognition model to be trained includes a first feature extraction network, an image matching network and a text recognition network;

[0126] Acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content;

[0127] Inputting the fused text image sample into the first feature extraction network to obtain a first text image feature vector;

[0128] Inputting the first text image feature vector into the image matching network and the text recognition network to obtain an image matching result and a text recognition result;

[0129] The parameters of the handwritten character recognition model to be trained are trained according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain a trained handwritten character recognition model.

[0130] or,

[0131] Based on the handwriting recognition model training method described in Example 1, a trained handwriting recognition model is obtained, wherein the handwriting recognition model includes a first feature extraction network and a character recognition network;

[0132] Obtain the handwritten text image to be recognized;

[0133] Inputting the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector;

[0134] The text image feature vector is input into the text recognition network to obtain the text recognition result.

[0135] In short, the above are only preferred embodiments of this specification and are not intended to limit the protection scope of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification should be included in the protection scope of this specification.

[0136] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0137] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0138] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0139] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0140] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0141] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0142] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A handwritten character recognition model training method, characterized in that: include: Build a handwritten text recognition model to be trained; The handwritten text recognition model includes a first feature extraction network, an image matching network and a text recognition network; Acquire a text image sample set; wherein the text image samples include printed text images and handwritten text images having the same text content; Inputting the fused text image sample into the first feature extraction network to obtain a first text image feature vector; Input the first character image feature vector into the image matching network and the character recognition network respectively, obtain an image matching result corresponding to the image matching network, and obtain a character recognition result corresponding to the character recognition network; Training the parameters of the handwriting recognition model according to the model training target loss function until a preset iteration termination condition is reached, so as to obtain the trained handwriting recognition model; When training the model, two task network branches are used simultaneously. At the entrance of the text recognition network branch, the printed text image sample and the handwritten text image sample are simultaneously subjected to image fusion to obtain a text image fusion sample, which is then subjected to a first feature extraction network to obtain a first text image feature vector, and then the text recognition result is output based on the first text image feature vector; at the entrance of the image matching network branch, a printed text image is input, and the printed supervision feature vector obtained by the second feature extraction network is combined with the first text image feature vector to obtain an image matching result; The target loss function is obtained by weighting the CTC loss function of the text recognition network and the MSE loss function of the image matching network, that is: ； Among them, Loss CTC is the CTC loss function of the text recognition network, Loss MSE is the MSE loss function of the image matching network, and λ1 and λ2 are weights.

2. The handwriting recognition model training method according to claim 1, characterized in that: The first feature extraction network is constructed based on a ResNet network and is used to obtain a first text image feature vector; And / or, obtaining the first text image feature vector includes: Determining a preset step size of the ResNet network and a preset size of the first text image feature vector; According to the preset step size, the text image sample is passed through multiple preset convolutional layers of the ResNet network so as to adjust the vertical size of the image based on the horizontal set size of the image to obtain the first text image feature vector that meets the preset size.

3. The handwriting recognition model training method according to claim 2, characterized in that: The text image samples are fused, including: according to a preset probability and a preset order, the printed text image and the handwritten text image are superimposed to obtain a text image fusion sample; wherein the preset probability includes that the printed text image and the handwritten text image are each 50%.

4. The handwriting recognition model training method according to any one of claims 1 to 3, characterized in that: The image matching network includes a second feature extraction network and a feature matching module; And / or, inputting the first text image feature vector into the image matching network to obtain an image matching result, including: Inputting the printed text image into the second feature extraction network to obtain a printed text supervision feature vector, and using the printed text supervision feature vector as a label; The first text image feature vector and the printed text supervision feature vector are input into the feature matching module to obtain the image matching result; the feature matching module includes calculating the similarity between feature vectors, and the similarity includes a mean square error.

5. The handwriting recognition model training method according to claim 4, characterized in that: The text recognition network includes a feature encoding module and a feature decoding module; And / or, inputting the first character image feature vector into the character recognition network to obtain a character recognition result, including: Inputting the first character image feature vector into the feature encoding module to obtain a second character image feature vector; The feature encoding module includes being constructed based on a BiLSTM network; The second text image feature vector is input into the feature decoding module to obtain the text recognition result; the feature decoding module is constructed based on a CTC decoder.

6. The handwriting recognition model training method according to claim 5, characterized in that: The model training target loss function is determined by a first loss function corresponding to the text recognition network and a second loss function corresponding to the image matching network.

7. The handwriting recognition model training method according to claim 6, characterized in that: The first loss function is determined based on the text recognition result; the second loss function is determined based on the image matching result.

8. A handwritten character recognition method, characterized in that: include: Based on the handwriting recognition model training method according to any one of claims 1 to 7, a trained handwriting recognition model is obtained, wherein the handwriting recognition model includes a first feature extraction network and a character recognition network; Obtain the handwritten text image to be recognized; Inputting the handwritten text image to be recognized into the first feature extraction network to obtain a text image feature vector; The text image feature vector is input into the text recognition network to obtain the text recognition result.

9. An electronic device, characterized in that: include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, which, when executed by an electronic device including a plurality of application programs, cause the electronic device to perform the steps of the method according to any one of claims 1 to 8.

Images