Handwritten text recognition method and device, equipment and storage medium

Abstract

The invention discloses a handwritten text recognition method and device, equipment and a storage medium, and belongs to the field of text recognition. The method comprises the following steps: acquiring a handwritten text image; extracting handwriting style characteristics according to the handwritten text image, wherein the handwriting style characteristics are used for reflecting a writing style corresponding to the handwritten text image; calibrating a feature extraction position of the handwritten text image according to the handwritten style feature, and extracting a text feature of the handwritten text image according to the calibrated feature extraction position, the text feature being used for reflecting a text in the handwritten text image; and predicting a prediction text corresponding to the handwritten text image according to the text features. According to the method, the accuracy of extracting the text features of the handwritten text image can be improved, so that the recognition performance of handwritten texts of different writing styles is improved.

Description

Technical Field

[0001] This application relates to the field of text recognition, and in particular to a method, apparatus, device and storage medium for recognizing handwritten text. Background Technology

[0002] Currently, text recognition technology has been widely applied, for example in street view recognition, formula recognition, and seal recognition. Handwritten text recognition is one of the more challenging scenarios, mainly because different people may use different writing styles when writing the same text, which leads to a decrease in the text recognition performance of the text recognition model.

[0003] In related technologies, to improve the performance of handwritten text recognition, two types of datasets are typically used to train the text recognition model simultaneously: a generalization set and a user set. The generalization set usually consists of handwritten text images from different individuals with different writing styles, while the user set typically consists of handwritten text images from the specific user(s) whose handwritten text needs to be recognized. By training the text recognition model simultaneously using both the generalization set and the user set, the generalization of handwritten text recognition and its adaptability to the writing styles of specific users can be ensured, thereby improving the model's recognition ability.

[0004] The approach of training text recognition models using generalization sets and user sets can only improve the recognition ability of text recognition models for handwritten text of specific users. For users other than those specific users, the handwritten text recognition performance is still poor. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for recognizing handwritten text, which can improve the recognition performance of handwritten text with different writing styles. The technical solution is as follows:

[0006] According to one aspect of this application, a method for recognizing handwritten text is provided, the method comprising:

[0007] Acquire images of handwritten text;

[0008] Handwriting style features are extracted from the handwritten text image, and the handwriting style features are used to reflect the writing style corresponding to the handwritten text image;

[0009] The feature extraction position of the handwritten text image is calibrated according to the handwriting style features, and the text features of the handwritten text image are extracted according to the calibrated feature extraction position. The text features are used to reflect the text in the handwritten text image.

[0010] Based on the text features, predict the text corresponding to the handwritten text image.

[0011] According to another aspect of this application, a handwritten text recognition device is provided, the device comprising:

[0012] The acquisition module is used to acquire images of handwritten text;

[0013] The extraction module is used to extract handwriting style features from the handwritten text image, wherein the handwriting style features are used to reflect the writing style corresponding to the handwritten text image;

[0014] The extraction module is further configured to calibrate the feature extraction position of the handwritten text image according to the handwriting style features, and extract the text features of the handwritten text image according to the calibrated feature extraction position, wherein the text features are used to reflect the text in the handwritten text image;

[0015] The prediction module is used to predict the text corresponding to the handwritten text image based on the text features.

[0016] In an optional design, the feature extraction position is the position where the convolution kernel convolves the handwritten text image; the extraction module is used to predict the position offset of the convolution kernel based on the handwriting style features using a handwritten text recognition model, the position offset being used to reflect the offset of the calibrated convolution position of the convolution kernel relative to the default convolution position; the text features of the handwritten text image are obtained by convolving the handwritten text image with the convolution kernel based on the calibrated convolution position.

[0017] In an optional design, the handwritten text recognition model includes an encoder, which includes a text feature extraction network, which includes a variable convolutional layer; the extraction module is used to convolve the handwriting style features through the first convolutional kernel of the variable convolutional layer to obtain the position offset of the second convolutional kernel of the variable convolutional layer; and to convolve the handwritten text image through the second convolutional kernel of the variable convolutional layer based on the calibrated convolutional position of the second convolutional kernel to obtain the text features of the handwritten text image.

[0018] In an optional design, the extraction module is used to convolve the handwriting style features with the first convolution kernel to predict the offset of local image features of the handwritten text image under the handwriting style compared to the standard handwriting style, and to obtain the position offset of the second convolution kernel.

[0019] In an optional design, the extraction module is used to convolve the handwritten text image with the second convolution kernel of the variable convolution layer based on the default convolution position of the second convolution kernel to obtain the intermediate text features of the handwritten text image; and to convolve the handwriting style features and the intermediate text features with the first convolution kernel of the variable convolution layer to obtain the position offset of the second convolution kernel of the variable convolution layer.

[0020] In an optional design, the encoder further includes a style feature extraction network cascaded with the text feature extraction network; the extraction module is used to extract the handwritten style features of the handwritten text image through the style feature extraction network.

[0021] In an optional design, the encoder further includes a self-attention network cascaded with the text feature extraction network; the device further includes a processing module for enhancing the text features based on the positional correlation between different characters corresponding to the text features through the self-attention network.

[0022] In an optional design, the handwritten text recognition model further includes a decoder, which includes an autoregressive decoding network and is cascaded with the encoder; the prediction module is used to decode the text features based on autoregression through the autoregressive decoding network to obtain the predicted text corresponding to the handwritten text image.

[0023] In an optional design, the device further includes a training module for acquiring sample handwritten text images and corresponding real text; extracting sample handwriting style features from the sample handwritten text images using the handwritten text recognition model; extracting text features from the sample handwritten text images using the handwritten text recognition model based on the sample handwritten text style features; predicting sample text corresponding to the sample handwritten text images using the handwritten text recognition model based on the text features of the sample handwritten text images; calculating an error loss based on the error between the sample text and the real text; and training the handwritten text recognition model using the error loss.

[0024] According to another aspect of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the handwritten text recognition method as described above.

[0025] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one program is stored therein, the at least one program being loaded and executed by a processor to implement the handwritten text recognition method as described above.

[0026] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the handwritten text recognition method provided in various alternative implementations of the above aspects.

[0027] The beneficial effects of the technical solution provided in this application include at least the following:

[0028] This paper proposes a method to extract handwriting style features from handwritten text images during the handwritten text recognition process and then calibrate the feature extraction location based on these features. This achieves feature extraction guided by handwriting style features from handwritten text images. This allows the feature extraction process to adapt to the handwriting style features of the handwritten text images, enabling the extraction of more consistent text features for the same handwritten text with different writing styles. It avoids the influence of different writing styles on the extracted text features of the same handwritten text, improving the accuracy of text feature extraction and thus comprehensively enhancing the recognition performance of handwritten text with different writing styles. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of handwritten text provided in an exemplary embodiment of this application;

[0031] Figure 2 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application;

[0032] Figure 3 This is a schematic diagram of a handwritten text recognition model provided in an exemplary embodiment of this application;

[0033] Figure 4 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application;

[0034] Figure 5 This is a flowchart illustrating a training method for a handwritten text recognition model provided in an exemplary embodiment of this application;

[0035] Figure 6 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application;

[0036] Figure 7 This is a schematic diagram of a convolution process provided in an exemplary embodiment of this application;

[0037] Figure 8 This is a schematic diagram of a variable convolutional layer provided in an exemplary embodiment of this application;

[0038] Figure 9 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application;

[0039] Figure 10 This is a schematic diagram of the structure of a handwritten text recognition device provided in an exemplary embodiment of this application;

[0040] Figure 11 This is a schematic diagram of the structure of a handwritten text recognition device provided in an exemplary embodiment of this application;

[0041] Figure 12 This is a schematic diagram of the structure of a handwritten text recognition device provided in an exemplary embodiment of this application;

[0042] Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] First, let me introduce the terms used in this application:

[0046] Handwritten Text Recognition: Handwritten text recognition is a technology that uses computer vision and pattern recognition techniques to convert the text image of handwritten text into corresponding text information. This technology can be widely applied to various scenarios, such as document digitization, exam marking, street view recognition, formula recognition, seal recognition, etc. Handwritten text recognition generally involves the following steps: (1) Image acquisition: Obtain the handwritten text image of handwritten text. Specifically, it can be completed through a scanner, camera, or other image acquisition devices. (2) Text feature extraction: Extract text features from the handwritten text image, and these features can include information such as strokes, shapes, sizes, etc. (3) Feature classification: Classify the extracted text features to determine the corresponding characters of the text features.

[0047] In practical applications, since different people may adopt different writing styles when writing the same characters, this poses challenges to the performance of handwritten text recognition. Exemplarily, Figure 1 is a schematic diagram of handwritten text provided by an exemplary embodiment of this application. As Figure 1 shown, when different people write the same character "Hu", they will adopt different writing styles, which will increase the difficulty of handwritten text recognition.

[0048] Artificial Intelligence (AI): It is a theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0049] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include, for example, sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, pre-trained model technologies, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various major directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0050] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning. Pre-trained models are the latest development in deep learning, integrating the above techniques. In some embodiments, the handwritten text recognition model in this application can be considered a machine learning model.

[0051] Computer Vision (CV) technology: Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the vision field, such as Swin-transformer, ViT, V-MOE, and MAE, can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition. In some embodiments, the handwritten text recognition model in this application is used to process computer vision tasks.

[0052] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0053] Figure 2 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application. The computer system includes a terminal 210 and a server 220, which are connected via a communication network 230. In some embodiments, the terminal 210 deploys a client involving handwritten text recognition functionality. This client can be a standalone client, a small program that runs dependent on a host program, or a web page; this embodiment of the application does not limit this. In some embodiments, the aforementioned client can be an instant messaging client, a social networking client, a game client, a shopping client, a video client, a music client, a live streaming client, a short video client, a built-in client of the operating system, etc.; this embodiment of the application does not limit the type of client. The terminal 210 communicates with the server 220 through this client.

[0054] It should be noted that, Figure 2 The number of terminals 210 and servers 220 is used as an example only and is not intended to limit the structure of the computer system provided in the embodiments of this application. It is understood that server 220 may be connected to multiple terminals 210.

[0055] In some embodiments, terminal 210 is used to send a handwritten text image to server 220, the handwritten text image being an image including handwritten text. In some embodiments, the handwritten text image is acquired by a scanner, camera, or other image acquisition device.

[0056] The handwritten text recognition method provided in this application embodiment can be implemented by the terminal 210 alone, by the server 220, or by the terminal 210 and the server 220 through data interaction. This application embodiment does not limit this implementation. In this embodiment, after acquiring a handwritten text image, the terminal 210 sends the handwritten text image to the server 220. Illustratively, the example given is that the server 220 uses the handwritten text recognition model 221 to recognize the handwritten text image, thereby obtaining the predicted text corresponding to the handwritten text image. The handwritten text recognition model 221 is used for handwritten text recognition, and the handwritten text image acquired by the server 220 is used as input to the handwritten text recognition model 221. In some embodiments, the handwritten text recognition model 221 is deployed in the server 220. In some embodiments, the handwritten text recognition model 221 is not deployed on the server 220. The server 220 inputs information to the handwritten text recognition model 221 and obtains the information output by the handwritten text recognition model 221 through the application programming interface (API) corresponding to the handwritten text recognition model 221.

[0057] Optionally, after acquiring the handwritten text image, server 220 inputs the handwritten text image into handwritten text recognition model 221. Handwritten text recognition model 221 extracts handwriting style features from the handwritten text image and calibrates the feature extraction positions of the handwritten text image based on these handwriting style features, thereby extracting the text features of the handwritten text image according to the calibrated feature extraction positions. For example, Figure 2 The solid circle 222 in the image represents the default feature extraction location. Figure 2 The dashed circle 223 in the diagram represents the calibrated feature extraction position, which shifts downwards after calibration. Then, the predicted text corresponding to the handwritten text image is predicted based on the text features of the handwritten text image, thus achieving handwritten text recognition. Specifically, handwriting style features reflect the writing style corresponding to the handwritten text image, and text features reflect the text within the handwritten text image. After obtaining the predicted text corresponding to the handwritten text image, the server 220 sends the predicted text to the terminal 210, where it is displayed.

[0058] It is worth noting that the aforementioned terminals include, but are not limited to, mobile phones, tablets, portable laptops, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc., and can also be desktop computers, etc.; the aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0059] Cloud technology refers to a hosting technology that unifies hardware, applications, networks, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Based on the cloud computing business model, cloud technology encompasses network technology, information technology, integration technology, management platform technology, and application technology. It can form resource pools, providing flexible and convenient on-demand access.

[0060] In some embodiments, the server described above can also be implemented as a node in a blockchain system.

[0061] Figure 3 This is a schematic diagram of a handwritten text recognition model provided in an exemplary embodiment of this application. For example... Figure 3As shown, the handwritten text recognition model 300 includes an encoder 301 and a decoder 302, which are cascaded. The handwritten text recognition model 300 is built based on Convolutional Neural Networks (CNNs). The encoder 301 includes a style feature extraction network 3011, a text feature extraction network 3012, and a self-attention network 3013. The text feature extraction network 3012 includes variable convolutional layers. The style feature extraction network 3011 is cascaded with the text feature extraction network 3012, and the text feature extraction network 3012 is cascaded with the self-attention network 3013.

[0062] In the process of recognizing handwritten text, the computer device acquires a handwritten text image 303 and inputs it into a style feature extraction network 3011. The style feature extraction network 3011 extracts handwriting style features 304 from the handwritten text image 303, which reflect the writing style corresponding to the handwritten text image 303. Then, the first convolutional kernel of the variable convolutional layer of the text feature extraction network 3012 is used to convolve the handwriting style features 304, resulting in the position offset of the second convolutional kernel of the variable convolutional layer. The second convolutional kernel of the variable convolutional layer is then used to convolve the handwritten text image 303 based on the calibration convolutional position, thereby obtaining the text features of the handwritten text image 303. The position offset reflects the shift of the calibration convolutional position of the second convolutional kernel compared to its default convolutional position. Next, a self-attention network 3013 enhances the text features based on the positional correlation between different characters corresponding to the text features. Finally, an autoregressive decoding network 3021 decodes the text features in an autoregressive manner, thereby obtaining the predicted text 305 corresponding to the handwritten text image 303.

[0063] During the training of the handwritten text recognition model 300, the computer device acquires sample handwritten text images and the corresponding real text. Using the methods described above, the handwritten text recognition model 300 extracts sample handwriting style features from the sample handwritten text images, extracts text features from the sample handwritten text images based on the sample handwriting style features, and predicts the corresponding sample text based on the text features of the sample handwritten text images. Then, an error loss is calculated based on the error between the sample text and the real text, and the handwritten text recognition model 300 is trained using this error loss.

[0064] This paper proposes a method to guide the feature extraction process of handwritten text images by extracting handwriting style features during the handwritten text recognition process and then using these features to calibrate the convolutional kernel positions. This approach adapts the feature extraction process to the handwriting style features of the handwritten text images, enabling the extraction of more consistent text features from the same handwritten text with different writing styles. This avoids the influence of different writing styles on the extracted text features, improving the accuracy of text feature extraction and thus comprehensively enhancing the recognition performance of handwritten text with different writing styles.

[0065] Figure 4 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application. The method can be used on a computer device or a client on a computer device. Figure 4 As shown, the method includes:

[0066] Step 402: Obtain the image of the handwritten text.

[0067] A handwritten text image is an image that includes handwritten text, which is text written by hand. The handwritten text image includes one or more characters (or words). Optionally, when a handwritten text image includes multiple characters, all the characters in the handwritten text image belong to the same language, or some of the characters belong to the same language, or the different characters in the handwritten text image belong to different languages.

[0068] In some embodiments, the handwritten text image is acquired using a scanner, camera, or other image acquisition device. In other embodiments, the handwritten text image is uploaded by a user, retrieved from local storage by a computer device, or obtained from another computer device by a computer device.

[0069] In some embodiments, after acquiring a handwritten text image, the computer device will preprocess the handwritten text image, such as extracting the image region containing the handwritten text from the handwritten text image, splitting the handwritten text image according to the text lines, and performing binarization processing on the handwritten text image to highlight the handwritten text in the handwritten text image, and then identifying the handwritten text in the handwritten text image based on the preprocessing results of the handwritten text image.

[0070] Step 404: Extract handwriting style features from the handwritten text image.

[0071] The handwritten style features are used to reflect the writing style corresponding to the handwritten text image. This writing style is related to the writer who writes the handwritten text in the handwritten text image. For the same text, different writers may write handwritten texts with different styles. Optionally, the writing style is used to reflect the shape, length, curvature, direction, etc. of the strokes when writing the handwritten text. By way of example, continuing to refer to Figure 1 when different writers write the same character "Hu", they will get the character "Hu" with different writing styles.

[0072] Optionally, the computer device extracts the handwritten style features of the handwritten text image through a handwritten text recognition model, and the handwritten text recognition model is used to recognize the handwritten text in the handwritten text image. Optionally, the handwritten text recognition model is constructed based on a convolutional neural network. By convolving the handwritten text image through the handwritten text recognition model, the handwritten style features of the handwritten text image can be extracted.

[0073] Step 406: Calibrate the feature extraction position of the handwritten text image according to the handwritten style features, and extract the text features of the handwritten text image according to the calibrated feature extraction position.

[0074] The feature extraction position is the position where feature extraction is performed on the handwritten text image during the process of extracting the text features of the handwritten text image. Optionally, for the feature extraction process of the handwritten text image, there is a default feature extraction position. The computer device can predict the feature extraction position offset according to the handwritten style features. The feature extraction position offset is used to reflect the offset of the calibrated feature extraction position compared with the default feature extraction position. Thus, the calibrated feature extraction position can be determined to achieve the calibration of the feature extraction position of the handwritten text image. Feature extraction is performed on the handwritten text image based on the calibrated feature extraction position, and thus the text features of the handwritten text image can be obtained. These text features are used to reflect the text in the handwritten text image. Optionally, the text features are used to reflect the structure and shape of the text in the handwritten text image. By classifying the text features corresponding to different characters in the handwritten text image, the handwritten text corresponding to the handwritten text image can be predicted.

[0075] Since there are differences in the styles of handwritten texts with different writing styles, by calibrating the feature extraction position of the handwritten text image according to the handwritten style features, it is possible to perform feature extraction on the handwritten text image more pertinently based on the writing style of the handwritten text image, thereby improving the accuracy of extracting the text features of the handwritten text image.

[0076] In some embodiments, feature extraction of handwritten text images is achieved by convolving the handwritten text image with a convolutional kernel. The feature extraction location is the position where the convolutional kernel convolves with the handwritten text image. The computer device can predict the positional offset of the convolutional kernel based on handwriting style features, and then convolve the handwritten text image with the kernel based on the calibrated convolutional position to obtain the text features of the handwritten text image. This positional offset reflects the shift of the calibrated convolutional position of the kernel compared to the default convolutional position.

[0077] Optionally, by convolving handwriting style features, it is possible to predict the offset of local image features of a handwritten text image relative to the standard handwriting style under the handwriting style feature, thereby obtaining the position offset of the convolution kernel. Although the overall difference may be significant under different handwriting styles, the local feature changes are relatively small. For example, the stroke structure at the same location in the same text is similar under different handwriting styles. Predicting the offset of local image features of a handwritten text image, that is, predicting the offset of features reflecting the same stroke structure in the handwritten text relative to the standard handwriting style under the handwriting style feature, guides the text feature extraction process. This enables the extraction of consistent text features for handwritten texts with different handwriting styles, contributing to the accurate classification of handwritten text.

[0078] For example, continue to refer to Figure 2 , Figure 2 The solid circle 222 in the image indicates the default convolution position of the convolution kernel. Figure 2 The dashed circle 223 in the diagram represents the calibrated convolution position of the convolution kernel. As shown by the arrow above the dashed circle 223, the convolution position of the kernel shifts downward after calibration.

[0079] When using a convolution kernel to convolve a handwritten text image, the kernel is moved across the image at different convolution positions. Each time the kernel reaches a certain position, the matrix corresponding to the kernel is multiplied by the pixel values ​​of the image region corresponding to that position. By repeating this process, the convolution results at different positions are summed, thereby extracting the text features of the handwritten text image.

[0080] Optionally, the computer device uses a handwritten text recognition model to calibrate the convolution position of the convolution kernel of the handwritten text recognition model based on the handwriting style features of the handwritten text image. Then, it convolves the handwritten text image with the convolution kernel after calibrating the convolution position to obtain the text features of the handwritten text image.

[0081] Step 408: Predict the text corresponding to the handwritten text image based on text features.

[0082] Predicted text is used to predict the text contained in a handwritten text image. For example, continue to refer to... Figure 3 The handwritten text image includes an image of the handwritten "XXX Limited Liability Company". The predicted text obtained by the computer device based on the text features of the handwritten text image is "XXX Limited Liability Company".

[0083] Optionally, the computer device uses a handwritten text recognition model to predict the text corresponding to the handwritten text image based on text features. For example, the computer device uses a handwritten text recognition model to classify text features, thereby obtaining the predicted text corresponding to the handwritten text image.

[0084] In summary, the method provided in this embodiment extracts handwriting style features from handwritten text images during the handwritten text recognition process and calibrates the feature extraction positions of the handwritten text images based on these handwriting style features. This achieves feature extraction of handwritten text images guided by handwriting style features. It allows the feature extraction process to adapt to the handwriting style features of the handwritten text images, thereby enabling the extraction of more consistent text features for the same handwritten text with different writing styles. This avoids the influence of different writing styles on the extracted text features of the same handwritten text, improving the accuracy of text feature extraction from handwritten text images and thus comprehensively enhancing the recognition performance of handwritten text with different writing styles.

[0085] Figure 5 This is a flowchart illustrating a training method for a handwritten text recognition model provided in an exemplary embodiment of this application. This method can be used on a computer device or a client on a computer device. Figure 5 As shown, the method includes:

[0086] Step 502: Obtain the sample handwritten text image and the corresponding real text.

[0087] A sample handwritten text image is an image containing sample handwritten text, which is manually written text. The sample handwritten text image includes one or more characters. In some embodiments, the sample handwritten text image is acquired using a scanner, camera, or other image acquisition device. The actual text is the text actually contained within the sample handwritten text image, which is manually set based on the sample handwritten text image.

[0088] Step 504: Extract sample handwriting style features from the sample handwritten text images using a handwritten text recognition model.

[0089] The sample handwriting style features are used to reflect the writing style corresponding to the sample handwritten text image, which is related to the writer of the sample handwritten text in the sample handwritten text image. Optionally, the handwritten text recognition model is built based on a convolutional neural network. By convolving the sample handwritten text image with the handwritten text recognition model, the sample handwriting style features of the sample handwritten text image can be extracted.

[0090] Step 506: Extract text features from the sample handwritten text image based on the handwritten style features of the sample using the handwritten text recognition model.

[0091] Computer equipment uses a handwritten text recognition model to calibrate the feature extraction positions of sample handwritten text images based on the handwriting style characteristics, thereby extracting text features from the sample handwritten text images according to the calibrated feature extraction positions. The feature extraction position is the location where feature extraction is performed on the sample handwritten text image during the process of extracting text features. The text features of the sample handwritten text image are used to reflect the text within the sample handwritten text image.

[0092] In some embodiments, feature extraction of the sample handwritten text image is achieved by convolving the sample handwritten text image with a convolution kernel. The feature extraction position is the position where the convolution kernel convolves the sample handwritten text image. Based on the handwriting style features of the sample, the handwritten text recognition model can predict the position offset of the convolution kernel. Then, by convolving the sample handwritten text image with the convolution kernel based on the calibrated convolution position, the text features of the sample handwritten text image can be obtained. This position offset reflects the offset of the calibrated convolution position of the convolution kernel compared to the default convolution position; that is, the calibrated convolution position can be determined based on the position offset and the default convolution position.

[0093] Step 508: Predict the sample text corresponding to the sample handwritten text image based on the text features of the sample handwritten text image using the handwritten text recognition model.

[0094] The sample text is used to predict the text contained in the sample handwritten text image. Optionally, the computer device classifies the text features of the sample handwritten text image using a handwritten text recognition model to obtain the sample text corresponding to the sample handwritten text image.

[0095] Step 510: Calculate the error loss based on the error between the sample text and the real text.

[0096] Computer devices can determine the error loss of a handwritten text recognition model based on the error between sample text and real text, combined with an error loss function. In some embodiments, the error loss function is a cross-entropy loss function; however, this application does not limit the type of error loss function.

[0097] Step 512: Train the handwritten text recognition model using error loss.

[0098] After obtaining the error loss, the computer device backpropagates it through the handwritten text recognition model, thereby adjusting the model parameters to continuously reduce the error loss. The computer device repeatedly executes the above steps, and when the error loss meets the training completion condition, the computer device completes the training of the handwritten text recognition model.

[0099] It should be noted that the implementation of extracting sample handwriting style features from sample handwritten text images through the handwritten text recognition model, extracting text features from sample handwritten text images through the handwritten text recognition model, and predicting sample text corresponding to sample handwritten text images through the handwritten text recognition model can be referred to the relevant descriptions in the embodiments below. The embodiments of this application will not be further described here.

[0100] In summary, the method provided in this embodiment, by training a handwritten text recognition model, can extract handwriting style features from handwritten text images during the handwritten text recognition process, and calibrate the feature extraction positions of the handwritten text images based on these handwriting style features. This achieves feature extraction of handwritten text images guided by handwriting style features. It allows the feature extraction process to adapt to the handwriting style features of the handwritten text images, thereby enabling the extraction of more consistent text features for the same handwritten text with different writing styles. This avoids the influence of different writing styles on the extracted text features of the same handwritten text, improving the accuracy of text feature extraction from handwritten text images, and thus comprehensively improving the recognition performance of handwritten text with different writing styles.

[0101] Figure 6 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application. The method can be used on a computer device or a client on a computer device. Figure 6 As shown, the method includes:

[0102] Step 602: Obtain the image of the handwritten text.

[0103] A handwritten text image is an image that includes handwritten text, which is manually written text, and the handwritten text image includes one or more characters. In some embodiments, the handwritten text image is acquired by a scanner, camera, or other image acquisition device. In some embodiments, the handwritten text image is uploaded by a user, or obtained by a computer device from local storage, or obtained by a computer device from another computer device.

[0104] Step 604: Extract handwriting style features from the handwritten text image using a handwritten text recognition model.

[0105] Handwriting style features are used to reflect the writing style corresponding to the handwritten text image, which is related to the writer of the handwritten text in the handwritten text image.

[0106] Optionally, the handwritten text recognition model includes an encoder for encoding the handwritten text image to extract features from it. The encoder includes a style feature extraction network, through which the handwritten text recognition model extracts handwritten style features from the handwritten text image. Optionally, the style feature extraction network includes one or more convolutional layers, through which the handwritten text image is convolved to extract handwritten style features.

[0107] For example, the style feature extraction network includes five 3×3 convolutional layers, where 3×3 indicates that each convolutional layer uses a 3×3 array to convolve the input features (9 convolutions) to obtain the output of that convolutional layer. Optionally, the style feature extraction network also includes three 2×2 downsampling layers to reduce the dimensionality of the extracted features and decrease the computational cost of the model. For example, the size of the handwritten text image input to the style feature extraction network is H×W, and the handwritten style features output by the style feature extraction network are a C-dimensional vector, where C is a positive integer.

[0108] By extracting handwriting style features from handwritten text images using a style feature extraction network, the recognition of handwritten text images can be guided based on these handwriting style features, which helps improve the recognition performance of handwritten text with different writing styles.

[0109] Step 606: Based on the handwriting style features, predict the position offset of the convolution kernel of the handwritten text recognition model.

[0110] This convolutional kernel is used to convolve around the handwritten text image to extract text features, which reflect the text in the handwritten text image. The position offset is used to calibrate the convolutional kernel's position when convolving with the handwritten text image. The position offset reflects the deviation of the calibrated convolutional position of the kernel from the default convolutional position; that is, the calibrated convolutional position of the kernel can be determined based on the position offset and the default convolutional position of the kernel.

[0111] By predicting the position offset of the convolution kernel based on handwriting style features, the process of guiding the convolution of handwritten text images based on handwriting style features is realized, which helps to improve the accuracy of feature extraction of handwritten text images, thereby improving the recognition performance of handwritten text with different writing styles.

[0112] Optionally, the encoder of the handwritten text recognition model includes a text feature extraction network, which comprises variable convolutional layers, and a style feature extraction network is cascaded with the text feature extraction network. The handwritten text recognition model convolves the handwritten style features through the first convolutional kernel of the variable convolutional layer, thereby obtaining the positional offset of the second convolutional kernel of the variable convolutional layer. The first and second convolutional kernels may be the same or different; the second convolutional kernel is used to convolve the handwritten text image, thereby extracting the text features of the handwritten text image.

[0113] In some embodiments, the variable convolutional layer can also predict the position offset of the convolutional kernel based on handwriting style features in other ways. For example, the variable convolutional layer is provided with a feature mapping layer, which maps the handwriting style features into a one-dimensional vector, thereby obtaining the position offset of the convolutional kernel (second convolutional kernel) of the variable convolutional layer.

[0114] By using variable convolutional layers to convolve handwriting style features, the position offset of the convolution kernel used when performing feature extraction on handwritten text images is predicted. This enables the feature extraction process of handwritten text images to be guided by handwriting style features, which helps to improve the accuracy of feature extraction from handwritten text images and thus improves the recognition performance of handwritten text with different writing styles.

[0115] Optionally, the handwritten text recognition model convolves the handwriting style features with a first convolutional kernel to predict the offset of local image features of the handwritten text image relative to the standard handwriting style under that handwriting style feature, thereby obtaining the position offset of the second convolutional kernel. Since the same text may appear significantly different overall under different handwriting styles, its local feature changes are relatively small. By predicting the offset of local image features of the handwritten text image, and using this offset to guide the text feature extraction process, consistent text features can be extracted for handwritten text with different handwriting styles, contributing to the accurate classification of handwritten text.

[0116] The above process of predicting the position offset of the convolution kernel only refers to the handwriting style features. In some embodiments, the handwritten text recognition model can also predict the position offset of the convolution kernel based on the handwriting style features and the features of the extracted handwritten text image.

[0117] Optionally, the handwritten text recognition model convolves the handwritten text image through the second convolutional kernel of the deformable convolutional layer based on the default convolutional position of the second convolutional kernel, so as to obtain the intermediate text features of the handwritten text image. The intermediate text features are the text features of the handwritten text image extracted without calibrating the convolutional position of the second convolutional kernel. Then, the handwritten style features and the intermediate text features are convolved through the first convolutional kernel of the deformable convolutional layer, so as to obtain the position offset of the second convolutional kernel of the deformable convolutional layer. Optionally, before convolving the handwritten style features and the intermediate text features, the computer device concatenates the handwritten style features and the intermediate text features.

[0118] The handwritten style features and the intermediate text features are convolved through the deformable convolutional layer, so as to jointly predict the position offset of the convolutional kernel used during feature extraction based on the handwritten style features and the intermediate text features. The predicted position offset not only references the writing style of the handwritten text, but also references the text features of the handwritten text itself, which helps to improve the accuracy of predicting the position offset. Guiding the feature extraction process of the handwritten text image through the position offset helps to improve the accuracy of extracting the features of the handwritten text image, thereby improving the recognition performance of handwritten text with different writing styles.

[0119] Step 608: Convolve the handwritten text image through the convolutional kernel of the handwritten text recognition model based on the calibrated convolutional position to obtain the text features of the handwritten text image.

[0120] The text features are used to reflect the text in the handwritten text image, and the calibrated convolutional position is determined according to the position offset and the default convolutional position. Optionally, the encoder of the handwritten text recognition model includes a text feature extraction network, and the text feature extraction network includes a deformable convolutional layer. After predicting the position offset of the second convolutional kernel, the handwritten text recognition model convolves the handwritten text image through the second convolutional kernel of the deformable convolutional layer based on the calibrated convolutional position of the second convolutional kernel, so as to obtain the text features of the handwritten text image.

[0121] Exemplarily, the size of the handwritten text image input to the text feature extraction network is H×W, and the size of the text features (feature map) of the handwritten text image extracted by the text feature extraction network is H / 8×W / 8.

[0122] Exemplarily, Figure 7 is a schematic diagram of the convolutional process provided by an exemplary embodiment of the present application. As Figure 7 shown in (a) of Figure 7The circles in [description] represent the position offsets of the convolutional kernels and their corresponding convolutional positions, so as to obtain the calibrated convolutional positions 703 of the convolutional kernels. This position offset is used to reflect the offset of the calibrated convolutional position 703 of the convolutional kernel compared to the default convolutional position 702. As Figure 7 As shown in (b) of [description], the handwritten text image 704 includes the character "Hu" in the second writing style. By extracting the handwritten style features of the handwritten text image 704, the computer device can predict the position offset of the convolutional kernel, so as to obtain the calibrated convolutional position 706 of the convolutional kernel. This position offset is used to reflect the offset of the calibrated convolutional position 706 of the convolutional kernel compared to the default convolutional position 705.

[0123] According to Figure 7 It can be seen that for the default convolutional position 702 of the convolutional kernel corresponding to the handwritten text image 701 and the default convolutional position 705 of the convolutional kernel corresponding to the handwritten text image 704, these two convolutional positions are the same, both arranged in a 3×3 array, and the sizes of the convolutional kernels are the same. For the calibrated convolutional position 703 of the convolutional kernel corresponding to the handwritten text image 701 compared to the default convolutional position 702, there is mainly an up-and-down contraction offset, and the size of the convolutional kernel remains unchanged. For the calibrated convolutional position 706 of the convolutional kernel corresponding to the handwritten text image 704 compared to the default convolutional position 705, there is mainly a left-and-right contraction offset, and the size of the convolutional kernel remains unchanged. The above position offsets of the convolutional kernels are determined by the different writing styles of the character "Hu". It can be seen that after calibrating the convolutional positions of the convolutional kernels according to the position offsets, the convolutional positions of the convolutional kernels can be better adapted to the handwritten text with different writing styles. For example Figure 7 As shown in [description], the convolutional kernel is located within the image area corresponding to the handwritten text, so as to improve the accuracy of extracting the text features of the handwritten text image.

[0124] Exemplarily, Figure 8 is a schematic diagram of a variable convolutional layer provided by an exemplary embodiment of the present application. As Figure 8 shown, the variable convolutional layer 800 includes a first convolutional kernel 801 and a second convolutional kernel 802. During the process of extracting the text features of the handwritten text image, the computer device inputs the handwritten style feature 803 and the handwritten text image 805 into the variable convolutional layer 800. The variable convolutional layer 800 convolves the handwritten style feature 803 through the first convolutional kernel 801, so as to obtain the position offset 804 of the second convolutional kernel 802. The position offset 804 indicates that 4 convolutional positions are offset, and the remaining 5 convolutional positions remain default unchanged. Then, the variable convolutional layer 800 convolves the handwritten text image 805 through the second convolutional kernel 802 according to the calibrated convolutional positions calibrated based on the position offset 804, so as to obtain the text features 806 of the handwritten text image.

[0125] Optionally, the encoder of the handwritten text recognition model also includes a self-attention network, which is cascaded with the text feature extraction network. After obtaining the text features of the handwritten text image, the handwritten text recognition model enhances the text features through the self-attention network based on the positional correlation between different characters corresponding to the text features; that is, it enhances the text features through context (which can be understood as semantics). For example, the self-attention network can establish the correlation between characters at different positions corresponding to the text features; the closer the positions, the greater the correlation, and the farther the positions, the smaller the correlation, thereby enhancing the text features.

[0126] For example, the size of the text features input to the self-attention network is H / 8×W / 8, and the size of the text features output after enhancement by the self-attention network is H / 8×W / 8.

[0127] By using self-attention networks to enhance text features based on the positional relationships between characters, it is possible to improve the accuracy of text recognition by referencing the features of other characters when predicting characters in handwritten text based on text features.

[0128] Step 610: Predict the text corresponding to the handwritten text image based on the text features using the handwritten text recognition model.

[0129] The predicted text is used to predict the text contained in a handwritten text image. Optionally, a computer device classifies the text features using a handwritten text recognition model to obtain the predicted text corresponding to the handwritten text image.

[0130] In some embodiments, the handwritten text recognition model further includes a decoder, which is cascaded with the encoder. The decoder decodes the features of the handwritten text image extracted by the encoder to predict the predicted text corresponding to the handwritten text image. The decoder includes an autoregressive decoding network. After obtaining the text features of the handwritten text image, the handwritten text recognition model decodes the text features in an autoregressive manner through the autoregressive decoding network to obtain the predicted text corresponding to the handwritten text image.

[0131] For example, in the process of decoding text features using an autoregressive decoding network, the computer device first inputs a start symbol (customizable) to the network to indicate the start of decoding. After inputting the start symbol, the network decodes the features of the first character (e.g., the leftmost character in a line) corresponding to the text features, obtaining the decoding result of the first character, i.e., the predicted first character. During the decoding of the features of the i-th character, the network inputs the decoding result of the (i-1)-th character to predict the decoding result of the i-th character, where i is a positive integer greater than 1. By repeatedly executing these steps, the network obtains the decoding results of the features of all characters corresponding to the text features, thus obtaining the predicted text corresponding to the handwritten text image.

[0132] By using an autoregressive decoding network to predict the text corresponding to a handwritten text image, the prediction result of the previous text can be referenced when predicting a certain character in the handwritten text based on the text features, thereby improving the accuracy of text recognition.

[0133] In summary, the method provided in this embodiment extracts handwriting style features from handwritten text images during the handwritten text recognition process and calibrates the convolution position of the convolution kernel used when extracting text features based on these handwriting style features. This achieves feature extraction of handwritten text images guided by handwriting style features. It allows the feature extraction process to adapt to the handwriting style features of the handwritten text images, thereby enabling the extraction of more consistent text features for the same handwritten text with different writing styles. This avoids the influence of different writing styles on the extracted text features of the same handwritten text, improving the accuracy of text feature extraction and thus comprehensively enhancing the recognition performance of handwritten text with different writing styles.

[0134] This application utilizes a handwritten text recognition model to recognize handwritten text with different writing styles. For example, continue to refer to... Figure 3 The handwritten text recognition model consists of two parts: an encoder 301 and a decoder 302. The encoder 301 incorporates variable convolutional layers, while the decoder 302 is a text classifier based on an autoregressive mechanism. The encoder 301 contains three networks: a style feature extraction network 3011, a text feature extraction network 3012 based on variable convolution, and a self-attention network 3013 that enhances features based on context.

[0135] The encoder 301 receives a handwritten text image of size H×W as input. The handwritten text image first enters the style feature extraction network 3011 to extract handwritten style features. The style feature extraction network 3011 has a network structure of 5 layers of 3×3 convolutional layers and 3 layers of 2×2 downsampling layers; this network is a lightweight image feature extraction network. After obtaining the handwritten style features, both the handwritten style features and the handwritten text image are simultaneously input into the text feature extraction network 3012. The handwritten style features are used to guide the deformation of the convolutional kernels (convolutional position shift) in the text feature extraction network 3012, while the handwritten text image is used to extract the corresponding text features. The handwritten text image input to the text feature extraction network 3012 has a size of H×W, the handwritten style features are C-dimensional vectors, and the text features output by the text feature extraction network 3012 have a size of H / 8×W / 8. The self-attention network 3013 takes text features of size H / 8 × W / 8 as input and outputs enhanced text features of size H / 8 × W / 8. The structure of the self-attention network 3013 is based on a self-attention mechanism. The autoregressive decoding network 3021 of the decoder 302 takes text features of size H / 8 × W / 8 as input and iteratively inputs historical decoding results to achieve the recognition of the entire handwritten text. The details of the handwritten text recognition model are described below.

[0136] (1) Style feature extraction network 3011: 5-layer convolutional neural network, the input is a handwritten text image with size H×W, and the output is C-dimensional handwritten style features.

[0137] (2) Text Feature Extraction Network 3012: The text feature extraction network 3012 can be implemented using a convolutional neural network (e.g., Resnet-50, Resnet-50-DC5, Resnet-101, etc.). The input is a handwritten text image and handwriting style features, and the output is the text features (convolutional feature map) of the handwritten text image, with a size of H / 8×W / 8.

[0138] (3) Variable Convolutional Layer: The convolutional layers in the text feature extraction network 3012 are implemented as variable convolutional layers. The input of the variable convolutional layer is the handwritten text image and handwriting style features. After the handwriting style features are input into the variable convolutional layer, the output is the position offset of each convolutional kernel used to extract text features. When extracting text features, the convolution position of the convolutional kernel is adjusted by the position offset.

[0139] (4) Self-attention network 3013: The input is text features, and the output is text features enhanced by context (encoded features). The input size is H / 8×W / 8, and the output size is H / 8×W / 8. Self-attention network 3013 can enhance features using semantic information for some handwritten texts with large deformation or unclear text.

[0140] (5) Autoregressive decoding network 3021: The input is text features and historical decoding information (decoding result of the previous character). The first frame of the historical decoding information is the start character. The autoregressive decoding network 3021 focuses on the position of the feature of the first character in the text features based on the start character, extracts the content and classifies it. The classification result is the category of the first character in the text features. This process continues until all characters in the text features have been decoded.

[0141] (6) Loss Function: During the training of the handwritten text recognition model, a teacher-forcing method is used to input the true classification of the previous character during the decoding of the autoregressive decoding network 3021, thereby aligning the classification features with the classification categories. The classification loss is calculated using cross-entropy loss.

[0142] The method provided in this application calibrates the convolution position of the convolution kernel based on the handwriting style of the handwritten text image. A style feature extraction network extracts the handwriting style of the handwritten text image, and a text feature extraction network adjusts the offset of the convolution kernel using these handwriting style features, thereby completing the extraction of text features and solving the problem of difficulty in effectively extracting text features due to the diversity of handwritten styles. Furthermore, contextual feature enhancement via a self-attention network further weakens the interference caused by noise. This application addresses the problem of poor robustness in extracting text features from handwritten text images with different handwriting styles, achieving improved robustness without increasing the number of feature channels in the model. By adjusting the convolution kernel form according to handwriting style features, the increased recognition difficulty caused by distorted handwriting is alleviated, improving the recognition performance of handwritten text.

[0143] In one possible example, the method provided in this application embodiment is applied to a scenario where handwritten text recognition is performed by an input method client in handwritten input mode. Figure 9 This is a flowchart illustrating a handwritten text recognition method provided in an exemplary embodiment of this application. This method can be used in input method clients. Figure 9 As shown, the method includes:

[0144] Step 902: In handwriting input mode, acquire the handwritten text image of the user's handwritten text.

[0145] When the input method client is in handwriting input mode, a writing area will be displayed. This writing area is for the user to write handwritten text. After the user writes handwritten text in the writing area, the input method client will capture the content written in the writing area, thus obtaining an image of the user's handwritten text.

[0146] Step 904: Extract handwriting style features from the handwritten text image.

[0147] Handwriting style features are used to reflect the writing style corresponding to the handwritten text image, which is related to the user of the input method client. Optionally, the input method client extracts the handwriting style features of the handwritten text image through a handwritten text recognition model, which is used to recognize the handwritten text in the handwritten text image. The input method client extracts the handwriting style features of the handwritten text image by convolving the handwritten text image through the handwritten text recognition model.

[0148] Step 906: Calibrate the feature extraction position of the handwritten text image according to the handwriting style features, and extract the text features of the handwritten text image according to the calibrated feature extraction position.

[0149] The feature extraction location is the position where feature extraction is performed on the handwritten text image during the process of extracting text features. Optionally, for the feature extraction process of the handwritten text image, there is a corresponding default feature extraction location. The input method client can predict the feature extraction location offset based on the handwriting style characteristics. The feature extraction location offset is used to reflect the offset of the calibration feature extraction location compared to the default feature extraction location, thereby determining the calibration feature extraction location and realizing the calibration of the feature extraction location of the handwritten text image. Based on the calibration feature extraction location, feature extraction is performed on the handwritten text image to obtain the text features of the handwritten text image, which are used to reflect the text in the handwritten text image.

[0150] In some embodiments, feature extraction of handwritten text images is achieved by convolving the handwritten text image with a convolution kernel, and the feature extraction position is the position where the convolution kernel convolves with the handwritten text image. Optionally, the input method client uses the handwritten text recognition model to calibrate the convolution position of the convolution kernel of the handwritten text recognition model based on the handwriting style features of the handwritten text image, and then convolves the handwritten text image with the calibrated convolution kernel to obtain the text features of the handwritten text image.

[0151] Step 908: Predict the text corresponding to the handwritten text image based on text features.

[0152] The predicted text is used to predict the text contained in a handwritten text image, that is, the text written by the user in the writing area of ​​the input method client. Optionally, the input method client predicts the predicted text corresponding to the handwritten text image based on text features using a handwritten text recognition model. For example, the input method client classifies text features using a handwritten text recognition model to obtain the predicted text corresponding to the handwritten text image.

[0153] Step 910: Display the predicted text among the candidate input texts.

[0154] Candidate input text is used by the user to select the text to input. After obtaining the predicted text, the input method client displays candidate input text in the input candidate box, which includes the predicted text. In response to the selection of the displayed predicted text, the input method client displays the selected predicted text in the input control, such as a text input box, thereby completing the text input.

[0155] In summary, the method provided in this embodiment extracts handwriting style features from the handwritten text image during the recognition of handwritten text in the input method, and calibrates the feature extraction position of the handwritten text image based on the handwriting style features. This achieves feature extraction of the handwritten text image guided by its handwriting style features. This allows the feature extraction process to adapt to the handwriting style features of the handwritten text image, thereby enabling the extraction of more consistent text features for the same handwritten text with different writing styles. It avoids the influence of different writing styles on the extracted text features of the same handwritten text, improving the accuracy of text feature extraction from the handwritten text image. This comprehensively improves the recognition performance of handwritten text with different writing styles, helping to improve the efficiency of users inputting text using the input method in handwriting input mode.

[0156] It should be noted that this application may display a prompt interface, pop-up window, or output voice prompts before and during the collection of user-related data (such as handwritten text images in this application). These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their relevant data is being collected. This ensures that the application only begins executing the steps related to collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps related to collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of relevant user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0157] It should be noted that the order of the method steps provided in the embodiments of this application can be appropriately adjusted, and the steps can also be added or removed as appropriate. Any method variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0158] Figure 10 This is a schematic diagram of the structure of a handwritten text recognition device provided in an exemplary embodiment of this application. Figure 10 As shown, the device includes:

[0159] Module 1001 is used to acquire handwritten text images;

[0160] Extraction module 1002 is used to extract handwriting style features from the handwritten text image, wherein the handwriting style features are used to reflect the writing style corresponding to the handwritten text image;

[0161] The extraction module 1002 is further configured to calibrate the feature extraction position of the handwritten text image according to the handwriting style features, and extract the text features of the handwritten text image according to the calibrated feature extraction position, wherein the text features are used to reflect the text in the handwritten text image;

[0162] The prediction module 1003 is used to predict the predicted text corresponding to the handwritten text image based on the text features.

[0163] In an optional design, the feature extraction position is the position where the convolution kernel convolves the handwritten text image; the extraction module 1002 is used to predict the position offset of the convolution kernel based on the handwriting style features using a handwritten text recognition model, the position offset being used to reflect the offset of the calibrated convolution position of the convolution kernel relative to the default convolution position; the text features of the handwritten text image are obtained by convolving the handwritten text image with the convolution kernel based on the calibrated convolution position.

[0164] In an optional design, the handwritten text recognition model includes an encoder, which includes a text feature extraction network, which includes a variable convolutional layer; the extraction module 1002 is used to convolve the handwriting style features through the first convolutional kernel of the variable convolutional layer to obtain the position offset of the second convolutional kernel of the variable convolutional layer; and to convolve the handwritten text image through the second convolutional kernel of the variable convolutional layer based on the calibrated convolutional position of the second convolutional kernel to obtain the text features of the handwritten text image.

[0165] In an optional design, the extraction module 1002 is used to convolve the handwriting style features with the first convolution kernel to predict the offset of local image features of the handwritten text image under the handwriting style compared to the standard handwriting style, and to obtain the position offset of the second convolution kernel.

[0166] In an optional design, the extraction module 1002 is used to convolve the handwritten text image with the second convolution kernel of the variable convolution layer based on the default convolution position of the second convolution kernel to obtain the intermediate text features of the handwritten text image; and to convolve the handwriting style features and the intermediate text features with the first convolution kernel of the variable convolution layer to obtain the position offset of the second convolution kernel of the variable convolution layer.

[0167] In an optional design, the encoder further includes a style feature extraction network cascaded with the text feature extraction network; the extraction module 1002 is used to extract the handwritten style features of the handwritten text image through the style feature extraction network.

[0168] In an optional design, the encoder further includes a self-attention network cascaded with the text feature extraction network; such as Figure 11 As shown, the device further includes a processing module 1004, which is used to enhance the text features by means of the self-attention network based on the positional correlation between different characters corresponding to the text features.

[0169] In an optional design, the handwritten text recognition model further includes a decoder, which includes an autoregressive decoding network and is cascaded with the encoder; the prediction module 1003 is used to decode the text features based on autoregression through the autoregressive decoding network to obtain the predicted text corresponding to the handwritten text image.

[0170] In an optional design, such as Figure 12 As shown, the device further includes a training module 1005, used to acquire sample handwritten text images and the corresponding real text; extract sample handwriting style features from the sample handwritten text images using the handwritten text recognition model; extract text features from the sample handwritten text images using the handwritten text recognition model based on the sample handwritten text style features; predict sample text corresponding to the sample handwritten text images using the handwritten text recognition model based on the text features of the sample handwritten text images; calculate error loss based on the error between the sample text and the real text; and train the handwritten text recognition model using the error loss.

[0171] It should be noted that the handwritten text recognition device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the handwritten text recognition device and the handwritten text recognition method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0172] Embodiments of this application also provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the handwritten text recognition method provided in the above-described method embodiments.

[0173] For example, Figure 13 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0174] The computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including random access memory (RAM) 1302 and read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The computer device 1300 also includes a basic input / output system (I / O system) 1306 to facilitate information transfer between various components within the computer device, and a mass storage device 1307 for storing the operating system 1313, application programs 1314, and other program modules 1315.

[0175] The basic input / output system 1306 includes a display 1308 for displaying information and an input device 1309 for user input, such as a mouse or keyboard. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include the input / output controller 1310 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0176] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable storage media provide non-volatile storage for the computer device 1300. That is, the mass storage device 1307 may include computer-readable storage media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0177] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1304 and mass storage device 1307 described above can be collectively referred to as memory.

[0178] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the above method embodiments, and the central processing unit 1301 executes the one or more programs to implement the methods provided by the various method embodiments described above.

[0179] According to various embodiments of this application, the computer device 1300 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1300 can be connected to a network 1312 via a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 can be used to connect to other types of networks or remote computer device systems (not shown).

[0180] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0181] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the handwritten text recognition method provided in the above-described method embodiments is implemented.

[0182] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the handwritten text recognition method provided in the above-described method embodiments.

[0183] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0184] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing handwritten text, characterized in that, The method includes: Acquire images of handwritten text; Handwriting style features are extracted from the handwritten text image, and the handwriting style features are used to reflect the writing style corresponding to the handwritten text image; The feature extraction position of the handwritten text image is calibrated according to the handwriting style features, and the text features of the handwritten text image are extracted according to the calibrated feature extraction position. The text features are used to reflect the text in the handwritten text image. Based on the text features, predict the text corresponding to the handwritten text image.

2. The method according to claim 1, characterized in that, The feature extraction location is the position where the convolution kernel convolves the handwritten text image; The step of calibrating the feature extraction position of the handwritten text image based on the handwriting style features, and extracting the text features of the handwritten text image according to the calibrated feature extraction position, includes: The handwritten text recognition model predicts the position offset of the convolution kernel based on the handwriting style features. The position offset is used to reflect the offset of the calibrated convolution position of the convolution kernel compared with the default convolution position. The text features of the handwritten text image are obtained by convolving the handwritten text image with the convolution kernel based on the calibrated convolution position.

3. The method according to claim 2, characterized in that, The handwritten text recognition model includes an encoder, which includes a text feature extraction network, which includes variable convolutional layers. The step of predicting the position offset of the convolutional kernel based on the handwriting style features using a handwritten text recognition model includes: The handwriting style features are convolved by the first convolution kernel of the variable convolution layer to obtain the position offset of the second convolution kernel of the variable convolution layer; The step of convolving the handwritten text image with the convolution kernel based on the calibrated convolution position to obtain the text features of the handwritten text image includes: The handwritten text image is convolved using the second convolution kernel of the variable convolution layer, based on the calibrated convolution position of the second convolution kernel, to obtain the text features of the handwritten text image.

4. The method according to claim 3, characterized in that, The step of convolving the handwriting style features with the first convolution kernel of the variable convolutional layer to obtain the position offset of the second convolution kernel of the variable convolutional layer includes: The handwriting style features are convolved by the first convolution kernel to predict the offset of local image features of the handwritten text image under the handwriting style compared to the standard handwriting style, and the position offset of the second convolution kernel is obtained.

5. The method according to claim 3, characterized in that, The method further includes: The handwritten text image is convolved using the second convolution kernel of the variable convolution layer based on the default convolution position of the second convolution kernel to obtain the intermediate text features of the handwritten text image. The step of convolving the handwriting style features with the first convolution kernel of the variable convolutional layer to obtain the position offset of the second convolution kernel of the variable convolutional layer includes: The position offset of the second convolution kernel of the variable convolution layer is obtained by convolving the handwriting style features and the intermediate text features with the first convolution kernel of the variable convolution layer.

6. The method according to claim 3, characterized in that, The encoder further includes a style feature extraction network, which is cascaded with the text feature extraction network. The step of extracting handwriting style features from the handwritten text image includes: The handwriting style features of the handwritten text image are extracted using the style feature extraction network.

7. The method according to any one of claims 3 to 6, characterized in that, The encoder further includes a self-attention network, which is cascaded with the text feature extraction network. The method further includes: The self-attention network enhances the text features based on the positional correlation between different characters corresponding to the text features.

8. The method according to any one of claims 3 to 6, characterized in that, The handwritten text recognition model further includes a decoder, which includes an autoregressive decoding network, and the decoder is cascaded with the encoder. The step of predicting the predicted text corresponding to the handwritten text image based on the text features includes: The predicted text corresponding to the handwritten text image is obtained by decoding the text features using the autoregressive decoding network in an autoregressive manner.

9. The method according to any one of claims 2 to 6, characterized in that, The method further includes: Obtain sample handwritten text images and the corresponding real text; The handwritten text recognition model extracts sample handwriting style features based on the sample handwritten text images; The handwritten text recognition model extracts text features from the sample handwritten text image based on the handwriting style features of the sample. The handwritten text recognition model predicts the sample text corresponding to the sample handwritten text image based on the text features of the sample handwritten text image. Calculate the error loss based on the error between the sample text and the real text; The handwritten text recognition model is trained using the error loss method.

10. A handwritten text recognition device, characterized in that, The device includes: The acquisition module is used to acquire images of handwritten text; The extraction module is used to extract handwriting style features from the handwritten text image, wherein the handwriting style features are used to reflect the writing style corresponding to the handwritten text image; The extraction module is further configured to calibrate the feature extraction position of the handwritten text image according to the handwriting style features, and extract the text features of the handwritten text image according to the calibrated feature extraction position, wherein the text features are used to reflect the text in the handwritten text image; The prediction module is used to predict the text corresponding to the handwritten text image based on the text features.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the handwritten text recognition method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the handwritten text recognition method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions to cause the computer device to perform the handwritten text recognition method as described in any one of claims 1 to 9.

Images