Handwriting recognition method and apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of pattern recognition, and provides a handwriting recognition method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: determining handwriting data to be recognized, the handwriting data comprising matched stroke trajectories and handwriting images; based on a trajectory encoder, trajectory features of the stroke trajectories are extracted, and based on an image encoder, image features of the handwriting images are extracted, the trajectory encoder and the image encoder being obtained through differentiated training strategies; based on a dual-modal decoder, differentiated information between the image features and the trajectory features is applied to fuse the trajectory features and the image features, and handwriting recognition is performed based on the fused features. The handwriting recognition method and device, the electronic equipment and the storage medium provided by the application can significantly improve the recognition capability in a dual-modal data mismatching scene.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition technology, and in particular to a handwriting recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] As a simple, convenient, and efficient input method, handwriting recognition has changed the traditional keyboard mode based on complex encoding or Pinyin input, providing convenient conditions for natural and human-computer interaction.

[0003] Currently, handwriting recognition is mainly divided into two types: online modality and offline modality. Online modality is based on the stroke trajectory signal collected during the user's handwriting process for handwriting recognition; offline modality is based on the image obtained from the user's handwriting for handwriting recognition.

[0004] Although there are schemes that combine online and offline modalities for joint modeling, in actual handwriting recognition scenarios, the cost of online handwriting trajectory acquisition is high, and the number of offline images is often greater than the number of online trajectories, resulting in an imbalance between online and offline modalities and greatly reducing recognition accuracy. Summary of the Invention

[0005] This invention provides a handwriting recognition method, device, electronic device, and storage medium to address the shortcomings of existing technologies, such as limited single-modal recognition performance and unbalanced dual-modal data leading to poor recognition accuracy.

[0006] This invention provides a handwriting recognition method, comprising:

[0007] The handwritten data to be identified includes the matching stroke trajectory and the handwritten image;

[0008] Based on the trajectory encoder, the trajectory features of the stroke trajectory are extracted, and based on the image encoder, the image features of the handwritten image are extracted. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0009] Based on a dual-modal decoder, the differential information between the image features and the trajectory features is applied to fuse the trajectory features and the image features, and handwriting recognition is performed based on the fused features.

[0010] According to the handwriting recognition method provided by the present invention, the step of fusing the trajectory features and the image features by applying the differential information between the image features and the trajectory features includes:

[0011] The image features of the handwritten image are context-encoded to obtain the image context features at the current decoding time, and the trajectory features of the stroke trajectory are context-encoded to obtain the trajectory context features at the current decoding time.

[0012] The trajectory context features and the image context features are fused by applying the differential information between the image context features and the trajectory context features.

[0013] According to the handwriting recognition method provided by the present invention, the step of applying the differential information between the image context features and the trajectory context features to fuse the trajectory context features and the image context features includes:

[0014] Projecting either the image context feature or the trajectory context feature onto the feature space of the other feature yields orthogonal feature components in the same feature that are orthogonal to the other feature. These orthogonal feature components are used to characterize the differential information between the image context feature and the trajectory context feature.

[0015] The orthogonal feature component of any one of the features is fused with the other feature.

[0016] According to the handwriting recognition method provided by the present invention, the step of extracting trajectory features of the stroke trajectory based on the trajectory encoder includes:

[0017] Based on the trajectory encoder, the first trajectory feature of the stroke trajectory is extracted, and based on the image encoder, the first image feature of the handwritten image is extracted.

[0018] Based on the position of each trajectory point in the stroke trajectory in the first image feature, the first image feature is sampled to obtain the second image feature of the handwritten image;

[0019] The first trajectory feature is fused with the second image feature, and the fused feature is used as the trajectory feature of the stroke trajectory.

[0020] According to the handwriting recognition method provided by the present invention, the handwriting data further includes non-matching stroke trajectories and / or non-matching handwriting images, and the method further includes:

[0021] Based on the trajectory encoder, the trajectory features of the non-matching stroke trajectories are extracted, and based on the trajectory decoder, handwriting recognition is performed on the trajectory features of the non-matching stroke trajectories; and / or,

[0022] Based on the image encoder, image features of the non-matching handwritten image are extracted, and based on the image decoder, handwriting recognition is performed on the image features of the non-matching handwritten image.

[0023] According to the handwriting recognition method provided by the present invention, the trajectory encoder, the image encoder, the dual-modal decoder, and the single-modal decoder constitute a handwriting recognition model, wherein the single-modal decoder includes the trajectory decoder and / or the image decoder, and the steps for obtaining the handwriting recognition model include:

[0024] Obtain an initial trajectory encoder, an initial image encoder, an initial bimodal decoder, and an initial unimodal decoder, and randomly initialize the parameters of the initial trajectory encoder, wherein the initial image encoder is a pre-trained image text recognition model;

[0025] Based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

[0026] According to the handwriting recognition method provided by the present invention, the step of iterating the parameters of the handwriting recognition model based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder to obtain the handwriting recognition model includes:

[0027] Based on the prediction results of the initial bimodal decoder, distillation learning is performed on the initial unimodal decoder to obtain the distillation loss of the initial unimodal decoder;

[0028] Based on the distillation loss of the initial single-modal decoder, the decoding loss of the initial single-modal decoder, and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

[0029] The present invention also provides a handwriting recognition device, comprising:

[0030] A data determination unit is used to determine the handwritten data to be identified, wherein the handwritten data includes matching stroke trajectories and handwritten images;

[0031] The feature extraction unit is used to extract the trajectory features of the stroke trajectory based on the trajectory encoder and to extract the image features of the handwritten image based on the image encoder. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0032] The handwriting recognition unit is used to fuse the trajectory features and the image features based on the dual-modal decoder, by applying the differential information between the image features and the trajectory features, and to perform handwriting recognition based on the fused features.

[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the handwriting recognition method as described above.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the handwriting recognition method as described above.

[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the handwriting recognition method as described above.

[0036] The handwriting recognition method, device, electronic device, and storage medium provided by this invention fully decouple stroke trajectory and handwritten image, extracting their respective features. At the same time, to address the problem of data imbalance between stroke trajectory and handwritten image, a differentiated training strategy is adopted to train a trajectory encoder and an image encoder. Based on this, by learning the differential information between online trajectory modality and offline image modality, the recognition ability in dual-modal data mismatch scenarios can be significantly improved. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of the handwriting recognition method provided by the present invention;

[0039] Figure 2 This is one of the flowcharts of step 130 in the handwriting recognition method provided by the present invention;

[0040] Figure 3 This is the second flowchart of step 130 in the handwriting recognition method provided by the present invention;

[0041] Figure 4 This is a schematic diagram of the fusion process of the first fusion module provided by the present invention;

[0042] Figure 5 This is a flowchart illustrating step 120 in the handwriting recognition method provided by the present invention;

[0043] Figure 6 This is a schematic diagram of the fusion process of the second fusion module provided by the present invention;

[0044] Figure 7 This is the second flowchart of the handwriting recognition method provided by the present invention;

[0045] Figure 8 This is one of the flowcharts illustrating the handwriting recognition model acquisition method provided by the present invention;

[0046] Figure 9 This is the second flowchart illustrating the handwriting recognition model acquisition method provided by the present invention;

[0047] Figure 10 This is a schematic diagram of the structure of the handwriting recognition model provided by the present invention;

[0048] Figure 11 This is a schematic diagram of the handwriting recognition device provided by the present invention;

[0049] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] In recent years, with the rapid development of Convolutional Neural Networks (CNNs), Optical Character Recognition (OCR) technology based on deep learning has been widely used in image recognition systems such as photo document recognition, form recognition, and ID card recognition, as well as handwritten text recognition systems.

[0052] Handwritten character recognition can be broadly categorized into offline and online handwritten character recognition based on different data acquisition methods. Online handwritten character recognition typically uses the stroke trajectory signals of the writer's handwriting, sampled periodically by a physical acquisition device such as a digital pen or touchscreen. The acquired writing trajectory is a one-dimensional temporal sequence. Offline character recognition, on the other hand, typically uses two-dimensional images captured by image capture devices such as scanners or cameras.

[0053] Because the objects being recognized differ, offline and online handwritten character recognition employ different methods and strategies. Compared to images, which only contain spatial information about the writing, stroke trajectories include not only spatial information but also temporal information about the writing process. Therefore, under the same conditions, online character recognition often achieves better recognition results. Furthermore, photographic scanning devices are affected by different lighting conditions, writing paper, resolution, and other factors, introducing more noise during the digitization process. Due to the richer information contained, online handwriting recognition can usually handle scenarios with illegible or ambiguous handwriting well, while offline handwriting recognition, lacking temporal information, often cannot solve these problems. However, online handwriting trajectories lack global image information and cannot effectively handle repetitive strokes and inserted strokes, while offline handwriting recognition can effectively address these issues. Therefore, jointly modeling online trajectory modalities and offline image modalities, and complementing the information from different modalities, can achieve better handwriting recognition performance.

[0054] In related technologies, online trajectory and offline image data are mostly paired and balanced. However, online handwritten trajectory acquisition is cumbersome and costly. In real-world scenarios, the number of offline images often far exceeds the number of online trajectories, leading to a data imbalance between online and offline modalities and significantly reducing recognition accuracy. Furthermore, if only the matched data is used, a large amount of offline image data will be discarded and cannot be effectively utilized.

[0055] Based on the above considerations, in order to improve the recognition accuracy in complex scenes, the inventive concept of this invention is as follows: to address the limited effect of single-modal recognition and the problem of imbalanced dual-modal data, online trajectory modality and offline image modality are jointly modeled. On this basis, by learning the differential information between online trajectory modality and offline image modality, the recognition ability in dual-modal data mismatch scenarios can be significantly improved.

[0056] Based on the above-mentioned inventive concept, the present invention provides a handwriting recognition method, device, electronic device and storage medium, which are applied to handwriting recognition scenarios in artificial intelligence technology, such as handwriting input methods, photographed documents, forms, and document recognition, so as to improve the recognition accuracy in complex scenarios.

[0057] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings. Figure 1 This is one of the flowcharts illustrating the handwriting recognition method provided by the present invention. The execution entity for each step in this method can be a handwriting recognition device, which can be implemented through software and / or hardware. This device can be integrated into an electronic device, which can be a terminal device (such as a smartphone, personal computer, learning machine, scanning pen, etc.), a server (such as a local server or cloud server, or a server cluster, etc.), a processor, or a chip, etc. Figure 1As shown, the method may include the following steps:

[0058] Step 110: Determine the handwritten data to be identified, which includes the matching stroke trajectory and the handwritten image.

[0059] Specifically, the handwritten data to be recognized is the data for which handwriting recognition is required. This handwritten data can be matched stroke trajectories and handwritten images; that is, the stroke trajectories and handwritten images are mutually matched and correspond to each other. Each point in the stroke trajectory has a corresponding pixel in the handwritten image. The stroke trajectory is the trajectory information of the strokes written by the user, including the position coordinates of each trajectory point and the input timing of each trajectory point. The matched stroke trajectories and handwritten images can be understood as bimodal data.

[0060] Step 120: Based on the trajectory encoder, extract the trajectory features of the stroke trajectory, and based on the image encoder, extract the image features of the handwritten image. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0061] Specifically, the stroke trajectory not only contains the spatial information of the strokes written by the user, but also the temporal information of the writing. By extracting the trajectory features of the stroke trajectory, we can obtain the relationship between each trajectory point and its adjacent trajectory points in the spatial and temporal dimensions, and thus obtain the trajectory features.

[0062] Handwritten images contain only the spatial information of the strokes written by the user. By extracting spatial features from handwritten images, we can obtain the spatial relationship between each pixel and its neighboring pixels, thus deriving image features. These image features include the image features of each pixel itself and its neighborhood space.

[0063] Here, both the trajectory encoder and the image encoder can be implemented using CNN (Convolutional Neural Networks) architectures such as ResNet (Deep Residual Network) and DenseNet. The trajectory encoder uses 1D convolutional layers, and the image encoder uses 1D convolutional layers.

[0064] It should be noted that, considering the vastly larger number of offline handwritten images compared to online stroke trajectories in real-world applications, a differentiated training strategy is employed for the trajectory encoder and image encoder to fully utilize unpaired offline images while saving training costs. This differentiated training strategy can be reflected in various aspects such as the selection of optimization algorithms, the setting and adjustment of learning rates, and the setting of iterative periods.

[0065] Taking optimization algorithms as an example, different optimization algorithms will result in different convergence speeds and convergence qualities. An appropriate learning rate is also crucial. If the learning rate is set too high, training may diverge; if it is set too low, although it may converge to the optimal solution, the training speed will be too slow, wasting a significant amount of time. Considering the difference in data volume between offline handwritten images and online stroke trajectories, differentiated learning rates can be set for the trajectory encoder and image encoder respectively. The iterable period refers to the maximum number of times a complete training cycle is allowed, which can be determined by the size of the training set. Therefore, differentiated training strategies can be adopted for the trajectory encoder and image encoder from various aspects, including the selection of optimization algorithms, the setting and adjustment of learning rates, and the setting of iterable periods.

[0066] Preferably, considering the learning rate, the image encoder can use a pre-trained model, that is, load the parameters of the pre-trained model; while the trajectory encoder can use a random parameter initialization method, which can ensure that the trajectory encoder learns from scratch.

[0067] Step 130: Based on the dual-modal decoder, apply the differential information between image features and trajectory features to fuse trajectory features and image features, and perform handwriting recognition based on the fused features.

[0068] Specifically, after obtaining the image features of the handwritten image and the trajectory features of the stroke trajectory, the trajectory features and image features can be fused based on the dual-modal decoder by applying the differential information between the image features and the trajectory features.

[0069] The differential information between image features and trajectory features can reflect the different data representations of trajectory modalities and image modalities, as well as the different characterizations of data distribution due to differences in data scale. Since trajectory features and image features are completely decoupled and obtained by different encoders, the trajectory encoder and image encoder can continuously learn the differential information between image features and trajectory features during training.

[0070] In some embodiments, the differential information between image features and trajectory features can be obtained through orthogonal projection. Image features can be projected onto the feature space of trajectory features, and vice versa, thereby obtaining the differential information between image features and trajectory features.

[0071] After obtaining the differential information between image features and trajectory features, the image features and trajectory features can be fused based on the differential information, and handwriting recognition can be performed based on the fused features. A dual-modal decoder can be used to decode the fused features at each time step, and the final decoding result is the handwriting recognition result.

[0072] Furthermore, the aforementioned handwriting recognition method can be implemented using a pre-trained handwriting recognition model. This model is built upon an encoder-decoder architecture, specifically including a trajectory encoder, an image encoder, and a bimodal decoder. The trajectory encoder encodes the stroke trajectories to obtain their trajectory features; the image encoder encodes the handwritten image to obtain its image features; and the bimodal decoder decodes the obtained trajectory features of the stroke trajectories and the image features of the handwritten image to obtain the handwriting recognition result. The recognition result can be the character represented by the stroke trajectory and its corresponding handwritten image.

[0073] The method provided in this invention fully decouples stroke trajectory and handwritten image, extracts their respective features, and adopts a differentiated training strategy to train trajectory encoder and image encoder to address the data imbalance problem between stroke trajectory and handwritten image. Based on this, by learning the differential information between online trajectory modality and offline image modality, the recognition ability in dual-modal data mismatch scenarios can be significantly improved.

[0074] Based on the above embodiments, Figure 2 This is one of the flowcharts illustrating step 130 in the handwriting recognition method provided by the present invention, such as... Figure 2 As shown, step 130 applies the differential information between image features and trajectory features to fuse the trajectory features and image features, specifically including:

[0075] Step 131: Context-encode the image features of the handwritten image to obtain the image context features at the current decoding time; Context-encode the trajectory features of the stroke trajectory to obtain the trajectory context features at the current decoding time.

[0076] Step 132: Apply the differential information between image context features and trajectory context features to fuse the trajectory context features and image context features.

[0077] Specifically, in the dual-modal decoder decoding process, because the handwriting process itself is temporal, its decoding process is also a serialized output process. During decoding, the importance of the trajectory features of each trajectory point at each decoding moment is different, and the importance of the image features of each pixel is also different.

[0078] Therefore, in order to eliminate the interference of irrelevant trajectory features and irrelevant pixel features on the decoding operation at the current decoding moment and improve the decoding accuracy, the image features of the handwritten image can be context-encoded to obtain the image context features at the current decoding moment, and the trajectory features of the stroke trajectory can be context-encoded to obtain the trajectory context features at the current decoding moment.

[0079] Considering that each pixel in a handwritten image cannot be isolated, and that a pixel must have some relationship with its surrounding pixels, a handwritten image is formed through the interconnections of a large number of pixels. Therefore, context encoding of image features, compared to ordinary encoding, can fuse the contextual information of image features and represent a certain relationship between pixels and their surrounding pixels. Features with strong contextual relationships can be given a larger fusion weight, while features with weak contextual relationships can be given a smaller fusion weight. Context encoding of image features not only fuses the contextual information of the image but also needs to determine which image features should be given priority at the current decoding moment, thus encoding the image context features at the current decoding moment. Here, the image context features at the current decoding moment highlight the image features that should be given priority at the current decoding moment.

[0080] Accordingly, context encoding of trajectory features can fuse the contextual information of the trajectory features, representing a certain relationship between the trajectory point and its surrounding trajectory points. Features with strong contextual relationships can be given a larger fusion weight, while features with weak contextual relationships can be given a smaller fusion weight. The process of context encoding trajectory features also needs to determine which trajectory features in the trajectory point sequence should be focused on at the current decoding time, thereby encoding the trajectory context features at the current decoding time. Here, the trajectory context features at the current decoding time highlight the trajectory features that should be focused on at the current decoding time.

[0081] Furthermore, context encoding of image features and trajectory features can be performed using an attention mechanism. The bimodal decoder may include an image attention module for context encoding of image features and a trajectory attention module for context encoding of trajectory features.

[0082] Based on this, the differential information between image context features and trajectory context features is applied to fuse the trajectory context features and image context features.

[0083] The method provided in this invention improves the accuracy of decoding by applying differential information between image context features and trajectory context features and fusing trajectory context features with image context features, thereby improving the accuracy of handwriting recognition.

[0084] Based on any of the above embodiments Figure 3 This is the second flowchart of step 130 in the handwriting recognition method provided by the present invention, as shown below. Figure 3 As shown, step 132 specifically includes:

[0085] Step 132-1: Project any one of the image context features and trajectory context features onto the feature space of the other feature to obtain the orthogonal feature component in any feature that is orthogonal to the other feature; the orthogonal feature component in any feature is used to characterize the differential information between the image context features and the trajectory context features.

[0086] Step 132-2: Fuse the orthogonal feature component of any feature with another feature.

[0087] Specifically, the differential information between image context features and trajectory context features can be obtained through orthogonal projection.

[0088] First, project either the image context feature or the trajectory context feature onto the feature space of the other feature to obtain the orthogonal feature component of any feature orthogonal to the other feature. Here, if either feature is an image context feature, then the other feature is a trajectory context feature; if either feature is a trajectory context feature, then the other feature is an image context feature.

[0089] Then, the orthogonal feature component in any feature orthogonal to the other feature obtained after projection can be used to characterize the differential information between image context features and trajectory context features.

[0090] Then, by fusing the orthogonal feature component of any feature with another feature, the fused feature of trajectory context feature and image context feature can be obtained.

[0091] In some embodiments, the fusion process in step 132 can be implemented based on the first fusion module in the dual-modal decoder. Figure 4 This is a schematic diagram of the fusion process provided by the present invention, such as... Figure 4 As shown, vector AD represents the trajectory context features. Vector BC represents the image context features. Vector BD is the orthogonal feature component in the trajectory context features that is orthogonal to the image context features. Vector BD can represent the differential information between image context features and trajectory context features.

[0092] Trajectory context features in online modality Image context features projected onto the offline modality The feature space is obtained. Zhongyu orthogonal eigencomponents It can be represented in the following form:

[0093]

[0094]

[0095] In the formula, Represents trajectory context features, F represents the contextual features of an image. i on Represents the i-th trajectory feature. This represents the weight of the i-th trajectory feature at the t-th decoding time. Represents the j-th image feature. This represents the weight of the j-th image feature at the t-th decoding time.

[0096] It can be considered as a cross-modal attention mechanism between online trajectory features and offline image features, which greatly reduces the amount of computation compared to conventional cross-modal attention schemes.

[0097] The method provided in this invention fuses data from both online and offline modalities by learning the differential information between image context features and trajectory context features, thereby improving recognition accuracy while reducing computational load.

[0098] Based on any of the above embodiments Figure 5 This is a flowchart illustrating step 120 of the handwriting recognition method provided by the present invention, as shown below. Figure 5 As shown, step 120, based on the trajectory encoder, extracts the trajectory features of the stroke trajectory, specifically including:

[0099] Step 121: Based on the trajectory encoder, extract the first trajectory features of the stroke trajectory, and based on the image encoder, extract the first image features of the handwritten image;

[0100] Step 122: Based on the position of each trajectory point in the stroke trajectory in the first image features, sample the first image features to obtain the second image features of the handwritten image;

[0101] Step 123: Fuse the first trajectory features with the second image features, and use the fused features as the trajectory features of the stroke trajectory.

[0102] Specifically, for the matched handwritten data, in addition to fusing the extracted image features and trajectory features during decoding, since the matched stroke trajectory and the handwritten image have a one-to-one spatial correspondence, in order to better extract cross-modal features, a fusion method can also be used for trajectory feature extraction during the feature extraction stage.

[0103] Figure 6 This is a schematic diagram of the structure of the second fusion module provided by the present invention, as shown below. Figure 6 As shown, steps 121-123 can be implemented through the second fusion module. Based on the trajectory encoder, the first trajectory features of the stroke trajectory are extracted, and based on the image encoder, the first image features of the handwritten image are extracted. Then, using the spatial correspondence between the image mode and the trajectory mode, and using the position of the coordinates of each trajectory point in the stroke trajectory in the first image features, the first image features are sampled to obtain the second image features of the handwritten image.

[0104] Subsequently, the sampled second image features are fused with the first trajectory features, and the fused features are used as the trajectory features of the stroke trajectory. The fusion can be performed by feature concatenation or feature addition, and this embodiment of the invention does not specifically limit the method.

[0105] The trajectory features of the extracted stroke trajectories can represent cross-modal spatiotemporally consistent feature information that integrates image feature spatial information.

[0106] The method provided in this invention, for matching handwritten images and stroke trajectories, adopts a fusion approach in the feature extraction stage to obtain cross-modal spatiotemporally consistent trajectory features, thereby achieving better extraction of cross-modal features and improving the accuracy of handwriting recognition.

[0107] Based on any of the above embodiments Figure 7 This is the second flowchart of the handwriting recognition method provided by the present invention, as shown below. Figure 7 As shown, the handwritten data also includes non-matching stroke trajectories and / or non-matching handwritten images, and the handwriting recognition method also includes:

[0108] Step 710: Based on the trajectory encoder, extract the trajectory features of the non-matching stroke trajectories, and based on the trajectory decoder, perform handwriting recognition on the trajectory features of the non-matching stroke trajectories; and / or,

[0109] Step 720: Based on the image encoder, extract the image features of the non-matching handwritten images, and based on the image decoder, perform handwriting recognition on the image features of the non-matching handwritten images.

[0110] Specifically, the method provided in this embodiment of the invention can not only identify matching handwritten images and stroke trajectories, but also identify non-matching stroke trajectories and / or non-matching handwritten images.

[0111] For non-matching stroke trajectories, a trajectory encoder is used to extract the trajectory features of the non-matching stroke trajectories, and a trajectory decoder is used to perform handwriting recognition on the trajectory features of the non-matching stroke trajectories.

[0112] For non-matching handwritten images, image features of the non-matching handwritten images are extracted based on an image encoder, and handwriting recognition is performed on the image features of the non-matching handwritten images based on an image decoder.

[0113] Among them, the trajectory decoder and the image decoder can be decoders with decoding functions. The one used to decode the trajectory features of non-matching stroke trajectories can be called a trajectory decoder, and the one used to decode the image features of non-matching handwritten images can be called an image decoder.

[0114] Preferably, the trajectory decoder and the image decoder can have the same structure, for example, both consisting of attention units and gated recurrent units (GRUs).

[0115] For a one-dimensional stroke trajectory, the trajectory feature sequence obtained by the trajectory encoder can be represented as follows:

[0116]

[0117] In the formula, F on f represents the trajectory feature sequence. i on This represents the i-th trajectory feature.

[0118] Since the final predicted character sequence is a variable-length sequence, a fixed-dimensional trajectory context feature is calculated at each time step using an attention mechanism. And a one-way GRU module is used to predict string sequences.

[0119]

[0120]

[0121] In the formula, w represents the trajectory context features. i e represents the attention weight of the i-th trajectory feature. i This represents the attention score for the i-th trajectory feature. and U att This is the weight matrix.

[0122] Similarly, for non-matching handwritten images, the encoded image features can be obtained through an image encoder:

[0123]

[0124] In the formula, F off f represents the image feature sequence. i off This represents the i-th image feature.

[0125] Attention mechanisms are used to compute fixed-dimensional trajectory context features at each time step. And a one-way GRU module is used to predict string sequences.

[0126]

[0127]

[0128] In the formula, w represents the contextual features of an image. i e represents the attention weight of the i-th image feature. i This represents the attention score for the i-th trajectory feature. and U att This is the weight matrix.

[0129] The method provided in this invention, through a trajectory decoder and / or an image decoder, can not only process text recognition of paired handwritten data, but also has good compatibility with text recognition of unpaired handwritten data.

[0130] Based on any of the above embodiments Figure 8 This is one of the flowcharts illustrating the handwriting recognition model acquisition method provided by the present invention, such as... Figure 8 As shown, the handwriting recognition model consists of a trajectory encoder, an image encoder, a dual-modal decoder, and a single-modal decoder. The single-modal decoder includes a trajectory decoder and / or an image decoder. The steps for obtaining the handwriting recognition model include:

[0131] Step 810: Obtain the initial trajectory encoder, initial image encoder, initial bimodal decoder, and initial unimodal decoder, and randomly initialize the parameters of the initial trajectory encoder, wherein the initial image encoder is a pre-trained image text recognition model;

[0132] Step 820: Based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder, perform parameter iteration on the handwriting recognition model to obtain the handwriting recognition model.

[0133] Specifically, in order to obtain a handwriting recognition model, an initial trajectory encoder, an initial image encoder, an initial bimodal decoder, and an initial unimodal decoder can be obtained first. Here, the initial unimodal decoder can be the initial trajectory decoder; it can also be the initial image decoder; or it can include both the initial trajectory decoder and the initial image decoder. This embodiment of the invention does not specifically limit this.

[0134] Both the initial trajectory encoder and the initial image encoder can be implemented using CNN (Convolutional Neural Networks) architectures such as ResNet (Deep Residual Network) and DenseNet. The initial trajectory encoder uses 1D convolutional layers, and the initial image encoder also uses 1D convolutional layers.

[0135] The initial trajectory decoder and the initial image decoder have basically the same structure, both consisting of attention units and gated recurrent units (GRUs).

[0136] Considering that in real-world scenarios, the number of offline images far exceeds the number of online trajectories, in order to make full use of unpaired offline images while saving training costs, the initial image encoder can use a pre-trained offline image text recognition model as a pre-trained model.

[0137] Furthermore, since the amount of offline modal data is far greater than the amount of online trajectories, online trajectory encoders require differential learning. Unlike offline image encoders that load pre-trained parameters, online trajectory encoders initialize their model parameters randomly. This random initialization ensures that the model learns differential knowledge from scratch. Pre-trained online trajectory models may limit the model to a small amount of online trajectory data and ignore the differential cross-modal data distribution.

[0138] Online trajectory encoders also update at a different speed than offline image encoder models. Online trajectory encoders learn differentiated information from scratch (different data representations of trajectory modalities and image modalities, as well as different characterizations of handwritten data distributions due to differences in data scale between different modalities).

[0139] During training, the handwriting recognition model can be iterated by taking the initial decoding loss of the single-modal decoder and the initial decoding loss of the dual-modal decoder to obtain the handwriting recognition model.

[0140] In one embodiment, the initial unimodal decoder includes an initial trajectory decoder and an initial image decoder, and the model loss of the handwriting recognition model can be expressed as follows:

[0141] l all =lmul +l online +l offline

[0142] In the formula, l all The model loss of the handwriting recognition model is represented by l. online l represents the decoding loss of the initial trajectory decoder. offline l represents the decoding loss of the initial image decoder. mul This represents the decoding loss of the initial bimodal decoder.

[0143] Understandably, the learning speed of the initial unimodal decoder can be dynamically adjusted during training based on its decoding loss. When the initial unimodal decoder only includes the initial trajectory decoder, its learning speed can be dynamically adjusted based on the decoding loss of the initial trajectory decoder; when the initial unimodal decoder only includes the initial image decoder, its learning speed can be dynamically adjusted based on the decoding loss of the initial image decoder; when the initial unimodal decoder includes both the initial trajectory decoder and the initial image decoder, the learning rate is adjusted by the decoding losses of the initial trajectory decoder and the initial image decoder to dynamically balance the learning speeds of the online trajectory decoder and the offline image decoder, thereby enabling the online trajectory decoder and the offline image decoder to achieve the best recognition results simultaneously, further improving the recognition performance.

[0144] It should be noted that before performing step 810, a bimodal handwriting dataset can be prepared. This dataset can contain either matched handwriting images and stroke trajectories, or it can contain unpaired stroke trajectories or handwriting images.

[0145] Online stroke trajectories can be represented as a variable-length sequence of the writing process, as follows:

[0146] {[x1,y1,s1],[x2,y2,s2],…,[x N ,y N ,s N ]}

[0147] Where x i ,y i Let s represent the coordinates of the trajectory in the xy coordinate system, respectively. i This indicates which trajectory the i-th trajectory point belongs to. The relative position changes of each trajectory point can be used as input to the model, and each trajectory point can be represented as:

[0148] [Δx i ,Δy i ,δ(s i =s i+1 )]

[0149] Based on any of the above embodiments Figure 9 This is the second flowchart illustrating the handwriting recognition model acquisition method provided by the present invention, as shown below. Figure 9 As shown, step 810 specifically includes:

[0150] Step 811: Based on the prediction results of the initial dual-modal decoder, perform distillation learning on the initial single-modal decoder to obtain the distillation loss of the initial single-modal decoder;

[0151] Step 812: Based on the distillation loss of the initial single-modal decoder, the decoding loss of the initial single-modal decoder, and the decoding loss of the initial dual-modal decoder, the parameters of the handwriting recognition model are iterated to obtain the handwriting recognition model.

[0152] Specifically, to further utilize the features between different modalities, the initial bimodal decoder can be used as the teacher model, and the initial unimodal decoder as the student model. Based on the prediction results of the initial bimodal decoder, distillation learning is performed on the initial unimodal decoder to obtain the distillation loss of the initial unimodal decoder. It is understood that the distillation loss of the initial unimodal decoder may include the distillation loss of the initial trajectory decoder and / or the initial image decoder.

[0153] Based on this, the handwriting recognition model is obtained by iterating the parameters of the handwriting recognition model using the distillation loss of the initial single-modal decoder, the decoding loss of the initial single-modal decoder, and the decoding loss of the initial dual-modal decoder.

[0154] In some embodiments, the model loss of a handwriting recognition model can be expressed as follows:

[0155] l all =l mul +l online +l offline +l distill-onlin +l distill-offline

[0156] In the formula, l all The model loss of the handwriting recognition model is represented by l. online l represents the decoding loss of the initial trajectory decoder. offline l represents the decoding loss of the initial image decoder. distill*onlin The distillation loss of the initial trajectory decoder is represented by l. distill-offli This represents the distillation loss of the initial image decoder.

[0157] The method provided in this invention optimizes the single-mode prediction capability using the prediction results of dual-mode prediction, thereby further improving the prediction results of dual-mode prediction.

[0158] Based on any of the above embodiments Figure 10This is a schematic diagram of the structure of the handwriting recognition model provided by the present invention, as shown below. Figure 10 As shown, the handwriting recognition model includes five sub-networks: trajectory encoder, image encoder, bimodal decoder, trajectory decoder, and image decoder.

[0159] The overall framework of the handwriting recognition model is based on an encoder-decoder structure. The encoder extracts features from the input and mainly consists of two parts: an offline image encoder and an online trajectory encoder. The offline image encoder encodes the image into two-dimensional image modal features, while the online trajectory encoder encodes the writing trajectory into one-dimensional writing trajectory features. The decoder then temporally decodes the features encoded by the encoder to obtain the character sequence.

[0160] For paired trajectory and image pairs, the bimodal decoder decodes both image features and trajectory features into a character sequence. For unpaired images, the image decoder decodes image features into a character sequence independently; for unpaired trajectories, the trajectory decoder decodes trajectory features into a character sequence.

[0161] Therefore, this handwriting recognition model can effectively handle both paired and unpaired text recognition while processing text recognition of paired data.

[0162] The trajectory decoder includes a trajectory attention module and a trajectory gating loop module. The trajectory attention module is used to perform context encoding on the trajectory features of the stroke trajectory to obtain the trajectory context features at the current decoding time. The trajectory gating loop module is used to perform handwriting recognition based on the trajectory context features to obtain the predicted string sequence.

[0163] The image decoder includes an image attention module and an image gated loop module. The image attention module is used to perform context encoding on the image features of the handwritten image to obtain the image context features at the current decoding time. The image gated loop module is used to perform handwriting recognition based on the image context features to obtain the predicted string sequence.

[0164] The dual-modal decoder includes a first fusion module and a dual-modal gated loop module. The first fusion module is used to fuse the trajectory context features and the image context features based on the differential information between the image context features and the trajectory context features to obtain the fused features.

[0165] The dual-modal gated loop module is used for handwriting recognition based on fused features.

[0166] The handwriting recognition device provided by the present invention is described below. The handwriting recognition device described below can be referred to in correspondence with the handwriting recognition method described above.

[0167] Based on any of the above embodiments Figure 11This is a schematic diagram of the handwriting recognition device provided by the present invention, as shown below. Figure 11 As shown, the handwriting recognition device includes a data determination unit 1110, a feature extraction unit 1120, and a handwriting recognition unit 1130, wherein:

[0168] A data determination unit is used to determine the handwritten data to be identified, wherein the handwritten data includes matching stroke trajectories and handwritten images;

[0169] The feature extraction unit is used to extract the trajectory features of the stroke trajectory based on the trajectory encoder and to extract the image features of the handwritten image based on the image encoder. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0170] The handwriting recognition unit is used to fuse the trajectory features and the image features based on the dual-modal decoder, by applying the differential information between the image features and the trajectory features, and to perform handwriting recognition based on the fused features.

[0171] The handwriting recognition device provided in this embodiment of the invention fully decouples the stroke trajectory and the handwritten image, extracts their respective features, and adopts a differentiated training strategy to train the trajectory encoder and the image encoder to address the problem of data imbalance between the stroke trajectory and the handwritten image. Based on this, by learning the differential information between the online trajectory modality and the offline image modality, the recognition ability in the dual-modal data mismatch scenario can be significantly improved.

[0172] Based on any of the above embodiments, the handwriting recognition unit is specifically used for:

[0173] The image features of the handwritten image are context-encoded to obtain the image context features at the current decoding time, and the trajectory features of the stroke trajectory are context-encoded to obtain the trajectory context features at the current decoding time.

[0174] The trajectory context features and the image context features are fused by applying the differential information between the image context features and the trajectory context features.

[0175] Based on any of the above embodiments, the handwriting recognition unit is further specifically used for:

[0176] Projecting either the image context feature or the trajectory context feature onto the feature space of the other feature yields orthogonal feature components in the same feature that are orthogonal to the other feature. These orthogonal feature components are used to characterize the differential information between the image context feature and the trajectory context feature.

[0177] The orthogonal feature component of any one of the features is fused with the other feature.

[0178] Based on any of the above embodiments, the feature extraction unit is specifically used for:

[0179] Based on the trajectory encoder, the first trajectory feature of the stroke trajectory is extracted, and based on the image encoder, the first image feature of the handwritten image is extracted.

[0180] Based on the position of each trajectory point in the stroke trajectory in the first image feature, the first image feature is sampled to obtain the second image feature of the handwritten image;

[0181] The first trajectory feature is fused with the second image feature, and the fused feature is used as the trajectory feature of the stroke trajectory.

[0182] Based on any of the above embodiments, the handwriting recognition device further includes a single-modal recognition unit, used for:

[0183] In cases where the handwritten data includes non-matching stroke trajectories and / or non-matching handwritten images, the trajectory features of the non-matching stroke trajectories are extracted based on the trajectory encoder, and handwriting recognition is performed on the trajectory features of the non-matching stroke trajectories based on the trajectory decoder; and / or,

[0184] Based on the image encoder, image features of the non-matching handwritten image are extracted, and based on the image decoder, handwriting recognition is performed on the image features of the non-matching handwritten image.

[0185] Based on any of the above embodiments, the handwriting recognition device further includes a model acquisition unit, wherein the model acquisition unit is specifically used for:

[0186] Obtain an initial trajectory encoder, an initial image encoder, an initial bimodal decoder, and an initial unimodal decoder, and randomly initialize the parameters of the initial trajectory encoder, wherein the initial image encoder is a pre-trained image text recognition model;

[0187] Based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

[0188] Based on any of the above embodiments, the model acquisition unit is further specifically used for:

[0189] Based on the prediction results of the initial bimodal decoder, distillation learning is performed on the initial unimodal decoder to obtain the distillation loss of the initial unimodal decoder;

[0190] Based on the distillation loss of the initial single-modal decoder, the decoding loss of the initial single-modal decoder, and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

[0191] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12 As shown, the electronic device may include: a processor 1210, a communications interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communications interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 can call logical instructions in the memory 1230 to execute a handwriting recognition method, which includes:

[0192] The handwritten data to be identified includes the matching stroke trajectory and the handwritten image;

[0193] Based on the trajectory encoder, the trajectory features of the stroke trajectory are extracted, and based on the image encoder, the image features of the handwritten image are extracted. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0194] Based on a dual-modal decoder, the differential information between the image features and the trajectory features is applied to fuse the trajectory features and the image features, and handwriting recognition is performed based on the fused features.

[0195] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the handwriting recognition method provided by the above methods, the method comprising:

[0197] The handwritten data to be identified includes the matching stroke trajectory and the handwritten image;

[0198] Based on the trajectory encoder, the trajectory features of the stroke trajectory are extracted, and based on the image encoder, the image features of the handwritten image are extracted. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0199] Based on a dual-modal decoder, the differential information between the image features and the trajectory features is applied to fuse the trajectory features and the image features, and handwriting recognition is performed based on the fused features.

[0200] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the handwriting recognition method provided by the methods described above, the method comprising:

[0201] The handwritten data to be identified includes the matching stroke trajectory and the handwritten image;

[0202] Based on the trajectory encoder, the trajectory features of the stroke trajectory are extracted, and based on the image encoder, the image features of the handwritten image are extracted. The trajectory encoder and the image encoder are trained using differentiated training strategies.

[0203] Based on a dual-modal decoder, the differential information between the image features and the trajectory features is applied to fuse the trajectory features and the image features, and handwriting recognition is performed based on the fused features.

[0204] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of handwriting recognition, characterized by, include: The handwritten data to be identified includes the matching stroke trajectory and the handwritten image; Based on the trajectory encoder, the trajectory features of the stroke trajectory are extracted, and based on the image encoder, the image features of the handwritten image are extracted. The trajectory encoder and the image encoder are trained using a differentiated training strategy. The trajectory features are obtained by fusing the first trajectory features of the stroke trajectory with the second image features. The second image features are obtained by sampling the first image features based on the position of each trajectory point in the stroke trajectory in the first image features of the handwritten image. Based on a dual-modal decoder, the image features of the handwritten image are context-encoded to obtain the image context features at the current decoding time. The trajectory features of the stroke trajectory are also context-encoded to obtain the trajectory context features at the current decoding time. Any one of the image context features and trajectory context features is projected onto the feature space of the other feature to obtain the orthogonal feature component of the any one feature that is orthogonal to the other feature. The orthogonal feature component of the any one feature is fused with the other feature, and handwriting recognition is performed based on the fused features.

2. The handwriting recognition method according to claim 1, characterized in that, The steps for obtaining the first trajectory feature of the stroke trajectory and the first image feature of the handwritten image include: Based on the trajectory encoder, the first trajectory feature of the stroke trajectory is extracted, and based on the image encoder, the first image feature of the handwritten image is extracted.

3. The handwriting recognition method according to any one of claims 1-2, characterized in that, The handwritten data also includes non-matching stroke trajectories and / or non-matching handwritten images, and the method further includes: Based on the trajectory encoder, the trajectory features of the non-matching stroke trajectories are extracted, and based on the trajectory decoder, handwriting recognition is performed on the trajectory features of the non-matching stroke trajectories; and / or, Based on the image encoder, image features of the non-matching handwritten image are extracted, and based on the image decoder, handwriting recognition is performed on the image features of the non-matching handwritten image.

4. The handwriting recognition method according to claim 3, characterized in that, The trajectory encoder, the image encoder, the dual-modal decoder, and the single-modal decoder constitute a handwriting recognition model. The single-modal decoder includes the trajectory decoder and / or the image decoder. The steps for obtaining the handwriting recognition model include: Obtain an initial trajectory encoder, an initial image encoder, an initial bimodal decoder, and an initial unimodal decoder, and randomly initialize the parameters of the initial trajectory encoder, wherein the initial image encoder is a pre-trained image text recognition model; Based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

5. The handwriting recognition method according to claim 4, characterized in that, The handwriting recognition model is obtained by iterating the parameters of the handwriting recognition model based on the decoding loss of the initial single-modal decoder and the decoding loss of the initial dual-modal decoder, including: Based on the prediction results of the initial bimodal decoder, distillation learning is performed on the initial unimodal decoder to obtain the distillation loss of the initial unimodal decoder; Based on the distillation loss of the initial single-modal decoder, the decoding loss of the initial single-modal decoder, and the decoding loss of the initial dual-modal decoder, the handwriting recognition model is iterated to obtain the handwriting recognition model.

6. A handwriting recognition device, characterized in that, include: A data determination unit is used to determine the handwritten data to be identified, wherein the handwritten data includes matching stroke trajectories and handwritten images; The feature extraction unit is used to extract trajectory features of the stroke trajectory based on the trajectory encoder and to extract image features of the handwritten image based on the image encoder. The trajectory encoder and the image encoder are trained using a differentiated training strategy. The trajectory features are obtained by fusing the first trajectory features of the stroke trajectory with the second image features. The second image features are obtained by sampling the first image features based on the positions of each trajectory point in the stroke trajectory in the first image features of the handwritten image. The handwriting recognition unit is used to perform context encoding on the image features of the handwritten image based on a dual-modal decoder to obtain the image context features at the current decoding time, perform context encoding on the trajectory features of the stroke trajectory to obtain the trajectory context features at the current decoding time, project any one of the image context features and trajectory context features onto the feature space of the other feature to obtain the orthogonal feature component of the any one feature orthogonal to the other feature, fuse the orthogonal feature component of the any one feature with the other feature, and perform handwriting recognition based on the fused features.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the handwriting recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the handwriting recognition method as described in any one of claims 1 to 5.

Images