Handwritten form identification method and device, and computer readable storage medium

Abstract

The invention discloses a handwritten form recognition method and device and a computer readable storage medium, and the method comprises the steps: obtaining text image data and text track data, and the text image data and the text track data are used for identifying the same to-be-recognized handwritten form content; obtaining a first text feature according to the text image data, and obtaining a second text feature according to the text track data; and obtaining an identification result according to the first text feature and the second text feature.

Description

Handwriting recognition method and device, and computer-readable storage medium

[0001] The embodiments of the present disclosure relate to, but are not limited to, the field of artificial intelligence technology, and in particular, to a handwriting recognition method and device, and a computer-readable storage medium.

[0002] At present, with the rapid development of artificial intelligence and computer technology, the industry has proposed many methods of handwriting recognition. For example, according to the different data forms of the input recognition network, handwriting recognition can be divided into online handwriting recognition and offline handwriting recognition.

[0003] Online handwriting recognition uses the trajectory points collected by the device as input and uses neural networks to extract timing information for handwriting recognition. It can better solve visually ambiguous information, but has strict requirements on the writing order. Offline handwriting recognition uses text images as input, and the recognition results are not affected by pen insertion, modification, etc. during the writing process, but the recognition accuracy for text with visually ambiguous information is poor.

[0004]

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present disclosure provides a handwriting recognition method, comprising: acquiring text image data and text trajectory data, wherein the text image data and text trajectory data are used to identify the same handwriting content to be recognized; obtaining a first text feature according to the text image data, and obtaining a second text feature according to the text trajectory data; and obtaining a recognition result according to the first text feature and the second text feature.

[0007] An embodiment of the present disclosure also provides a handwriting recognition device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the handwriting recognition method described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0008] The embodiments of the present disclosure further provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the handwriting recognition method described in any embodiment of the present disclosure is implemented.

[0009] Other aspects will become apparent upon reading and understanding the drawings and detailed description.

[0010] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shapes and sizes of the components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0011] 1A to 1C are schematic diagrams showing comparison results of some single-modal recognition effects;

[0012] FIG2 is a flow chart of a handwriting recognition method provided by an exemplary embodiment of the present disclosure;

[0013] 3A and 3B are schematic diagrams of convolutional methods of an offline recognition network and an online recognition network, respectively;

[0014] FIG3C is a schematic diagram of a stroke mask of an exemplary embodiment of the present disclosure;

[0015] FIG3D is a schematic diagram of a stroke mask merging result according to an exemplary embodiment of the present disclosure;

[0016] FIG4 is a schematic diagram of a method for mapping collected original trajectory point data into a text image;

[0017] FIG5A is a schematic diagram of a handwriting recognition network structure of an exemplary embodiment of the present disclosure;

[0018] FIG5B is a schematic diagram of the structure of an attention refining module according to an exemplary embodiment of the present disclosure;

[0019] FIG6A is a schematic diagram of a handwriting recognition process provided by an exemplary embodiment of the present disclosure;

[0020] FIG6B is a schematic diagram of another handwriting recognition network structure according to an exemplary embodiment of the present disclosure;

[0021] FIG6C is a schematic diagram of a recurrent neural network model structure of an exemplary embodiment of the present disclosure;

[0022] FIG. 7 is a schematic diagram of the structure of a handwriting recognition device provided by an exemplary embodiment of the present disclosure.

[0023] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present disclosure can be combined with each other arbitrarily without conflict.

[0024] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the ordinary meanings understood by those of ordinary skill in the field to which the present disclosure belongs. The terms "first", "second" and similar terms used in the embodiments of the present disclosure do not denote any order, quantity or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects before the term cover the elements or objects listed after the term and their equivalents, without excluding other elements or objects.

[0025] According to the different data forms of the input recognition network, handwritten character recognition can be divided into online handwritten character recognition (input trajectory point sequence) and offline handwritten character recognition (input image). However, both single-modal recognition networks have their own advantages and disadvantages. Among them, online handwritten character recognition takes the trajectory points collected by the device as input and uses a neural network to extract temporal information for handwritten character recognition, which can better solve visual ambiguity information, but has strict requirements on the writing order; offline handwritten character recognition takes a text image as input, and the recognition result is not affected by backtracking or modification during writing, but the recognition accuracy for text with visual ambiguity information is relatively poor. The above two modalities have obvious complementarity.

[0026] Figures 1A to 1C show schematic diagrams of the comparison results of the recognition effects of some single modalities. In Figure 1A, the ground truth (GT) is: "Business will necessarily be affected. People", the offline handwritten character recognition result is: "Business will be affected. People", and the online handwritten character recognition result is: "Business will necessarily be affected. People"; in Figure 1B, the ground truth is: "Among them, the Chinese business will necessarily be affected, and people have therefore speculated one after another", the offline handwritten character recognition result is: "Among them, the Chinese business will necessarily be affected, and people have therefore speculated one after another", and the online handwritten character recognition result is: "Among them, the Chinese business will necessarily be affected, and people have therefore speculated one after another"; in Figure 1C, the ground truth is: "Protractor; they carefully maintained the fire seed for lighting and heating", the offline handwritten character recognition result is: "Protractor; they carefully maintained the fire seed for heating", and the online handwritten character recognition result is: "Protractor, they carefully maintained the fire seed for lighting and heating".

[0027] The online recognition network that takes trajectory points as input makes full use of the temporal information of the trajectory points and is relatively accurate in recognizing characters with visual ambiguity, such as '照' in Figure 1C, etc. However, the recognition accuracy of punctuation marks is relatively low, and there are strict requirements on the stroke order when the user writes. When there are situations such as backtracking or modification during writing, it is very easy to be misrecognized; the offline recognition network takes an image as the network input, and the recognition performance is not affected by the stroke order when the user writes, but it is easy to misrecognize characters that are easily confused with visual ambiguity.

[0028] In order to combine the advantages of both, the disclosed embodiment designs a multimodal handwriting recognition network based on stroke-level text features, taking strokes as basic feature units, respectively adding stroke masks to the online recognition network and the offline handwriting recognition network, extracting online and offline stroke-level text features, concatenating and fusing the online and offline stroke-level text features and sending them to the decoder, outputting the final recognition result, thereby improving the network recognition performance. Furthermore, in order to improve the problem of feature overlap between different strokes in Chinese characters, the stroke-level text features are further adjusted to stroke group-level text features by integrating strokes in adjacent areas.

[0029] As shown in FIG2 , the present disclosure provides a handwriting recognition method, comprising the following steps:

[0030] Step 201: Acquire text image data and text trajectory data, wherein the text image data and text trajectory data are used to identify the same handwritten content to be recognized;

[0031] Step 202: obtaining a first text feature according to the text image data, and obtaining a second text feature according to the text trajectory data;

[0032] Step 203: Obtain a recognition result according to the first text feature and the second text feature.

[0033] The handwriting recognition method provided by the embodiment of the present disclosure obtains a first text feature based on text image data, obtains a second text feature based on text trajectory data, and obtains a recognition result based on the first text feature and the second text feature. The handwriting recognition method can simultaneously use trajectory information and text image information for handwriting recognition, effectively improving the defects of the online recognition network's recognition performance being heavily dependent on the writing order and the offline recognition network's poor recognition of visually ambiguous text, thereby improving the accuracy and robustness of the handwriting recognition method.

[0034] In some exemplary embodiments, the handwritten content to be recognized may include handwritten characters, graphics, formulas, etc.

[0035] In some exemplary embodiments, the handwritten characters may include at least one of the following: Chinese characters, English characters, numbers, punctuation marks, etc.

[0036] In the embodiment of the present disclosure, the handwritten characters may be Chinese characters, English characters, numbers, punctuation marks, or any other characters, which is not limited in the embodiment of the present disclosure.

[0037] In other exemplary embodiments, the handwritten characters may also include pinyin, operation symbols, and any other special characters.

[0038] In the embodiments of the present disclosure, special characters are symbols that are less frequently used and difficult to input directly, compared to traditional or commonly used symbols. There are many types of special characters. For example, special characters may include: mathematical symbols (e.g., ≈, ≡, ≠, =, ≤, ≥, <, >, etc.), unit symbols (e.g., °C, ﹪, ‰, ㎡, etc.), pinyin characters (for example, ā, á, ǎ, à, ō, ó, ǒ, ò, etc.), etc.

[0039] In some exemplary embodiments, obtaining a first text feature according to text image data and obtaining a second text feature according to text trajectory data includes:

[0040] Generate a first stroke mask according to the text image data, and generate a second stroke mask according to the text trajectory data;

[0041] Inputting text image data into a first encoder to obtain a first feature map; obtaining a first text feature according to the first feature map and a first stroke mask;

[0042] The text trajectory data is input into a second encoder to obtain a second feature map; and a second text feature is obtained according to the second feature map and a second stroke mask.

[0043] In this embodiment, after obtaining the first text feature and the second text feature, the first text feature and the second text feature can be spliced ​​to obtain a multimodal text feature; position coding information is added to the multimodal text feature, and the multimodal text feature with the image position coding information added is input into the decoder to obtain a recognition result.

[0044] In some exemplary embodiments, the dimensions of the first stroke mask may include: (stroke_num, H, W), where stroke_num is the number of strokes included in the text image, H is the height of the text image, and W is the width of the text image;

[0045] The dimension of the second stroke mask may include: (stroke_num, L), where L is the number of trajectory points in the text trajectory data.

[0046] In some exemplary embodiments, the convolution methods of the offline recognition network and the online recognition network under normal circumstances can be shown in Figures 3A and 3B respectively. The offline recognition network uses pixel points as the basic modeling unit, and each convolution operation acts on the pixel points in the adjacent area. The online recognition network uses trajectory points as the basic modeling unit, and each convolution operation acts on the trajectory points in the adjacent area. The two act on different objects and therefore cannot be directly fused. To address this problem, the embodiment of the present disclosure proposes to use stroke-level text features as the basic modeling unit, and obtain stroke masks corresponding to the image data and the trajectory point data based on the original trajectory point data, wherein the image data corresponds to the stroke mask dimension. The stroke mask dimension is as follows: (stroke_num, H, W), where stroke_num is the number of strokes in the current text image, H and W are the height and width of the current text image respectively, the pixel value in the mask corresponding to the area where the strokes are located in the text image is 1, and the rest of the area is 0 (the mask schematic diagram is shown in FIG3C ); similarly, the stroke mask dimension corresponding to the text trajectory data includes: (stroke_num, L), where stroke_num is the number of strokes in the current text trajectory data, L is the number of trajectory points in the current text trajectory data, the pixel value in the mask corresponding to the area where the strokes are located is 1, and the rest of the area is 0. The present disclosure converts the original pixel-level and trajectory point-level text features into stroke-level text features by adding a stroke mask.

[0047] In some exemplary embodiments, the dimensions of the first stroke mask may include: (stroke_group_num, H, W), where stroke_group_num is the number of stroke groups included in the text image, each stroke group includes one or more strokes, H is the height of the text image, and W is the width of the text image;

[0048] The dimension of the second stroke mask may include: (stroke_group_num, L), where L is the number of trajectory points in the text trajectory data.

[0049] In some exemplary embodiments, the method further comprises:

[0050] Calculate the degree of overlap between any two strokes;

[0051] According to the writing order, all strokes are traversed in turn and multiple strokes with overlap degrees higher than a preset threshold are merged into a stroke group.

[0052] In the embodiments of the present disclosure, the overlapping degree of any two strokes represents the distance between the two strokes. When the overlapping degree of two strokes is higher, it indicates that the distance between the two strokes is closer; when the overlapping degree of two strokes is lower, it indicates that the distance between the two strokes is farther. Each stroke in a stroke group includes at least one adjacent stroke, and the overlapping degree between the two adjacent strokes is higher than a preset threshold.

[0053] In some exemplary embodiments, the overlapping degree I of any two strokes can be calculated according to the following formula:

[0054] where (x1min, x1max) is the X-axis coordinate range of one stroke among any two strokes, and (x2min, x2max) is the X-axis coordinate range of the other stroke among any two strokes.

[0055] However, in the embodiments of the present disclosure, the method for calculating the overlapping degree between two strokes is not limited to this, and other methods can also be used to calculate the overlapping degree between any two strokes. For example, it can be calculated according to the Y-axis coordinate ranges of the two strokes or the X-axis coordinate ranges and Y-axis coordinate ranges of the two strokes. The embodiments of the present disclosure do not limit this.

[0056] Considering the particularity of Chinese characters compared to mathematical formulas, etc., common symbols and numbers in mathematical formulas usually have only 1 to 2 strokes. A Chinese character can be composed of multiple strokes, and the positions of multiple strokes are relatively close or have a 'nested' positional relationship. For example, the character '疑' contains a large number of strokes and the positions are relatively close. For an offline recognition network, when passing through the encoder, the receptive field of the same area may cover the information of multiple strokes at the same time, resulting in inaccurate stroke-level features when the output result of the encoder is multiplied by the stroke mask, and serious feature crossover between different strokes. To address this problem, the present disclosure proposes using a stroke group as the basic modeling unit to integrate the strokes in adjacent areas (after integrating the offline stroke mask (i.e., the first stroke mask), the online stroke mask (i.e., the second stroke mask) is adjusted accordingly to ensure the alignment of stroke-level text features in the offline and online modes) to reduce the problem of feature crossover between strokes. Specifically, it is integrated according to the overlapping degree of the X-axis coordinates of different strokes, and the calculation method of the overlapping degree I is as follows:

[0057] where (x1min, x1max) is the X-axis coordinate range of stroke 1, (x2min, x2max) is the X-axis coordinate range of stroke 2, and (the intersection of the X-axis coordinates of the two strokes / the width of the smaller stroke) represents the overlapping degree of stroke 1 and stroke 2.

[0058] During the stroke integration process, strokes with an X-axis coordinate overlap degree I> a preset threshold (exemplarily, the preset threshold may be 0.3) are traversed and merged in sequence according to the user's writing order. An exemplary stroke mask merging example diagram is shown in FIG3D , where the strokes in each rectangular box in the lower left figure of FIG3D are merged, and finally the stroke mask shown in the right figure of FIG3D is obtained after merging. The disclosed embodiment not only improves the problem of feature intersection between strokes, but also reduces the number of strokes by merging strokes in adjacent areas. Corresponding to the network, it reduces the dimension of stroke-level text features (number of stroke groups*1*684), and reduces the amount of network computation.

[0059] In some exemplary embodiments, acquiring text image data and text trajectory data includes:

[0060] Get the original trajectory points;

[0061] Generate text image data and text trajectory data based on the original trajectory points.

[0062] In the disclosed embodiment, the original trajectory point may be trajectory data obtained by a touch module or a camera module, or may be data obtained by smoothing or denoising the trajectory data obtained by the touch module or the camera module.

[0063] In some exemplary embodiments, generating text trajectory data according to original trajectory points includes:

[0064] Get the original trajectory points;

[0065] The image composed of the original trajectory points is scaled in equal proportion so that the height difference between the highest trajectory point and the lowest trajectory point after scaling is equal to the preset height;

[0066] The text trajectory data is generated according to the coordinate information of all trajectory points after scaling and the start and end information of the handwriting.

[0067] In some exemplary embodiments, generating text image data according to original trajectory points may include:

[0068] Get the original trajectory points;

[0069] The image composed of the original trajectory points is scaled in equal proportion so that the height difference between the highest trajectory point and the lowest trajectory point after scaling is equal to the preset height;

[0070] All trajectory points in the same stroke are connected with lines of preset line width to obtain text image data.

[0071] The user writes on the touch screen device to collect trajectory point data (the information recorded in the trajectory point data includes the current trajectory point X-axis coordinates, Y-axis coordinates, and whether the pen is lifted - (x, y, isLeave)), and the trajectory points are converted into text images through trajectory point mapping and then sent to the offline handwriting recognition network.

[0072] There are obvious differences in font sizes when different users or the same user write at different times. In order to make the network converge quickly during training, consider adjusting all data to the same font size and the same text line width. To this end, the present disclosure uses trajectory point mapping to uniformly convert the collected trajectory points into text images with fixed height and variable width, and the line width of the written content in the text image is fixed. As shown in Figure 4, the collected original trajectory points are mapped to a text image with a fixed height (exemplarily, the fixed height can be 80 pixels) and variable width, and then a line with a preset line width (exemplarily, the preset line width can be 2 pixels) is used to connect the trajectory points in each stroke.

[0073] In some exemplary embodiments, the first encoder may be a dense convolutional network (DenseNet), and obtaining the first text feature according to the first feature map and the first stroke mask may include:

[0074] Downsampling the first stroke mask to obtain a first downsampled stroke mask;

[0075] The first feature map F pixel With the first downsampled stroke mask M offline Multiply by the following formula to get the first text feature:

[0076] Among them, sum(M offline ) is the number of the first stroke mask.

[0077] In the offline recognition process, the offline recognition network inputs the text image to be recognized with a dimension of H*W*1, and after passing through the encoder DenseNet98, it outputs a feature map F with a dimension of H / 16*W / 16*684 pixel , the original offline stroke mask M offline The dimension is H*W*stroke_num (stroke_num is the number of strokes, specifically the number of stroke groups after the above stroke integration). First, the stroke mask is downsampled by 16 times to obtain a mask of stroke_num*H / 16*W / 16 dimensions. The downsampled stroke mask is multiplied by the encoder output feature map according to the following formula to obtain the offline stroke (group) level text feature F stroke :

[0078] F stroke The dimension is stroke_num*1*684.

[0079] In some exemplary embodiments, the data of each track point in the text track data may include an X-axis coordinate, a Y-axis coordinate, a start flag, and a pen lift flag.

[0080] The disclosed embodiment converts the original collected trajectory point data into a data format containing four features: x, y, start_state, and leave_state, where x and y are the XY coordinates of the trajectory point, respectively, start_state is the starting flag (the start_state of the starting trajectory point in each stroke is 1, and the rest are 0), and leave_state is the flag for whether the pen is lifted (the leave_state of the last trajectory point in each stroke is 1, and the rest are 0). Analogous to the input data format of the offline recognition network (height, width, number of channels), the 4-dimensional features in the online trajectory point data are regarded as the number of channels, the number of trajectory points is regarded as the width, and the height is regarded as 1, so that it can be used as the input of the encoder DenseNet98. Since the data height of the input online recognition network is 1, the 3*3 convolution in DenseNet98 is replaced by 1*3 convolution, the 7*7 convolution is replaced by 1*3 convolution, and the 2*2 pooling is replaced by 1*2 pooling. The network structure of other parts is the same as the above-mentioned offline recognition network.

[0081] In some exemplary embodiments, the second encoder may be a dense convolutional network (DenseNet), and obtaining the second text feature according to the second feature map and the second stroke mask may include:

[0082] Downsampling the second stroke mask to obtain a second downsampled stroke mask;

[0083] The second feature map F′ pixel With the second downsampled stroke mask M online Multiply by the following formula to get the second text feature:

[0084] Among them, sum(M online ) is the number of the second stroke mask.

[0085] During the online recognition process, the online recognition network inputs the trajectory point data to be recognized with a dimension of 1*L*4, and after passing through the encoder DenseNet98, it outputs a feature map F′ with a dimension of 1*L / 16*684. pixel , the original online stroke mask M onlineThe dimension is stroke_num*L (stroke_num is the number of strokes, specifically the number of stroke groups after the above stroke integration). First, the stroke mask is downsampled by 16 times to obtain a mask of stroke_num*L / 16 dimensions. The downsampled stroke mask is multiplied with the encoder output feature map according to the following formula to obtain the online stroke (group) level text feature F′ stroke :

[0086] F′ stroke The dimension is stroke_num*1*684.

[0087] In some exemplary embodiments, obtaining a recognition result based on a multimodal text feature may include: adding image position encoding information to the multimodal text feature, inputting the multimodal text feature with the image position encoding information into a decoder to obtain a recognition result, wherein the decoder is a Transformer decoder that adds an attention refinement module to multi-head attention.

[0088] The stroke-level text features (dimension: stroke_num*1*684) obtained after offline recognition network coding are concat-fused with the stroke-level text features (dimension: stroke_num*1*684) obtained after online recognition network coding to obtain a multimodal stroke-level text feature with a dimension of stroke_num*2*684. After that, the position coding is added in the above manner and sent to the decoder for decoding to output the final recognition result. The multimodal recognition network structure is shown in Figure 5A. The overall handwriting recognition network structure adopts an encoding-decoding structure. The input handwritten text image is encoded by DenseNet98_offline to obtain a high-dimensional vector representation of the text image. The input trajectory point data is encoded by DenseNet98_online to obtain a high-dimensional vector representation of the trajectory point data. The output feature maps of the two types of encoders are spliced ​​and the position coding (sine coding) information is added, and sent to the subsequent Transformer-based decoder for decoding.

[0089] Unlike RNN, which parses sequentially and contains position information between different elements, the Transformer architecture abandons the loop mechanism and only uses a multi-head self-attention mechanism. The model itself does not have any position information for each element. Therefore, it is necessary to integrate the order of elements into the model through position encoding. The calculation method of sinusoidal position encoding is as follows:

[0090] Among them, pos represents the position of the pixel to be encoded in the image (pos_x and pos_y are calculated respectively and then Cat-connected after position encoding), i represents the position encoding dimension index, and dmodel represents the position encoding dimension (when position encoding a two-dimensional image, dmodel is usually equal to (the number of channels of the feature map to be encoded / 2)).

[0091] The position encoding result is added to the encoder output feature map and then sent to the decoder for decoding. The decoder adopts a Transformer-based decoding structure. Different from the traditional Transformer decoder, referring to the idea of ​​Coverage attention, as shown in Figure 5B, an Attention Refinement Module (ARM) is added to the multi-head attention to solve the "over-parsing" and "under-parsing" problems in the text content parsing process ("over-parsing" - some parts of the image are unnecessarily parsed multiple times, and "under-parsing" - some areas are not parsed), so that the Transformer will allocate more attention to the areas that have not yet been parsed.

[0092] The handwriting recognition method of the embodiment of the present disclosure designs a multimodal recognition network based on stroke-level text features to address the inherent defects of single modalities (online handwriting recognition, offline handwriting recognition), fully combining the advantages of the two single-modal recognition methods and improving the recognition performance of the network; in addition, the handwriting recognition method of the embodiment of the present disclosure, in view of the overlap of features between different strokes of Chinese characters during the convolution process, proposes to merge strokes in adjacent areas and use stroke group-level text features as basic feature units.

[0093] The disclosed embodiment also proposes a method for handwriting recognition based on static input (picture) and dynamic input (real-time coordinate point information), wherein the static input picture is a picture of the writing trajectory, and the dynamic input coordinates are the trajectory information of real-time writing, generally including XY axis coordinate information, timestamp, pen start and pen lift information. The handwriting recognition method can use trajectory information and imaging information for handwriting recognition at the same time, solving the problems of character duplication and segmentation errors caused by the lack of semantic information in the image-based handwriting recognition method, and the problem of delayed strokes mistakenly recognized by the coordinate information-based handwriting recognition method.

[0094] As shown in FIG6A , when a user writes on an electronic screen, such as writing Chinese and English characters, drawing, or writing formulas, the trajectory information formed by the touch screen and the formed image information can be sent to the handwriting recognition method in real time after the writing is completed. The handwriting recognition method uses these two inputs to perform handwriting recognition, and the recognition result is a character in a label format, such as “BOE”, “acute triangle (figure)”, “x(a–1)=6t_{2}”, etc.

[0095] In the disclosed embodiment, the image input by the handwriting recognition method can be a three-channel image or a single-channel image; the input trajectory information can include XY axis coordinate information, handwriting start and end information, etc. The handwriting start and end information can be represented by coding, such as (0, 0) represents the middle writing point, (1, 0) represents the initial point of the pen, and (0, 1) represents the last point before the pen is lifted. In this way, a continuous 4-dimensional point sequence can be sent to the handwriting recognition model for handwriting recognition.

[0096] In some other exemplary embodiments, as shown in FIG6B , obtaining a first text feature according to text image data, obtaining a second text feature according to text trajectory data, and obtaining a recognition result according to the first text feature and the second text feature include:

[0097] Inputting the text image data into a first encoder to obtain a first text feature;

[0098] adding image position coding information to the first text feature, inputting the first text feature after adding the image position coding information and the random number into the first decoder to obtain a first target output;

[0099] Inputting the text trajectory data into a recurrent neural network (RNN) model to obtain a first prediction value, inputting the first prediction value into a first part of a second decoder to obtain a second text feature, wherein the first part of the second decoder includes a character position encoding layer and a self-attention layer;

[0100] Inputting the second text feature and the first text feature after adding the image position encoding information into a second part of the second decoder to obtain a second target output, the second part of the second decoder including other layers except the character position encoding layer and the self-attention layer;

[0101] splicing the first target output and the second target output to obtain a spliced ​​target;

[0102] The concatenated target is input into the linear layer and the softmax layer to obtain the recognition result.

[0103] In some exemplary embodiments, the first encoder may be a dense convolutional network DenseNet. Compared with ResNet, the dense connections of DenseNet on different scale feature maps can better reflect the scale features of characters of different sizes, which is beneficial for the subsequent decoding of the meanings of characters of different sizes at different positions. In order to align the output features of DenseNet with the size of the decoder model, the embodiment of the present disclosure adds a 1x 1 convolution layer at the end of the first encoder to obtain the output image features Among them, h0 and w0 represent the size of the input image, d modelRepresents the encoding dimension of the first encoder.

[0104] Since there is no spatial position relationship between the tokens (input) of the Transformer decoder, additional position information is necessary. Image position encoding information can be encoded using two-dimensional normalized position encoding. Since the model needs to focus on relative positions, the position coordinates must be normalized first. Given a two-dimensional coordinate tuple (x, y), the encoding dimension is d, and the two-dimensional image position encoding is calculated by concatenating the one-dimensional positions. The relevant calculation formula is as follows:

[0105] In some exemplary embodiments, both the first decoder and the second decoder may be Transformer decoders to which an Attention Refinement Module (ARM) is added in the multi-head attention.

[0106] According to actual tests, the Transformer decoder structure with the Attention Refinement Module (ARM) added to the multi-head attention has a good recognition effect on Chinese and English recognition, but the problem of repeated characters still occurs. As shown in Figure 6B, the lower left part is the trajectory data part processed by the RNN model, and the middle and right parts process the input text image part. The handwriting recognition method proposed in the present disclosure can simultaneously use trajectory data information and imaging information for handwriting recognition, solve the problems of character duplication and segmentation errors caused by the lack of semantic information based on the image recognition method, and solve the problem of recognition errors of delayed strokes based on coordinate information, thereby improving the accuracy and robustness of the algorithm. The present disclosure integrates Chinese and English data, formulas, trajectories of graphic tasks, and sample data, and reselects the recognition category according to the actual situation, and finally trains a comprehensive model.

[0107] As shown in Figure 6B, the role of the decoder (including the first decoder and the second decoder) is to convert features into target outputs. The right branch performs position encoding on the input characters (GT or random numbers) (forming multi-dimensional features) and performs a self-attention operation. The ultimate goal is to use it as a query vector in the scaled dot product attention layer and multiply it with the key vector (Key) output by the Encoder layer. The role is to correlate and match the character features on the image with all the input characters, calculate the correlation (also called Attention) to obtain the corresponding features, and then input these features into the subsequent layers to obtain the target output.

[0108] Similarly, after the RNN model on the left predicts the character output through the input coordinate information, the predicted character is input into the second decoder of the left branch, which is also to calculate the correlation between the image and the character (predicted by the coordinate information) by multiplying the query vector and the key value vector, thereby using the coordinate information to assist character recognition. By adding coordinate information to assist character recognition, the disclosed embodiment can solve the problem of repeated character recognition generated when image information is inferred alone.

[0109] For word positional encoding, we can use the 1D positional encoding introduced in Transformer. Given the encoding dimension d, position p, and feature dimension index i, the word position encoding vector It can be expressed as:

[0110] After the introduction of the self-attention layer, the algorithm can more easily capture the long-distance interdependent features in the sentence. For Self-Attention, the input is multiplied by the weight array to obtain three variables: query (Query, Q) vector, key value (Key, K) vector and value (Value, V) vector, and then the following formula is implemented:

[0111] The scale dot product attention layer is part of the multi-head-attention layer, which performs matrix multiplication of the query vector and the key vector and then scales them.

[0112] The Attention Refining Module (ARM) can refine the attention item E. It uses the dot product matrix in the Transformer as the attention item. The refined item matrix R needs to be calculated from the attention weight A after Softmax, and the attention weight A is used to provide historical alignment information. Define a As input, the output is a refined matrix The function φ is:

[0113] Among them, α t is the attention weight at time step t∈[0,T), represents a convolution kernel, * represents the convolution operation, b c is a bias term, is a linear projection matrix. The function φ can extract local coverage features to detect the edges of resolved regions and identify incoming unresolved regions, and refine the attention term E by subtracting the refinement term R.

[0114] The matrix multiplication (MatMul) layer is used to multiply the Value matrix with the (Query*Key) matrix processed by Attention.

[0115] The Feed-Forward Network (FFN) layer includes two fully connected layers and a nonlinear activation function ReLu, which enhances the expressive power of the model through nonlinear functions.

[0116] In some exemplary embodiments, as shown in FIG. 6C , the recurrent neural network model may include n layers of long short-term memory (LSTM) neural network layers, n layers of dropout layers and linear layers, n≥1, and the LSTM layers and the dropout layers are alternately arranged, and the linear layer is connected to the last dropout layer.

[0117] In some exemplary embodiments, the recurrent neural network model may include n layers of gated recurrent unit (GRU) layers, n layers of random dropout layers and linear layers, n≥1, and the GRU layers and the random dropout layers are alternately arranged, and the linear layer is connected to the last random dropout layer.

[0118] The input of the recurrent neural network model is the coordinate trajectory information (processed as four-dimensional data input). The coordinate trajectory information is output through the recurrent neural network model mainly composed of a bidirectional LSTM layer, a dropout layer, and a linear layer to output the prediction result (Predicted labels). The RNN model shown in Figure 6C mainly uses three LSTM layers for recognition. Depending on the difficulty of the task, the LSTM layer can be increased or replaced with a GRU layer.

[0119] As shown in FIG6B , the obtained first target output and the second target output are concatenated (concat), and concat is to concatenate several (two or more) arrays according to the specified dimension. For example, A is a 2x3-dimensional array, and B is a 4x3-dimensional array. A and B are concatenated according to dimension 1 (i.e., row concatenation), and the dimension of the obtained new array is 6x3; if concatenated according to dimension 2, the original arrays need to be consistent in length in other dimensions, such as A can be converted into a 4x3 array by repetition, and then concatenated with B in dimension 2 to obtain a 4x6 array. In the disclosed embodiment, the arrays output from the left and right sides (i.e., the first target output and the second target output) are consistent in all dimensions, so they can be concatenated in the required dimensions (such as 1 and 2). For example, they can be concatenated in dimension 2, and the original dimensions are (b, l, d), which become (b, l, 2d) after concatenation.

[0120] In some exemplary embodiments, during training, the loss function established is Loss = α × L CTC +β×L CE , where L CTC is the connectionist temporal classifier loss of the recognition result, L CE is the cross entropy loss of the first prediction value, α and β are preset weight coefficients, and α+β=1. For example, α=0.8, β=0.2, however, the embodiment of the present disclosure is not limited thereto.

[0121] The RNN model combines the semantic information of the handwritten coordinates to output a prediction result (first prediction value). When training the model, a Loss2 is calculated here to train and update the RNN model. The output of the RNN model can be regarded as an output label with semantic information. It is transformed with the first text feature after adding the image position encoding information to obtain the second target output, and then spliced ​​with the first target output obtained by only using the first text feature after adding the image position encoding information to obtain a reliable and information-rich spliced ​​target. The spliced ​​target passes through the linear layer and the softmax layer to obtain the overall recognition result. A Loss1 is calculated here. The final Loss is Loss = αLoss1 + βLoss2. The model updates the parameters through back propagation to obtain a more accurate model. Among them, the calculation of Loss1 can use the CTCloss loss function, and the calculation of Loss2 can use the CE cross entropy loss.

[0122] In the model training phase, data is prepared first, including about 50,000 pieces of data including coordinate tracks, handwriting images, and writing labels; during training, the text track data is input into the left branch, the text image data is input into the middle branch, and the sample labels are input into the right branch for training. In order to converge as quickly as possible, the RNN model part can load a pre-trained model. During testing, the text track data is input into the left branch, the text image data is input into the middle branch, and the random number is input into the right branch.

[0123] The handwriting recognition method of the disclosed embodiment can use trajectory information and imaging information to perform handwriting recognition at the same time, solving the problems of character duplication and segmentation errors caused by the lack of semantic information in the image-based handwriting recognition method, and the problem of delayed strokes in the coordinate information-based handwriting recognition method. The handwriting recognition method can be applied to application scenarios such as electronic examination paper grading (Chinese characters + numbers + formulas, etc.), note beautification and archiving.

[0124] An embodiment of the present disclosure also provides a handwriting recognition device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the handwriting recognition method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0125] As shown in FIG7 , in one example, the handwriting recognition device may include: a processor 710, a memory 720, a bus system 730, and a transceiver 740, wherein the processor 710, the memory 720, and the transceiver 740 are connected via the bus system 730, the memory 720 is used to store instructions, and the processor 710 is used to execute the instructions stored in the memory 720 to control the transceiver 740 to send and receive signals. Specifically, the transceiver 740 may obtain text image data and text trajectory data under the control of the processor 710, the text image data and the text trajectory data are used to identify the same handwriting content to be recognized, the processor 710 obtains a first text feature according to the text image data, and obtains a second text feature according to the text trajectory data; obtains a recognition result according to the first text feature and the second text feature; and outputs the recognition result to the text input interface through the transceiver 740.

[0126] It should be understood that the processor 710 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] The memory 720 may include a read-only memory and a random access memory, and provides instructions and data to the processor 710. A portion of the memory 720 may also include a nonvolatile random access memory. For example, the memory 720 may also store information on the device type.

[0128] In addition to the data bus, the bus system 730 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 730 in FIG. 7 .

[0129] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor 710 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 720, and the processor 710 reads the information in the memory 720 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0130] The embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the handwriting recognition method described in any embodiment of the present disclosure is implemented. The method of driving handwriting recognition by executing executable instructions is basically the same as the handwriting recognition method provided in the above embodiment of the present disclosure, and will not be described in detail here.

[0131] In some possible implementations, various aspects of the handwriting recognition method provided by the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the handwriting recognition method according to various exemplary embodiments of the present disclosure described above in this specification. For example, the computer device may execute the handwriting recognition method recorded in the embodiments of the present disclosure.

[0132] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0133] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0134] It should be noted that the above embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to the contents specifically shown and described herein. Various modifications, substitutions or omissions may be made to the forms and details of implementation without departing from the scope of the present disclosure.

Claims

A handwriting recognition method, comprising: Acquire text image data and text trajectory data, wherein the text image data and text trajectory data are used to identify the same handwritten content to be recognized; A first text feature is obtained according to the text image data, and a second text feature is obtained according to the text trajectory data; and a recognition result is obtained according to the first text feature and the second text feature. The handwriting recognition method according to claim 1, wherein: Obtaining a first text feature according to the text image data, and obtaining a second text feature according to the text trajectory data, includes: generating a first stroke mask according to the text image data, and generating a second stroke mask according to the text trajectory data; inputting the text image data into a first encoder to obtain a first feature map; obtaining a first text feature according to the first feature map and the first stroke mask; inputting the text trajectory data into a second encoder to obtain a second feature map; and obtaining a second text feature according to the second feature map and the second stroke mask. The handwriting recognition method according to claim 2, wherein: The dimensions of the first stroke mask include: (stroke_num, H, W), where stroke_num is the number of strokes contained in the text image in the text image data, H is the height of the text image in the text image data, and W is the width of the text image in the text image data; the dimensions of the second stroke mask include: (stroke_num, L), where L is the number of trajectory points in the text trajectory data. The handwriting recognition method according to claim 2, wherein: The dimensions of the first stroke mask include: (stroke_group_num, H, W), where stroke_group_num is the number of stroke groups contained in the text image in the text image data, each stroke group includes one or more strokes, H is the height of the text image in the text image data, and W is the width of the text image in the text image data; the dimensions of the second stroke mask include: (stroke_group_num, L), where L is the number of trajectory points in the text trajectory data. The handwriting recognition method according to claim 4, further comprising: Calculate the degree of overlap between any two strokes; According to the writing order, all strokes are traversed in turn and multiple strokes with a degree of overlap greater than a preset threshold are merged into a stroke group. The handwriting recognition method according to claim 2, wherein: The acquiring of text image data and text trajectory data comprises: acquiring original trajectory points; and generating the text image data and text trajectory data according to the original trajectory points. The handwriting recognition method according to claim 2, wherein: The first encoder is a dense convolutional network DenseNet, and obtaining the first text feature according to the first feature map and the first stroke mask includes: downsampling the first stroke mask to obtain a first downsampled stroke mask; pixel With the first downsampled stroke mask M offline The first text feature is obtained by multiplying by the following formula: Among them, sum(M offline ) is the number of the first stroke masks. The handwriting recognition method according to claim 2, wherein: The data of each track point in the text track data includes an X-axis coordinate, a Y-axis coordinate, a start flag, and a pen lift flag. The handwriting recognition method according to claim 2, wherein: The second encoder is a dense convolutional network DenseNet, and the second text feature is obtained according to the second feature map and the second stroke mask, including: downsampling the second stroke mask to obtain a second downsampled stroke mask; pixel With the second downsampled stroke mask M online The second text feature is obtained by multiplying by the following formula: Among them, sum(M online ) is the number of the second stroke masks. The handwriting recognition method according to claim 2, wherein: According to the first text feature and the second text feature, a recognition result is obtained, including: concatenating the first text feature and the second text feature to obtain a multimodal text feature; adding image position coding information to the multimodal text feature; inputting the multimodal text feature after adding the image position coding information into a decoder to obtain a recognition result, wherein the decoder is a Transformer decoder that adds an attention refinement module to the multi-head attention. The handwriting recognition method according to claim 1, wherein: A first text feature is obtained according to the text image data, a second text feature is obtained according to the text trajectory data, and a recognition result is obtained according to the first text feature and the second text feature, including: inputting the text image data into a first encoder to obtain a first text feature; adding image position coding information to the first text feature, inputting the first text feature after adding the image position coding information and a random number into a first decoder to obtain a first target output; inputting the text trajectory data into a recurrent neural network model to obtain a first prediction value, inputting the first prediction value into a first part of a second decoder to obtain a second text feature, the first part of the second decoder including a character position coding layer and a self-attention layer; inputting the second text feature and the first text feature after adding the image position coding information into a second part of the second decoder to obtain a second target output, the second part of the second decoder including other layers except the character position coding layer and the self-attention layer; splicing the first target output and the second target output to obtain a spliced ​​target; inputting the spliced ​​target into a linear layer and a softmax layer to obtain a recognition result. The handwriting recognition method according to claim 11, wherein: Both the first decoder and the second decoder are Transformer decoders that add an attention refinement module to the multi-head attention. The handwriting recognition method according to claim 11, wherein: The recurrent neural network model includes n LSTM layers, n random inactivation layers and linear layers, n≥1, and the LSTM layers and random inactivation layers are alternately arranged, and the linear layer is connected to the last random inactivation layer. The handwriting recognition method according to claim 11, wherein: The recurrent neural network model includes n GRU layers, n random inactivation layers and linear layers, n≥1, and the GRU layers and random inactivation layers are alternately arranged, and the linear layer is connected to the last random inactivation layer. The handwriting recognition method according to claim 11, wherein: During training, the loss function established is Loss = α × L CTC +β×L CE , where L CTC is the connectionist temporal classifier loss of the recognition result, L CE is the cross entropy loss of the first predicted value, α and β are preset weight coefficients, and α+β=1. A handwriting recognition device comprises a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the handwriting recognition method according to any one of claims 1 to 15 based on the instructions stored in the memory. A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the handwriting recognition method according to any one of claims 1 to 15.

Images