A Chinese handwritten text line recognition method based on visual language joint reasoning

Abstract

The application discloses a Chinese handwritten text line recognition method based on visual language joint reasoning, and establishes a Chinese handwritten text line recognition model based on visual language joint reasoning, adopts multi-modal attention to fuse visual features and text language features, and guides the model to learn to recognize each character by combining visual features under different semantic contexts through random sampling attention mask of a sliding window in a training process, so that a preliminary recognition result is obtained, and the decoding module of the model is fed with the preliminary recognition result again to perform iteration according to global context, so that a corrected result is obtained. The application realizes visual reasoning and language context reasoning simultaneously by using an end-to-end algorithm, and has the ability of iterative error correction, so that the application does not need to depend on an external language model, and the recognition accuracy of Chinese handwritten text lines is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of Chinese handwritten character recognition technology, and in particular to a method for recognizing Chinese handwritten text lines based on visual-language joint reasoning. Background Technology

[0002] Chinese handwritten character recognition has significant practical value, applicable to numerous applications such as automatic digitization of ancient documents and automated grading. However, the vast number of Chinese character categories, numerous similar-looking characters, and messy handwriting make Chinese handwritten character recognition very challenging. Currently, most advanced Chinese handwritten character recognition methods use a separate visual reasoning model for initial recognition, followed by correction using an external language model. The visual and language reasoning models are independent, but their drawbacks are: the language model not only increases the model's complexity but also only considers linguistic features, neglecting visual information. This makes it unsuitable for texts outside the corpus or lacking inherent linguistic features, potentially leading to incorrect interpretations. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention provides a method for recognizing Chinese handwritten text lines based on visual-language joint reasoning, aiming to simultaneously utilize visual information and linguistic contextual information for reasoning, thereby improving the accuracy of Chinese handwritten text recognition.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] The present invention provides a method for recognizing Chinese handwritten text lines based on visual-language joint reasoning, characterized by the following steps:

[0006] Step 1: Collect a set of images of Chinese handwritten text lines X = {X1, X2, ..., X...} i ,…,X n} and its corresponding text label set Y = {Y1,Y2,…,Y} i ,…,Y n} and used as the training sample set, where, Y represents the i-th training image sample. i Let X represent the i-th training sample image. i The text label, and Y i ={y i1 ,y i2 ,…,y it ,…,y iT}, y it Indicates text label Y i The t-th character in the image; T represents the total number of characters, n represents the number of sample images; H represents the height of the sample image, and W represents the width of the sample image;

[0007] Step 2: build a Chinese handwritten text line recognition network based on multi-modal attention, including: a visual feature encoding module, a multi-modal feature interaction decoding module and a text category prediction module;

[0008] Step 2.1: the visual feature encoding module includes a convolutional neural network, a position encoding layer and a self-attention layer;

[0009] Step 2.1.1: the convolutional neural network adopts a ResNet structure and sequentially includes 4 convolutional blocks and their corresponding maximum pooling layers, wherein each convolutional block includes a plurality of residual blocks;

[0010] input the i-th training text image into the Chinese handwritten text line recognition network, and sequentially perform convolution and down-sampling operations in the convolutional neural network of the visual feature encoding module, thereby outputting the i-th convolutional feature d represents the dimension of the convolutional feature;

[0011] Step 2.1.2: the position encoding layer includes a horizontal position encoding layer and a vertical direction position encoding layer, which respectively adopt sine and cosine position encoding methods, and the two are weighted and summed;

[0012] input the i-th convolutional feature CNN i into the position encoding layer, and respectively perform sine position encoding processing in the horizontal position encoding layer and cosine position encoding processing in the vertical direction position encoding layer, and then perform weighted summation on the obtained results, and expand into the i-th visual feature sequence

[0013] Step 2.1.3: the self-attention layer includes one multi-head self-attention layer and one feedforward network layer;

[0014] input the i-th visual feature sequence Sequence i into the self-attention layer for processing, and output the i-th visual encoding feature map

[0015] Step 2.2: the multi-modal feature interaction decoding module sequentially includes a character embedding layer, a second position encoding layer, two multi-head attention layers and a feedforward network layer connected by a residual connection;

[0016] Step 2.2.1: input the text label Y i into the Chinese handwritten text line recognition network, and perform mapping processing in the character embedding layer of the multi-modal feature interaction decoding module to obtain the i-th text vector ​

[0017] Step 2.2.2: the text label Y i is indexed by {1, 2, …, T} through the processing of the second position encoding layer, and the i-th position vector is obtained where p it represents the position vector of the t-th character of the text label Y i ;

[0018] Step 2.2.3: after adding C i and P i , the i-th fusion feature map is obtained where c it represents the fusion feature vector of the t-th character of the text label Y i ;

[0019] Step 2.2.4: define the attention mask matrix and take P i as the query vector query of the first multi-head attention layer, and take Context i as the key vector key and the value vector value of the first multi-head attention layer, so as to obtain the position context feature map output by the first multi-head attention layer by using formula (1) where d 1it represents the position context feature vector of the t-th position;

[0020]

[0021] In formula (1),'represents the matrix transposition operation;

[0022] Step 2.2.5: take D 1i as the query vector query of the second multi-head attention layer, and take V i as the key vector key and the value vector value of the second multi-head attention layer, so as to obtain the position visual semantic feature map D 2i output by the second multi-head attention layer by using formula (2)

[0023]

[0024] Step 2.2.6: the position visual semantic feature map D 2i is input into the feedforward network layer, and after being processed by the first linear layer, a relu layer and the second linear layer in turn, the decoding feature map D 3i output by the decoder is obtained;

[0025] Step 2.3: the text category prediction module comprises a fully connected layer and a softmax layer;

[0026] decoded feature map D 3i The predicted value of the i-th training text image X i is obtained after the text category prediction module is input and sequentially processed by the full connection layer and the softmax layer. wherein, represents the predicted character of the t-th position of the i-th training text image X i .

[0027] Step 4: according to and Y i , a cross-entropy loss function is constructed, and the Chinese handwritten text line recognition network is trained by using the gradient descent method until the cross-entropy loss function converges, so as to obtain the trained Chinese handwritten text line recognition model Model.

[0028] Step 5: inputting the image to be recognized X test into the model Model for processing to obtain a preliminary prediction result The preliminary prediction result is input again into the modal feature interaction decoding module of the model Model for processing, and sequentially processed by the text category prediction module to obtain a corrected recognition result

[0029] The Chinese handwritten text line recognition method based on visual language joint reasoning has the characteristics that the attention mask matrix M in step 2.2.4 is divided into a left-to-right mask a right-to-left mask and a probability random mask based on a sliding window. wherein, represents the value of the m-th row and the n-th column of the mask matrix M1, and represents the value of the m-th row and the n-th column of the mask matrix M2, and represents the value of the m-th row and the n-th column of the mask matrix M3, and s represents the size of the window, and random(0,1) represents a random function of 0 or 1.

[0030] In the training of the Chinese handwritten text line recognition network by using the gradient descent method, M1, M2 and M3 are respectively brought into formula (1) in each training process to obtain three position context feature maps output by the first multi-head attention layer and

[0031] the three position context feature maps and The subsequent multi-modal feature interaction decoding module and text category prediction module are input respectively for processing, so that three prediction values are obtained; three different cross-entropy loss functions are established in step 4, and the average value is taken as the final total loss function for model training.

[0032] In the recognition process of step 5, the left-to-right mask M1 is used to obtain a preliminary prediction result The preliminary prediction result is further input into the multi-modal feature interaction decoding module And the attention mask is set in the first multi-head attention layer So that the final recognition result is obtained from the text category prediction module Wherein, M4 represents the value of the mth row and the nth column of the mask matrix M4, and

[0033] Compared with the prior art, the present application has the following advantages:

[0034] 1. The Chinese handwritten text line recognition model is designed, the multi-modal attention layer is used in the decoding module, the features of two modalities of vision and language are fused, three attention masks are designed, including left-to-right, right-to-left and sliding window random sampling masks, the model is guided to combine the visual features to recognize each character under the condition of different semantic context, and the model has the ability of visual reasoning and visual language joint reasoning.

[0035] 2. The three attention masks are designed, including left-to-right, right-to-left and sliding window random sampling masks, the model is guided to combine the visual features to recognize each character under the condition of different degrees of bilateral context, and the limitation that the existing autoregressive method can only use unilateral context is overcome. The former two let the model learn to recognize characters according to the unilateral context of the left or right side, the sliding window mechanism in the third mask controls the model to only fuse the context features near the current position when predicting the character at the current position, and discards irrelevant context information far away from the position, which can reduce the proportion of irrelevant context. The random sampling mechanism in the third mask randomly samples different numbers of characters as context in the window. The model learns to decode jointly with the visual features under the conditions of no context, partial context and full context, so that the network learns pure visual reasoning and visual language reasoning under different degrees of context, further improves the language reasoning ability of the model, and makes the model have the ability of pure visual parallel reasoning and the ability of complete form filling iterative error correction.

[0036] 3. The decoding module designed in this invention has an iterative error correction function. The preliminary recognition result can be sent back to the decoding module of the model for iterative error correction based on the global context to obtain the corrected result, thereby further improving the accuracy of Chinese handwritten character recognition. Attached Figure Description

[0037] Figure 1 This is a framework diagram of a Chinese handwritten text line recognition method based on visual-language joint reasoning according to the present invention;

[0038] Figure 2 This is a schematic diagram of the random sampling attention mask for randomly selecting the context in the sliding window of the present invention;

[0039] Figure 3a This is a schematic diagram of the first type of left-to-right attention mask used in the training process of this invention;

[0040] Figure 3b This is a schematic diagram of the second type of right-to-left attention mask used in the training process of this invention.

[0041] Figure 3c This is a schematic diagram of the third type of sliding window random sampling attention mask used in the training process of this invention.

[0042] Figure 4 This is a schematic diagram of the testing process of this invention. Detailed Implementation

[0043] In this embodiment, a method for recognizing Chinese handwritten text lines based on visual-language joint reasoning is described, such as... Figure 1 As shown, it includes the following steps:

[0044] Step 1: Collect a set of images of Chinese handwritten text lines X = {X1, X2, ..., X...} i ,…,X n} and its corresponding text label set Y = {Y1,Y2,…,Y} i ,…,Y n} and used as the training sample set, where, Y represents the i-th training image sample. i Let X represent the i-th training sample image. i The text label, and Y i ={y i1 ,y i2 ,…,y it ,…,y iT}, y it Indicates text label Y i The t-th character in the image; T represents the total number of characters, n represents the number of sample images; H represents the height of the sample image, and W represents the width of the sample image;

[0045] Step 2: build a Chinese handwritten text line recognition network based on multi-modal attention, including: a visual feature encoding module, a multi-modal feature interaction decoding module and a text category prediction module;

[0046] Step 2.1: the visual feature encoding module includes a convolutional neural network, a position encoding layer and a self-attention layer;

[0047] Step 2.1.1: the convolutional neural network adopts a ResNet structure and sequentially includes 4 convolutional blocks and corresponding max-pooling layers, wherein each convolutional block includes a plurality of residual blocks; the convolutional neural network is used to extract local features of an image.

[0048] The i-th training text image is input into the Chinese handwritten text line recognition network and sequentially undergoes convolution and down-sampling operations in the convolutional neural network of the visual feature encoding module, thereby outputting the i-th convolutional feature d represents the dimension of the convolutional feature;

[0049] Step 2.1.2: the position encoding layer includes a horizontal position encoding layer and a vertical direction position encoding layer, which respectively adopt sine and cosine position encoding methods, and the two are weighted and summed;

[0050] The i-th convolutional feature CNN i is input into the position encoding layer and respectively undergoes sine position encoding processing of the horizontal position encoding layer and cosine position encoding processing of the vertical direction position encoding layer, and the results are weighted and summed and then unfolded into the i-th visual feature sequence

[0051] Step 2.1.3: the self-attention layer includes one multi-head self-attention layer and one feedforward network layer; the self-attention layer is used to build global dependency relationships of image features;

[0052] The i-th visual feature sequence Sequence i is input into the self-attention layer for processing and outputs the i-th visual encoding feature map

[0053] Step 2.2: the multi-modal feature interaction decoding module sequentially includes a character embedding layer, a second position encoding layer, two multi-head attention layers and a feedforward network layer connected in series through a residual connection;

[0054] Step 2.2.1: the text label Y i is input into the Chinese handwritten text line recognition network and undergoes mapping processing of the character embedding layer in the multi-modal feature interaction decoding module, thereby obtaining the i-th text vector

[0055] Step 2.2.2: Text label Y i The character position indices {1,2,…,T} are processed by the second positional encoding layer to obtain the i-th position vector. Where, p it Indicates text label Y i The position vector of the t-th character;

[0056] Step 2.2.3: Place C i and P i After addition, the i-th fused feature map is obtained. Among them, c it Indicates text label Y i The fused feature vector of the t-th character; fused feature vector c it By fusing text vector features and positional features, this design aims to selectively fuse text vector features from positions other than its own position t when predicting the character at the current position t, using an attention mask matrix as contextual information to infer the current character.

[0057] Step 2.2.4: Define the attention mask matrix and P i As the query vector of the first multi-head attention layer, the context will be used. i As the key vector and value vector of the first multi-head attention layer, the positional context feature map output by the first multi-head attention layer is obtained using Equation (1). Where, d 1it This represents the location context feature vector at position t.

[0058]

[0059] In equation (1), ' represents the matrix transpose operation.

[0060] The first multi-head attention layer queries the vector p at each location. it The attention mask matrix M is responsible for predicting the character at position t, and it selects the character c from the query vector at each position, excluding itself. it Context feature vectors at other locations {c i1 ,c i2 ,…,c iT The obtained location context features D 1i In the context feature vector d of each location 1it All of them contain contextual information selected by the attention mask M.

[0061] In specific implementation, the attention mask matrix M in the training process includes three forms, from left to right, from right to left, and a sliding window-based random sampling mask, as shown in Figure 3a 、 Figure 3b and Figure 3c . That is, divided into left-to-right mask right-to-left mask and a sliding window-based probabilistic random mask wherein represents the value of the mth row and the nth column of the mask matrix M1, and represents the value of the mth row and the nth column of the mask matrix M2, and represents the value of the mth row and the nth column of the mask matrix M3, and s represents the size of the window, and random(0,1) represents a random function of 0 or 1;

[0062] The role of the three attention masks is that the first two allow the model to learn character recognition based on the left or right context, and allow the model to learn the autoregressive reasoning. The third mask contains a sliding window mechanism and a random sampling mechanism, as shown in Figure 2 the left half, the sliding window mechanism controls the model to only fuse the context features near the position when predicting the character at each position, and discards irrelevant context information far away from the position. It can reduce the proportion of irrelevant context. The random sampling mechanism is shown in Figure 2 the right half, when predicting the character at the current position, the model randomly samples characters at different positions in the window as context. Let the model learn to decode jointly with visual features under the conditions of no context, partial context, and full context, so that the network can learn pure visual reasoning and visual language reasoning under different context conditions, as well as the ability of completing a blank. Further improve the language reasoning ability of the model.

[0063] Step 2.2.5: taking D 1i as the query vector query of the second multi-head attention layer, V i as the key vector key and the value vector value of the second multi-head attention layer, so as to obtain the position visual semantic feature map D 2i output by the second multi-head attention layer by using formula (2). 1i The second multi-head attention layer is a layer for multi-modal interaction, and the query vector D i with context information and the visual feature V 2iBoth visual feature information and context language feature information are contained, so that the subsequent network module can utilize the ability of visual and language feature information to jointly infer characters.

[0064]

[0065] Step 2.2.6: Position visual semantic feature map D 2i After being input into the feedforward network layer and sequentially processed by the first linear layer, the relu layer and the second linear layer, the decoding feature map D output by the decoder is obtained 3i ;

[0066] Step 2.3: The text category prediction module includes a fully connected layer and a softmax layer.

[0067] Decoding feature map D 3i After being input into the text category prediction module and sequentially processed by the fully connected layer and the softmax layer, the prediction value is obtained Wherein, represents the predicted character of the t-th position.

[0068] Step 4: According to and Y i , the cross-entropy loss function is constructed, and the Chinese handwritten text line recognition network is trained by using the gradient descent method until the cross-entropy loss function converges, so as to obtain the trained Chinese handwritten text line recognition model Model.

[0069] In the specific implementation, because three forms of attention mask matrix M are used, in the training of the Chinese handwritten text line recognition network by using the gradient descent method, M1, M2 and M3 are brought into formula (1) respectively in each training process to obtain three position context feature maps output by the first multi-head attention layer and The three position context feature maps and participate in the processing of the subsequent multi-modal feature interaction decoding module and the text category prediction module respectively, so as to obtain three prediction values; so that three different cross-entropy loss functions are established in step 4, and the average value is taken as the final total loss function for model training.

[0070] Step 5: The test process is as shown in Figure 4 , the image to be recognized X test is input into the model Model for processing to obtain the preliminary prediction result The preliminary prediction result The modal feature interaction decoding module input to the model Model is processed again, and a corrected recognition result is obtained after passing through the text category prediction module in turn Specifically, in the recognition of step 5, a preliminary prediction result is obtained by using the left-to-right mask M1 The preliminary prediction result is input to the multi-modal feature interaction decoding module as input The mask M1 is set in the first multi-head attention layer The final recognition result is obtained from the text category prediction module Wherein, M4 (m, n) represents the value of the mth row and the nth column of the mask matrix M4, and

[0071] The above scheme of the embodiment can improve recognition accuracy, and experiments prove that the following data sets are used in the experiments:

[0072] CASIA-HWDB: This is a large Chinese handwritten data set published by the Institute of Automation of the Chinese Academy of Sciences. Among them, CASIA-HWDB1.0-1.2 records 3895135 characters written by 1020 authors. CASIA-HWDB2.0-2.2 contains 52230 text line images written by 1019 authors.

[0073] ICDAR2013: This is a competition data set, which contains 3432 offline handwritten Chinese text line images written by 60 authors

[0074] In the experiment, the single word data set of CASIA-HWDB1.0-1.2 is used to synthesize 1 million text line data according to the collected corpus, and the training set of CASIA-HWDB2.0-2.2 is used as the training data of the experiment. The ICDAR2013 offline text line data set and the test set of HWDB2.0-2.2 are used as the test part of the experiment, and the scheme introduced in the foregoing is used to train the model.

[0075] The test results show that the method proposed by the present application achieves the most advanced performance in the task of Chinese handwritten text line recognition, and the character level accuracy on the ICDAR2013 and HWDB test sets is 97.74% and 98.91%, respectively.

[0076] As can be seen from the technical solutions provided by the application, in the training process, the multi-modal attention layer of the decoding module and the three kinds of attention masks guide the model to learn to recognize each character by combining visual features under different semantic contexts, so that the model has the ability of visual reasoning and visual language joint reasoning; in the test stage, the model has the ability of parallel prediction and autoregressive prediction, and can iteratively correct the preliminary recognition result, effectively improving the recognition accuracy of the Chinese handwritten text line.

Claims

1. A method for recognizing lines of Chinese handwritten text based on joint visual-language reasoning, characterized in that, Includes the following steps: Step 1: Collect a set of images of Chinese handwritten text lines. } and its corresponding text label set Y And used as the training sample set, where, This represents the i-th training image sample. Represents the i-th training sample image The text label, and , Represents text label The t-th character in the image; T represents the total number of characters, n represents the number of sample images; H represents the height of the sample image, and W represents the width of the sample image; Step 2: Build a Chinese handwritten text line recognition network based on multimodal attention, including: a visual feature encoding module, a multimodal feature interaction decoding module, and a text category prediction module; Step 2.1: The visual feature encoding module includes: a convolutional neural network, a position encoding layer, and a self-attention layer; Step 2.1.1: The convolutional neural network adopts a ResNet structure, which includes four convolutional blocks and their corresponding max pooling layers. Each convolutional block includes several residual blocks. The i-th training text image The input is fed into the Chinese handwritten text line recognition network, and sequentially passes through the convolution and downsampling operations in the convolutional neural network of the visual feature encoding module, thereby outputting the i-th convolutional feature. ; The dimension representing the convolutional feature; Step 2.1.2: The position coding layer includes a horizontal position coding layer and a vertical position coding layer, which respectively adopt sine and cosine position coding methods, and then the two are weighted and summed; The i-th convolutional feature The input is processed by the position encoding layer, and then by the sinusoidal position encoding of the horizontal position encoding layer and the cosine position encoding of the vertical position encoding layer. The results are then weighted and summed to expand into the i-th visual feature sequence. ; Step 2.1.3: The self-attention layer comprises one multi-head self-attention layer and one feedforward network layer; The i-th visual feature sequence The input is processed in the self-attention layer, and the i-th visual encoded feature map is output. ; Step 2.2: The multimodal feature interaction decoding module includes, in sequence: a character embedding layer, a second position encoding layer, two multi-head attention layers connected in series via residual connections, and a feedforward network layer; Step 2.2.1: The text label The input is fed into the Chinese handwritten text line recognition network and processed by the character embedding layer in the multimodal feature interaction decoding module to obtain the i-th text vector. ; Step 2.2.2: The text label Character position index After processing by the second positional encoding layer, the i-th position vector is obtained. ;in, Indicates the text label The position vector of the t-th character; Step 2.2.3: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] and After addition, the i-th fused feature map is obtained. ;in, Indicates the text label The fused feature vector of the t-th character; Step 2.2.4: Define the attention mask matrix M and will As the query vector of the first multi-head attention layer, As the key vector and value vector of the first multi-head attention layer, the positional context feature map output by the first multi-head attention layer is obtained using Equation (1). ,in, This represents the location context feature vector at position t. (1) In equation (1), This represents the matrix transpose operation; Step 2.2.5: with The query vector of the second multi-head attention layer is called query. As the key vector and value vector of the second multi-head attention layer, the positional visual semantic feature map output by the second multi-head attention layer is obtained using Equation (2). : (2) Step 2.2.6: Location Visual Semantic Feature Map The input is fed into the feedforward network layer and processed sequentially through the first linear layer, a ReLU layer, and the second linear layer to obtain the final decoded feature map output by the decoder. ; Step 2.3: The text category prediction module includes: a fully connected layer and a softmax layer; Decoding feature map The text image is input into the text category prediction module and processed sequentially through a fully connected layer and a softmax layer to obtain the i-th training text image. Predicted value ,in, Represents the i-th training text image The predicted character at the t-th position; Step 3: According to and A cross-entropy loss function is constructed, and the Chinese handwritten text line recognition network is trained using gradient descent until the cross-entropy loss function converges, thereby obtaining the trained Chinese handwritten text line recognition model. Step 4: Image to be recognized The data is input into the Model for processing to obtain preliminary prediction results. The preliminary prediction results The data is then input again into the modal feature interactive decoding module of the model for processing, and then sequentially passed through the text category prediction module to obtain the corrected recognition result. .

2. The Chinese handwritten text line recognition method based on visual-language joint reasoning according to claim 1, characterized in that, The attention mask matrix M in step 2.2.4 is divided into masks from left to right. Mask from right to left and probabilistic random masks based on sliding windows ,in, Represents the mask matrix The value in the m-th row and n-th column, and , Represents the mask matrix The value in the m-th row and n-th column, and , Represents the mask matrix The value in the m-th row and n-th column, and s represents the window size, and random(0,1) represents a random function of 0 or 1; In training the Chinese handwritten text line recognition network using gradient descent, during each training process, the following steps are performed: , and Substituting into equation (1), we obtain the three location context feature maps output by the first multi-head attention layer. , and , Three types of location context feature maps , and The values ​​are then input into the subsequent multimodal feature interaction decoding module and text category prediction module for processing, resulting in three predicted values. In step 3, three different cross-entropy loss functions are established, and the average value is taken as the final total loss function for model training.

3. The Chinese handwritten text line recognition method based on visual-language joint reasoning according to claim 2, characterized in that, In the identification process of step 4, a mask is used from left to right. Preliminary prediction results obtained Then the preliminary prediction results As input to the multimodal feature interaction decoding module, and with attention mask set in the first multi-head attention layer. Thus, the text category prediction module obtains the final recognition result. ,in, Represents the mask matrix The value in the m-th row and n-th column, and .

Images