Handwritten Chinese recognition method and system based on text detection and text recognition

Abstract

The invention discloses a handwritten Chinese recognition method and system based on text detection and text recognition, and the method comprises the steps: inputting a to-be-recognized document image into a trained text detection model, and obtaining a text detection result; the trained text detection model firstly performs preprocessing operation on a to-be-recognized document image, performs feature extraction on a preprocessing operation result, performs feature fusion on extracted features, performs regression operation and probability graph prediction on fused features, and generates a textbox based on a predicted probability graph; inputting the text detection result into the trained text recognition model to obtain a text recognition result; the method comprises the following steps of: preprocessing a text region image in a text box, performing image augmentation processing on the preprocessed text region image, executing block embedding operation on the image subjected to the augmentation processing, performing global fine-grained feature extraction on the image subjected to the block embedding operation, and identifying the extracted global fine-grained features. And obtaining a text recognition result.

Description

Technical Field

[0001] The present invention relates to the technical field of Chinese recognition, and in particular to a method and system for handwritten Chinese recognition based on text detection and text recognition. Background Art

[0002] Computer text recognition, commonly known as optical character recognition (OCR), is a subfield of computer vision and pattern recognition. It is a technology that uses computers to automatically identify character instances contained in images and process them into information that is recognizable to computers and understandable to people. With the development and popularization of artificial intelligence, especially computer vision technology, many applications related to text recognition have greatly facilitated our lives, such as license plate recognition, bank card recognition, invoice recognition, and picture recognition. Text recognition tasks are usually divided into two stages: text detection and text recognition. Text detection is a subtask of object detection. It is used to obtain the location information of multiple text instances contained in an image as the front end of the text recognition task; text recognition is used to identify and transcribe the detected text instances into structured data that is recognizable to computers and understandable to people.

[0003] With the popularization of computers and the Internet among the public, intelligent marking has gradually become popular in the field of education. As an important foundation of intelligent marking technology, handwriting intelligent recognition has also received important attention. This technology aims to transcribe the scanned images of candidates' answers into text, which is used as the input of intelligent marking to achieve the final score, and can optimize irregular handwritten text to a certain extent. Compared with ordinary handwriting recognition methods, handwritten Chinese recognition methods are more complex, which is specifically reflected in the diversity of Chinese characters and their structures. The similarity between the two is that because the writing styles of different writers vary greatly, large-scale data samples are needed to cover various writing styles and variants.

[0004] Handwritten Chinese recognition methods mainly include traditional machine learning methods and deep learning-based methods. Early traditional machine learning methods relied on manually designed feature extractors, which included edge detection, contour feature extraction, structural feature extraction, and other methods. The extracted features are then encoded and sent to the classifier for recognition. The classifiers used at this stage mainly include support vector machines (SVMs), random forests, and naive Bayes classifiers. This traditional method requires manual design and feature extraction. When processing handwritten Chinese texts, the generalization ability of the model will be limited due to the complex structure and great diversity of Chinese characters. When processing unknown or other writing style data, the model's performance will be relatively poor.

[0005] With the development of deep learning, many deep learning methods, including those based on CTC, Attention, and Transformer, have achieved good performance in scene text recognition, which has also led to the booming development of handwritten Chinese recognition. These deep learning-based models can automatically learn the optimal feature representation from the data, have strong generalization capabilities, and can handle larger data sets, which undoubtedly simplifies the steps of handwritten Chinese recognition and makes it possible to achieve end-to-end handwritten Chinese recognition. However, the current handwritten Chinese recognition is mainly based on text line recognition, that is, it can only predict the results of a line of text images, and due to the performance problems of the RNN used in the current CTC-based methods, and the attention-based methods will inevitably have attention drift when detecting long texts, resulting in problems such as missed recognition and wrong recognition in long text line scenarios. Summary of the invention

[0006] In order to solve the deficiencies of the prior art, the present invention provides a handwritten Chinese recognition method and system based on text detection and text recognition;

[0007] On the one hand, a handwritten Chinese recognition method based on text detection and text recognition is provided, including:

[0008] Obtaining a document image to be recognized;

[0009] The document image to be identified is input into the trained text detection model to obtain the text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then extracts features from the preprocessing operation result, then fuses the extracted features, then performs a row regression operation and a probability map prediction on the fused features, and finally generates a text box based on the predicted probability map;

[0010] The text detection result is input into the trained text recognition model to obtain the text recognition result; the text area image in the text box is preprocessed, the preprocessed text area image is image augmented, the augmented image is block embedded, the image after the block embedding operation is extracted from the global fine-grained feature, the extracted global fine-grained feature is recognized to obtain the text recognition result.

[0011] On the other hand, a handwritten Chinese recognition system based on text detection and text recognition is provided, including:

[0012] An acquisition module is configured to: acquire a document image to be recognized;

[0013] The text detection module is configured to: input the document image to be identified into the trained text detection model to obtain the text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then performs feature extraction on the preprocessing operation result, then performs feature fusion on the extracted features, then performs row regression operation and probability map prediction on the fused features, and finally generates a text box based on the predicted probability map;

[0014] The text recognition module is configured to: input the text detection result into the trained text recognition model to obtain the text recognition result; perform preprocessing operation on the text area image in the text box, perform image augmentation processing on the preprocessed text area image, perform block embedding operation on the augmented image, perform global fine-grained feature extraction on the image after the block embedding operation, and recognize the extracted global fine-grained features to obtain the text recognition result.

[0015] On the other hand, there is also provided an electronic device, comprising:

[0016] a memory for non-transitory storage of computer-readable instructions; and

[0017] a processor for executing the computer readable instructions,

[0018] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0019] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0020] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0021] The above technical solution has the following advantages or beneficial effects:

[0022] (1) For page-level handwriting recognition, this method proposes a two-stage method of detection and recognition. The excellent detection module can not only effectively replace manual segmentation, but also the novel recognition module can more effectively recognize the detected text lines.

[0023] (2) Aiming at the task of detecting text lines in handwriting recognition on a page, the present invention designs a novel line regression module, which can directly utilize the rich information of different feature dimensions contained in the fused feature map to obtain the position where the text line appears, thus helping the model to better locate the text area.

[0024] (3) The present invention adopts a pixel reorganization strategy to replace the interpolation upsampling strategy commonly used in the final prediction probability map, which helps to retain more information and improve detection performance.

[0025] (4) The present invention shrinks the label area in the preprocessing of the detection stage to avoid the occurrence of text box adhesion. Combined with post-processing, it can effectively help the detection model divide different text line areas.

[0026] (5) The present invention proposes an effective global fine-grained feature extraction module in the recognition stage, which can fully extract character fine-grained information and image global information. The module stacks multiple sub-modules, each of which alternately connects grouped convolution and multi-head self-attention mechanisms. The sub-modules are connected through width downsampling modules, so that the model can not only utilize the fine-grained structural features of characters and the contextual information provided by the global image, but also fully obtain the rich information contained in the text line level, thereby better completing the recognition task.

[0027] (6) The present invention proposes a novel sequence loss function to suppress negative samples contained in the final recognition result, assist the CTC decoder in jointly optimizing the objective function, and help improve the model recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 is a flow chart of the method of embodiment 1;

[0030] Figure 2 This is a structural diagram of the handwritten Chinese detection model of the first embodiment;

[0031] Figure 3 This is a diagram showing the internal structure of the fusion pyramid module of the first embodiment;

[0032] Figure 4 This is a diagram showing the internal structure of the row regression module of the first embodiment;

[0033] Figure 5 Schematic diagram of the internal structure of the width pooling module of the first embodiment;

[0034] Figure 6 This is a diagram showing the internal structure of the probability map prediction module of the first embodiment;

[0035] Figure 7 This is the main flow chart of the text box generation module of Example 1;

[0036] Figure 8 This is a structural diagram of the handwritten Chinese recognition model of the first embodiment;

[0037] Fig. 9 This is a diagram showing the internal structure of a block embedding module according to Embodiment 1;

[0038] Fig.10 This is an internal structure diagram of the global fine-grained feature extraction module of Example 1;

[0039] Fig.11 This is a diagram showing the internal structure of the feature extraction module of Embodiment 1;

[0040] Fig.12 This is a diagram showing the internal structure of the downsampling layer of Embodiment 1;

[0041] Fig.13 This is a diagram showing the internal structure of the high-pooling layer of Embodiment 1;

[0042] Fig.14 This is a diagram showing the internal structure of the output head module of Embodiment 1;

[0043] Fig.15 This is a schematic diagram of a label file of Example 1;

[0044] Fig.16 This is a schematic diagram of shrinking each line of text area corresponding to the label inward by a certain distance according to the first embodiment. DETAILED DESCRIPTION

[0045] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] Embodiment 1

[0047] This embodiment provides a handwritten Chinese recognition method based on text detection and text recognition;

[0048] like Figure 1 As shown, the handwritten Chinese recognition method based on text detection and text recognition includes:

[0049] S101: Acquire a document image to be recognized;

[0050] S102: Inputting the document image to be identified into the trained text detection model to obtain a text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then extracts features from the preprocessing operation result, then performs feature fusion on the extracted features, then performs a row regression operation and a probability map prediction on the fused features, and finally generates a text box based on the predicted probability map;

[0051] S103: Input the text detection result into the trained text recognition model to obtain the text recognition result; perform preprocessing operation on the text area image in the text box, perform image augmentation processing on the preprocessed text area image, perform block embedding operation on the augmented image, perform global fine-grained feature extraction on the image after the block embedding operation, recognize the extracted global fine-grained features, and obtain the text recognition result.

[0052] Furthermore, the step S101: obtaining a document image to be identified includes: obtaining a document image by scanning or photographing. The document image includes a test paper, and the test paper includes a handwritten answer area.

[0053] The basic architecture of the handwritten Chinese detection model used in the present invention is a deep learning model based on the fusion pyramid module. In actual application scenarios such as test paper images, there are usually multiple subjective question answering areas, and the answering area usually contains several text area rows, and also contains irrelevant information such as question numbers and answer boxes, which requires the use of a text detection model to detect the text area. Compared with the traditional regression-based direct regression text box method or the segmentation-based text detection method that directly returns a probability binary map, this detection model can better distinguish text instances and greatly improve the detection model performance.

[0054] The handwritten Chinese detection model performs text detection on the input scanned document image. The detection model includes a preprocessing module, a feature extraction module, a fusion pyramid module, a row regression module, a probability map prediction module, and a post-processing module. The basic structure diagram is shown in the figure below. Figure 2 shown.

[0055] Furthermore, the trained text detection model includes:

[0056] A first preprocessing module, a feature extraction module, a fusion pyramid module, a probability map prediction module and a post-processing module are connected in sequence; the post-processing module includes: a binarization submodule and a text box generation submodule; wherein the output end of the fusion pyramid module is also connected to the input end of the row regression module.

[0057] Furthermore, the first preprocessing module includes: performing grayscale processing, binarization processing and size adjustment processing on the document image to be recognized.

[0058] Furthermore, the first preprocessing module is used to preprocess the scanned document image and the label file: for the scanned document image, it is grayscaled and binarized (ie, 0, 1), and the height and width are uniformly adjusted.

[0059] For the label file, since some candidates actually write very densely between lines, directly using the original text line label file for training will result in multiple lines of text being detected as one line. The label file is a txt file, each line of which contains the file name and the text area coordinate value of the corresponding file. For example (file name 1x1.y1.x2.y2, x3.y3.x4.y4,...), the corresponding example image has been given, the location is Fig.15 .

[0060] like Fig.16 As shown, in order to avoid this phenomenon, the present invention innovatively adopts a method of shrinking each line of text area corresponding to the label inward by a certain distance. Here, the area D of the original text line area D is used. area , perimeter D perimeter And the scaling factor α (α is a hyperparameter) is used to calculate the shrinkage distance, and the boundary points of the original text area are shrunk inward by the distance:

[0061]

[0062] Then, a blank image of the same size as the original image is generated, and the scaled text area is marked in the blank image, and then the scaled text area is also binarized. Fig.16 shown.

[0063] Furthermore, the feature extraction module includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer connected in sequence.

[0064] Furthermore, the feature extraction module includes:

[0065]

[0066] Among them, conv1 represents the first convolutional layer, conv2 represents the second convolutional layer, conv3 represents the third convolutional layer, conv4 represents the fourth convolutional layer, and conv5 represents the fifth convolutional layer;

[0067] The image input to the feature extraction module is the grayscale image of the document to be recognized after preprocessing. Where H and W are the height and width of the image respectively, F1, F2, F3, F4, and F5 are the output results of the five-layer convolution of ResNet-50 respectively, among which F2, F3, F4, and F5 are sent to the fusion pyramid module.

[0068] It should be understood that the feature extraction module is used to extract low-dimensional and high-dimensional features of the preprocessed document image to be identified, and then input these features into the fusion pyramid module to fuse the feature information of each dimension. This module uses the residual network ResNet-50 to extract low-dimensional and high-dimensional features of the image.

[0069] ResNet (Residual Network) is a deep neural network architecture. Its key innovation is the introduction of residual connections, which allows the network to learn identity mappings more easily and make the network very deep.

[0070] Furthermore, the pyramid module is integrated, including:

[0071] The features F2, F3, F4, and F5 extracted by the last four layers of the feature extraction module have different channel numbers. First, their channel numbers are unified to C (C is a hyperparameter) through 1×1 convolution, and the unified channel number feature maps F′2, F′3, F′4, and F′5 are obtained;

[0072] Then F′5 is upsampled by 2 times, and the upsampling strategy adopts the nearest neighbor interpolation, and then fused with F′4 to obtain the sub-fusion feature map F″4;

[0073] Then F″4 is upsampled by 2 times and fused with F′3 to obtain the sub-fusion feature map F″3;

[0074] Then F″3 is upsampled by 2 times and fused with F′2 to obtain the sub-fusion feature map F″2;

[0075]

[0076] It should be understood that if the original features are directly upsampled, it may lead to information loss and feature confusion, affecting the quality of the final feature representation. Therefore, a gradual upsampling strategy is used to allow the network to merge information at different scales. This ensures the alignment of feature maps at different resolutions, allowing features at different levels to be better integrated, helping to retain the semantic information contained in high-dimensional features and the detail information contained in low-dimensional features, thereby better detecting text regions of different scales.

[0077] After that, F″2, F″3, F″4, and F″5 are subjected to a layer of 3×3 convolution to reduce their number of channels to C / 4. The use of feature compression here helps to reduce model complexity and improve computational efficiency, while also preventing overfitting caused by overly complex models.

[0078] Then, Conv2d(F″3), Conv2d(F″4), and Conv2d(F″5) are upsampled to the same spatial size as Conv2d(F″2). Finally, the upsampling results of F″′2, F″′3, F″′4, and F″′5 are sent to the connection layer and concatenated in the channel dimension to generate the final fusion feature map f.

[0079]

[0080] It should be understood that the fusion pyramid module is designed based on the feature pyramid module and is used to fuse feature information from low dimensions to high dimensions, so that the model can better distinguish text areas from non-text areas and detect short texts. The outputs F2, F3, F4, and F5 of the last four layers of the feature extraction module are used as input, and the final fusion feature map is obtained after convolution and upsampling. Its specific structure is as follows: Figure 3 shown.

[0081] The main idea of ​​the feature pyramid is to construct multi-scale feature maps in the neural network, so that the network can detect objects at different scales and retain high-level semantic information. This design allows the network to detect objects of different sizes, even if the size of the object varies greatly in the image. In actual application scenarios, text lines vary in length and size, and the use of feature pyramid strategies can more effectively help detect text areas of different shapes.

[0082] Furthermore, the row regression module includes:

[0083] The output f of the fusion pyramid module is used as the input of the row regression module. First, f1 is obtained through a 3×3 convolutional layer with a height dimension step of 1 and a width dimension step of 2 at the first level, a batch normalization layer, and a ReLu activation function layer. Its width dimension is compressed to W / 8, where W is the width of the initial image.

[0084]

[0085] Next, f1 passes through the second-level 1×1 convolution layer, batch normalization layer, and ReLu activation function layer to obtain f2, and its channel number dimension is compressed to C / 4, where C is the number of channels of f.

[0086]

[0087] Then, f2 passes through the third-level 3×3 convolutional layer with a height dimension step size of 1 and a width dimension step size of 2, a batch normalization layer, and a ReLu activation function layer to obtain f3, whose width dimension is compressed to W / 16.

[0088]

[0089] Finally, f3 obtains the text area height information through the width pooling module.

[0090] The width pooling module is used to compress the feature map so that the model can focus on extracting vertical features. Its specific structure is as follows: Figure 5 shown.

[0091] The width pooling module first compresses the f3 width to 1 through a global pooling operation, then sends the compressed image to a 1×1 convolution, changes the number of channels to 1, and then after a layer of ReLu activation function, uses the squeeze function to remove the width dimension and channel dimension, where squeeze is a function used to remove dimensions with a size of 1 in the tensor. Finally, the compressed result is sent to the linear layer to obtain the upper left corner height information and the lower right corner height information of the D text regions, and the final information is stored in Height Info (D is a hyperparameter).

[0092]

[0093] f squeeze =Squeeze(ReLu((Conv2d(f4))) (20)

[0094]

[0095] It should be understood that the present invention innovatively proposes a line regression module, which uses the features of the fusion graph to predict the vertical coordinates of the text line for the input text line data, thereby improving the detection model performance. Its basic structure is as follows: Figure 4 shown.

[0096] It should be understood that Squeeze is a function used to remove dimensions of a tensor whose size is 1. For example, a tensor of size (1,3,1,5) uses the Squeeze function to compress the first and third dimensions, and the tensor dimension becomes (3,5). Parameter Height Info It is an array of size D×2, where D represents the number. In each 1×2 array, the first number represents the height information of the upper left corner, and the second number represents the height information of the lower right corner.

[0097] Furthermore, the probability map prediction module includes:

[0098] The output f of the fusion pyramid module is used as the input of the probability map prediction module. First, a 3×3 convolution layer, a batch normalization layer, and a ReLu activation function layer are used to obtain the feature map f′ whose number of channels is compressed to 1 / 4 of the original number of channels:

[0099]

[0100] Then, f′ is passed through a 3×3 convolutional layer and a batch normalization layer to obtain f″, where the number of channels of f″ is 16:

[0101]

[0102] Finally, f″ is sent to the variable resolution layer to obtain f″′, where the number of channels of f″′ is stretched to 1, and its spatial resolution is the same as that of the document image to be recognized. The variable resolution layer uses a pixel reorganization strategy, which increases the height and width of f″ proportionally by rearranging the channel dimension information to other dimensions, thereby improving the image resolution. Compared with directly upsampling the image, the pixel reorganization strategy can retain more information.

[0103]

[0104] Finally, the image f″′ output by the variable resolution layer is sent to the Sigmoid activation function layer to obtain the probability map P, in which the value of each pixel is in the range of 0-1:

[0105] P = Sigmoid(f″′)(25).

[0106] It should be understood that the present invention uses the fusion feature map output by the fusion pyramid module to predict the probability map. Here, the pixel reorganization strategy is innovatively used to replace the previous method using direct upsampling or interpolation strategy to improve the image resolution. Its specific structure is as follows Figure 6 shown.

[0107] It should be understood that the present invention greatly improves the performance of the detection model in distinguishing different text instances through the innovatively designed generation strategy in the post-processing module. Specifically, this module includes a binarization submodule and a text box generation submodule.

[0108] Furthermore, the binarization submodule includes:

[0109] The output probability map P of the probability map prediction module is binarized to obtain the binarized map B. The specific method is to give a threshold λ (λ is a hyperparameter), traverse each pixel point of the probability map, and when the value of the pixel point is greater than or equal to λ, change its value to 1, otherwise change it to 0:

[0110]

[0111] Furthermore, the text box generation submodule includes:

[0112] (1) Find the contour set contours corresponding to the binary image: First, multiply the pixel value of the binary image by 255 to obtain an operable binary image, and then obtain the contour set contours corresponding to all text areas of the operable binary image. Contours initially includes several contours, and these contours are all irregular in shape, and each contour contains several points.

[0113] (2) Find the minimum enclosing rectangle of each contour: For each contour in contours, calculate the minimum enclosing rectangle of the corresponding point set of the contour. If the length or width of the minimum enclosing rectangle is less than the threshold l (l is a hyperparameter), filter out the minimum enclosing rectangle.

[0114] (3) Create a dynamic contour that can perform offset operations: Create a rectangular box that has the same size as the minimum enclosing rectangle rectangle but can perform offset operations.

[0115] In the preprocessing module, in order to avoid the occurrence of adhesion, the original label file is scaled, so the obtained rectangular box rectangle that meets the threshold l is scaled back to the size corresponding to the original text box.

[0116] (4) Operate the rectangular box to finally obtain the text box: For the obtained rectangular box, calculate the area of ​​the rectangular box area and perimeter box perimeter , and then the moving distance can be calculated together with the scaling factor α:

[0117]

[0118] Finally, the boundary points of the rectangular box are moved outward by distance to obtain the corrected rectangular box. Then the minimum enclosing rectangle of the corrected rectangular box is calculated. The minimum enclosing rectangle of the corrected rectangular box is the rectangular text box corresponding to the contour. In the final image, the rectangular text box is regarded as the text area, and the rest is regarded as the non-text area.

[0119] The text box corresponding to each text area is found according to the binary image B. The main process is as follows Figure 7As shown in the figure: first draw the outline of each text object in the binary image and store the result in contours; then find the minimum circumscribed rectangle rectangle of each contour contour in contours, and then construct a dynamic contour box that can be offset according to the four corner points of rectangle. Finally, calculate the moving distance distance according to the rectangle area, perimeter and scaling factor α (α is a hyperparameter), adjust the box according to the distance, and finally correct the adjusted box horizontally to get the rectangular text box corresponding to the contour contour.

[0120] Further, S102: inputting the document image to be recognized into the trained text detection model to obtain a text detection result; the training process of the trained text detection model includes:

[0121] Constructing a first training set, wherein the first training set is a document image of a known text detection area;

[0122] The first training set is input into the text detection model to train the text detection model. When the loss function value of the text detection model no longer decreases or the iteration reaches a set number of times, the training is stopped to obtain the trained text detection model.

[0123] Furthermore, the loss function of the text detection model includes:

[0124] L=L p +λL r (28)

[0125] Among them, L p is the cross entropy loss between the post-processing module result and the original unscaled label, L r is the SmoothL1 loss between the height information of the text area predicted by the row regression module and the original unscaled label, λ is the weight parameter, and the overall loss function L drives the text detection model training.

[0126] Where L p The specific formula is as follows:

[0127]

[0128] where y i represents the label value of sample i, p i is the probability that sample i is predicted to be a positive class.

[0129] L r The specific formula is as follows:

[0130]

[0131] Where x represents the predicted value Height Info The difference between the upper left corner height information and the lower right corner height information of the actual text area.

[0132] Furthermore, the S103: inputting the text detection result into the trained text recognition model to obtain the text recognition result, wherein the trained text recognition model includes:

[0133] The second preprocessing module, image augmentation module, block embedding module, global fine-grained feature extraction module and output head module are connected in sequence.

[0134] The basic architecture of the handwritten Chinese recognition model used in the present invention is a deep learning model based on a global fine-grained feature extraction module, which effectively solves the problems of missed recognition and incorrect recognition of long texts by other recognition models. Compared with previous methods, the present invention enables the model to make full use of the relationship between text areas and text areas, and between text areas and non-text areas in the picture, allowing the model to fully perceive global features and effectively solve the problem of attention drift encountered by other recognition models.

[0135] The document image handwritten Chinese recognition model recognizes the independent text area images obtained by the detection model. The recognition model includes a preprocessing module, an image augmentation module, a block embedding module, a global fine-grained feature extraction module, and an output head module. The basic structure is as follows: Figure 8 shown.

[0136] Furthermore, the second preprocessing module includes:

[0137] First, the text line image is grayed out, and then its width and height are scaled proportionally until the height is the same as the set height. At this time, if the image width is greater than the set width, it is directly compressed in the width dimension; otherwise, blank space is filled on the right side of the image until the set width is reached.

[0138] It should be understood that the preprocessing module is used to preprocess the text line images, dictionaries, and labels output by the detection module: Since the lengths of the input text line areas vary, if the shorter text line images are directly scaled to the size required by the model, it will cause misrecognition or non-recognition. Therefore, when processing text line images, first grayscale it, and then scale its width and height in proportion until its height is the same as the height of the input image required by the model. At this time, if the image width is greater than the width of the input image required by the model, it is directly compressed in the width dimension; otherwise, blanks are filled on the right side of the image until the width of the image required by the model is reached. For label files, a correspondence between images and labels is constructed, and the labels corresponding to each image are mapped to the index of the characters in the corresponding dictionary file, and then blank fillers are used. <pad>Make all labels of the same length.

[0139] Furthermore, the image augmentation module includes: image random cropping operation, image random rotation operation and image Gaussian blur operation;

[0140] The random image cropping operation means that due to the possible slight errors in the detection module, the segmented text line may have incomplete characters or too much blank space in the height dimension. Therefore, the original text line image is randomly cropped to simulate the situation where there are slight errors in image segmentation in the usage scenario, where Δh is the height cropping amount:

[0141] y t =y+Δh (31)

[0142] The random image rotation operation means that since the images in actual application scenarios may have tilted characters or the entire text line tilted, the original image can be rotated by a certain angle with the center of the image as the rotation point to simulate this tilt phenomenon, where θ is the rotation angle, x o ,y o The center point of the image:

[0143]

[0144] The image Gaussian blur operation means: in order to reduce image noise and reduce its detail level, the original image is Gaussian blurred to reduce the impact of noise on feature extraction.

[0145] The so-called Gaussian blur is the process of weighted averaging of the entire image. The value of each pixel is obtained by weighted averaging its own value and other pixel values ​​in the neighborhood.

[0146] Its specific operation is: use a mask to scan each pixel in the image, and replace the value of the center pixel of the mask with the weighted average gray value of the pixels in the neighborhood determined by the mask. Its mathematical expression is as follows, where x is the distance to the center of the pixel and σ is the standard deviation:

[0147]

[0148] It should be understood that, since images in actual application scenarios may be tilted, characters may not be fully displayed, etc., in order to improve the generalization ability of the model, and at the same time increase sample diversity and reduce the occurrence of model overfitting, the use of an image augmentation module can effectively improve the performance of the recognition model. Combined with the actual situation of the document image handwritten Chinese text line image, the image augmentation module of the present invention uses a series of image augmentation operations that retain the overall structure and local feature information of the characters.

[0149] Furthermore, the block embedding module includes:

[0150] First, the original image I is input into the block embedding module, where H and W are the height and width of the original image respectively; the original image I passes through a 3×3 convolution layer with a step size of 2, a batch normalization layer, and a ReLu activation function layer to obtain P′ emb ,in C0 is the number of channels of the block to be input into the global fine-grained feature extraction module (C0 is a hyperparameter).

[0151] P′ emb =ReLu(BatchNorm2d(Conv2d(I))) (34)

[0152] Then P′ emb Then, we pass through a 3×3 convolutional layer with a stride of 2, a batch normalization layer, and a ReLu activation function layer to obtain the block embedding P″. enb ,in

[0153] P″ emb =ReLu(BatchNorm2d(Conv2d(P′ emb ))) (35)

[0154] Finally, P″ emb Combined with position embedding, we get the final block embedding P emb , where position embedding is a learnable embedding that creates a trainable vector for each position. Position embedding can be used to explicitly provide the model with the position information of the elements in the sequence.

[0155] P emb =P″ emb +Position Embedding (36)

[0156] It should be understood that the block embedding module divides the original image data into several non-overlapping blocks (patches), and transforms the smallest unit of the image from pixels to blocks. Block embedding is obtained by performing overlapping block embedding operations using two 3×3 two-dimensional convolutions with a stride of 2. Its specific structure is as follows: Fig. 9 shown.

[0157] Furthermore, the global fine-grained feature extraction module includes: a first stage, a second stage, a third stage and a high pooling layer connected in sequence;

[0158] The first stage includes: N1 feature extraction submodules connected in series;

[0159] The second stage includes: a first downsampling layer and N2 feature extraction submodules connected in series;

[0160] The third stage includes: a second downsampling layer and N3 feature extraction submodules connected in series.

[0161] Among them, the feature extraction submodule adopts stacked grouped convolution and self-attention mechanism, which enables the model to capture features of different granularities. The specific process is to first send the block embedding module or the downsampling layer output image to the feature extraction module. The feature extraction submodule contains submodules that are alternately connected with stacked grouped convolution and self-attention. After the grouped convolution, the image is reshaped into a vector form, and then reshaped into the original image format after passing through multiple sets of self-attention layers. If it is in the first or second stage, it is sent to the downsampling layer of the next stage to downsample the height; otherwise, it is sent to the height pooling layer to compress the height to 1. The structure of the feature extraction submodule is as follows: Fig.11 shown.

[0162] For the image P input into the feature extraction submodule i-1 , first send it to the grouped convolutional layer with layer normalization. The specific steps are to normalize the layer to obtain the feature a i The spatial dimension is divided into k parts, each of which is subjected to a 3×3 convolution operation. Finally, the results of each group of convolutions are concatenated and a jump connection is introduced to obtain a′ i :

[0163] a i =LayerNorm(P i-1 )(37)

[0164]

[0165] After that, a′ i It is sent to the multi-layer perceptron (MLP) with layer normalization, and the skip connection is introduced again to obtain the feature c extracted by group convolution i :

[0166] c i = reshape(MLP(LayerNorm(a′ i ))+a′ i )(40)

[0167] Next, for c i Perform a reshape operation to convert it into a vector form b i , where reshape is a function used to change the shape of a tensor. The main operation here is to convert c i The last two dimensions of b are combined into one dimension to get b i . Then b i Feed into the layer normalization module:

[0168] b i = reshape(c i )(41)

[0169] b′ i =LayerNorm(b i )(42)

[0170] Then b′ i It is sent to the self-attention layer to obtain the dependency, and then a skip connection is introduced to generate the intermediate variable h i The multi-head self-attention mechanism is used in the self-attention layer, and the process is as follows:

[0171] Q,K,V=b′ i (43)

[0172]

[0173] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (45)

[0174] head i =Attention(QW i Q ,KW i K ,VW i V )(46)

[0175] h i =MultiHead(Q,K,V)+b i (47)

[0176] Among them, d k is the dimension of K, h is the number of self-attention heads (h is a hyperparameter), where d model P i-1 The number of channels, d v is the dimension of V, and d k =d v =d model / h.

[0177] Finally, h i It is sent to a normalized multi-layer perceptron (MLP), and the skip connection is introduced again. Finally, the reshape operation is performed to obtain P′ i :

[0178] P′ i = reshape(MLP(LayerNorm(h i ))+h i )(48)

[0179] Next, if P′ i If it is not in the last stage, it will be sent to the downsampling layer of the next stage to downsample the height. Its specific structure is as follows Fig.12 There are two main reasons why only the height is downsampled but not the width. One is that the recognition module mainly recognizes text line images, and the text information contained in the image width dimension is very rich. If the width is downsampled, it is easy to cause information loss. The other is that the width is related to the character length. The larger the width, the sparser the final prediction result of the recognition module, which helps to make the CTC decoding in the subsequent output head module more effective.

[0180] Specifically, P′ i First, it passes through a 3×3 convolution with a step size of 2 in the height dimension and a step size of 1 in the width dimension, which will halve the height dimension; then it is sent to the layer normalization to obtain P i , and will be sent to the next stage:

[0181]

[0182] Among them, H i-1 , W i-1 P i-1 The height and width of C i is the number of channels input to the next stage (C i is a hyperparameter). If P′ i At the last stage, it is sent to the high pooling layer, whose specific structure is as follows Fig.13 The height pooling layer first compresses the image height to 1 through a global pooling operation, and then sends the compressed image to a 3×3 convolution, changing the number of channels to C. out (C out is a hyperparameter), and finally a layer of activation function is used to obtain the output result P of the global fine-grained feature extraction module. out :

[0183]

[0184] Among them, W i-1 P u-1 width, and then P out By sending it to the output head module, the prediction results of the recognition model can be obtained.

[0185] It should be understood that Transformer usually refers to a model architecture based on the self-attention mechanism, which first performed well in the field of natural language processing and recently began to shine in the field of image processing. The Transformer model is different from the traditional convolutional neural network (CNN) or recurrent neural network (RNN). It is mainly composed of a self-attention layer. The self-attention mechanism can find the dependencies between each position in the input sequence, allowing the model to focus on different positions in the input sequence at the same time. Because the text line image contains text of different lengths, it is inevitable that the model will have problems such as missed recognition and wrong recognition during recognition. And due to the existence of longer texts, it is inevitable that the model will have attention drift problems, affecting the recognition effect.

[0186] In order to solve these problems, the present invention innovatively designs a global fine-grained feature extraction module to accurately perceive the text and its fine-grained features in the image, and can capture the feature differences between the text area and the non-text area, so as to accurately identify long texts. Its basic structure is as follows: Fig.10 shown.

[0187] Furthermore, the output head module comprises:

[0188] The output P extracted using global fine-grained features out As input, it is used to predict the final output result sequence. Its specific structure is as follows Fig.14 Specifically, the output head module first uses a squeeze operation to compress the height dimension, and then passes through a linear layer to obtain pred′, whose number of channels is C char (C char is the dictionary size), and then CTC decoding is used, which is a decoding method for sequence alignment. By calculating the CTC loss (L ctc ) guides the model output to align with the label. Finally, the text sequence pred is obtained:

[0189]

[0190] pred=CTC(pred′)(53)

[0191] Furthermore, in S103: inputting the text detection result into the trained text recognition model to obtain the text recognition result, the training process of the trained text recognition model includes:

[0192] Constructing a second training set, wherein the second training set is text region images of known text recognition results;

[0193] The second training set is input into the text recognition model to train the text recognition model. When the loss function value of the text recognition model no longer decreases or the iteration reaches a set number of times, the training is stopped to obtain a trained text detection model.

[0194] The loss function L of the text recognition model is as follows:

[0195] L=L ctc +λL seq (54)

[0196] Where L ctc is the CTC decoding loss, λ is the weight parameter, L seq is the sequence loss function, and the overall loss function L drives the recognition model training. ctc The specific formula is as follows:

[0197]

[0198] L ctc =-∑logp(l|x)(57)

[0199] here, Indicates that at time step t there is a character π t The probability of π represents the entire prediction path, and the prediction result of each time step is Multiplying them together, we can get the probability p(π|x) of the entire predicted path π. -1 (l) represents the prediction path π that can get the final sequence l after the B transformation. Since the prediction path π contains the separator "-", the two separators are only considered as one character. For example, (hh-e-ll-l-oo) and (he-lll-ll-o) have the same length and are both decoded as (hello). This shows that there are multiple prediction paths π that can get the final sequence l after the B transformation. Therefore, the probability p(l|x) of the prediction result being l is expressed as the sum of the probabilities p(π|x) of all prediction paths π that can get l after the B transformation. Finally, the objective function L ctc It is the negative log-likelihood of the conditional probability p(l|x) of the label sequence l.

[0200] Sequence loss function L seq The specific formula is as follows:

[0201]

[0202] k=(k1,k2,…,k |D| )(59)

[0203]

[0204] Here, l is the label, D is the dictionary sequence file, and y is the prediction result of the recognition model. |l| y represents the sum. k means that if the guide exists in the label, the guide position value is 0, otherwise it is 1. This makes it so that the results in y that exist in the label do not participate in the calculation, that is, the loss function L seq Only the prediction results that do not appear in the label. Therefore, the loss function L is used seq The original can suppress the results in y that do not exist in the label, and can guide the recognition model to predict the correct result.

[0205] Embodiment 2

[0206] This embodiment provides a handwritten Chinese recognition system based on text detection and text recognition, including:

[0207] An acquisition module is configured to: acquire a document image to be recognized;

[0208] The text detection module is configured to: input the document image to be identified into the trained text detection model to obtain the text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then performs feature extraction on the preprocessing operation result, then performs feature fusion on the extracted features, then performs row regression operation and probability map prediction on the fused features, and finally generates a text box based on the predicted probability map;

[0209] The text recognition module is configured to: input the text detection result into the trained text recognition model to obtain the text recognition result; perform preprocessing operation on the text area image in the text box, perform image augmentation processing on the preprocessed text area image, perform block embedding operation on the augmented image, perform global fine-grained feature extraction on the image after the block embedding operation, and recognize the extracted global fine-grained features to obtain the text recognition result.

[0210] It should be noted that the acquisition module, text detection module and text recognition module described above correspond to steps S101 to S103 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0211] Embodiment 3 This embodiment further provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory, and when the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in the above embodiment 1.

[0212] Embodiment 4 This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in Embodiment 1 is completed.

[0213] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.< / pad>

Claims

1. A handwritten Chinese recognition method based on text detection and text recognition, characterized in that: include: Obtaining a document image to be recognized; The document image to be identified is input into the trained text detection model to obtain the text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then extracts features from the preprocessing operation result, then fuses the extracted features, then performs a row regression operation and a probability map prediction on the fused features, and finally generates a text box based on the predicted probability map; The text detection result is input into the trained text recognition model to obtain the text recognition result; the text area image in the text box is preprocessed, the preprocessed text area image is image augmented, the augmented image is block embedded, the image after the block embedding operation is extracted from the global fine-grained feature, the extracted global fine-grained feature is recognized to obtain the text recognition result.

2. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The trained text detection model includes: A first preprocessing module, a feature extraction module, a fusion pyramid module, a probability map prediction module and a post-processing module connected in sequence; the post-processing module includes: a binarization submodule and a text box generation submodule; wherein the output end of the fusion pyramid module is also connected to the input end of the row regression module; The first preprocessing module is used to preprocess the scanned document image and the label file: for the scanned document image, grayscale and binarization are performed on it, and the height and width are uniformly adjusted; Shrink each line of text area corresponding to the label inward by a set distance, using the area D of the original text line area D area , perimeter D perimeter And the scaling factor α is used to calculate the shrinkage distance, and the boundary points of the original text area are shrunk inward by the distance: Then, a blank image of the same size as the original image is generated, the scaled text area is marked in the blank image, and then the blank image is also binarized; The feature extraction module comprises: Among them, conv1 represents the first convolutional layer, conv2 represents the second convolutional layer, conv3 represents the third convolutional layer, conv4 represents the fourth convolutional layer, and conv5 represents the fifth convolutional layer; the image input to the feature extraction module is the grayscale image of the document image to be recognized after preprocessing. Where H and W are the height and width of the image respectively, F1, F2, F3, F4, and F5 are the output results of the five-layer convolution of ResNet-50 respectively, among which F2, F3, F4, and F5 are sent to the fusion pyramid module.

3. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: Fusion pyramid module, including: The features F2, F3, F4, and F5 extracted by the last four layers of the feature extraction module have different channel numbers. First, their channel numbers are unified to C through 1×1 convolution, and the unified channel number feature maps F′2, F′3, F′4, and F′5 are obtained; Then F′5 is upsampled by 2 times, and the upsampling strategy adopts the nearest neighbor interpolation, and then fused with F′4 to obtain the sub-fusion feature map F″4; Then F″4 is upsampled by 2 times and fused with F′3 to obtain the sub-fusion feature map F″3; Then F″3 is upsampled by 2 times and fused with F′2 to obtain the sub-fusion feature map F″2; After that, F″2, F″3, F″4, and F′5 are subjected to a 3×3 convolution, and then Conv2d(F″ F ), Conv2d(F″4), and Conv2d(F′5) are upsampled to the same spatial size as Conv2d(F″2), and finally the upsampling results of F″′2, F″′3, F″′4, and F″5 are sent to the connection layer and concatenated in the channel dimension to generate the final fusion feature map f; 4. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The row regression module comprises: The output f of the fusion pyramid module is used as the input of the row regression module. First, f1 is obtained through a 3×3 convolutional layer with a height dimension step of 1 and a width dimension step of 2, a batch normalization layer, and a ReLu activation function layer. Its width dimension is compressed to W / 8, where W is the width of the initial image. Next, f1 passes through the second-level 1×1 convolution layer, batch normalization layer, and ReLu activation function layer to obtain f2, and its channel number dimension is compressed to C / 4, where C is the number of channels of f; Then, f2 is passed through the third-level 3×3 convolutional layer with a height dimension step size of 1 and a width dimension step size of 2, a batch normalization layer, and a ReLu activation function layer to obtain f3, whose width dimension is compressed to W / 16; Finally, f3 obtains the text area height information through the width pooling module; The width pooling module first compresses the f3 width to 1 through a global pooling operation, then feeds the compressed image into a 1×1 convolution, changes the number of channels to 1, and then after a layer of ReLu activation function, uses the squeeze function to remove the width dimension and channel dimension, where squeeze is a function used to remove dimensions with a size of 1 in the tensor; Finally, the compression result is sent to the linear layer to obtain the upper left corner height information and the lower right corner height information of D text regions, and the final information is stored in Height Info middle; 5. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The probability map prediction module comprises: The output f of the fusion pyramid module is used as the input of the probability map prediction module. First, a 3×3 convolution layer, a batch normalization layer, and a ReLu activation function layer are used to obtain the feature map f′ whose number of channels is compressed to 1 / 4 of the original number of channels: Then, f′ is passed through a 3×3 convolutional layer and a batch normalization layer to obtain f″, where the number of channels of f″ is 16: Finally, f″ is sent to the variable resolution layer to obtain f″′, where the number of channels of f″′ is stretched to 1, and its spatial resolution is the same as that of the document image to be recognized; Finally, the image f″′ output by the variable resolution layer is sent to the Sigmoid activation function layer to obtain the probability map P, in which the value of each pixel is in the range of 0-1: P = Sigmoid(f″′)(25); The document image to be recognized is input into the trained text detection model to obtain the text detection result; the training process of the trained text detection model includes: Constructing a first training set, wherein the first training set is a document image of a known text detection area; Inputting the first training set into the text detection model, training the text detection model, and stopping the training when the loss function value of the text detection model no longer decreases or when the iteration reaches a set number of times, thereby obtaining a trained text detection model; The loss function of the text detection model includes: L=L p +λL r (28) Among them, L p is the cross entropy loss between the post-processing module result and the original unscaled label, L r is the SmoothL1 loss between the height information of the text area predicted by the row regression module and the original unscaled label, λ is the weight parameter, and the overall loss function L drives the text detection model training; Among them, L p The specific formula is as follows: Among them, y i represents the label value of sample i, p i The probability of predicting sample i as a positive class; L r The specific formula is as follows: Among them, x represents the predicted value Height Info The difference between the upper left corner height information and the lower right corner height information of the actual text area.

6. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The text detection result is input into the trained text recognition model to obtain the text recognition result, wherein the trained text recognition model includes: A second preprocessing module, an image augmentation module, a block embedding module, a global fine-grained feature extraction module, and an output head module connected in sequence; The second preprocessing module includes: firstly graying the text line image, then scaling its width and height in equal proportion until the height is the same as the set height, and if the image width is greater than the set width, directly compressing the width dimension; otherwise, filling blanks on the right side of the image until the set width is reached; The image augmentation module includes: an image random cropping operation, an image random rotation operation and an image Gaussian blur operation.

7. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The block embedding module comprises: First, the original image I is input into the block embedding module, where H and W are the height and width of the original image respectively; the original image I passes through a 3×3 convolution layer with a step size of 2, a batch normalization layer, and a ReLu activation function layer to obtain P′ emb ,in C0 is the number of channels of the block to be input into the global fine-grained feature extraction module; P′ emb =ReLu(BatchNorm2d(Conv2d(I))) (34) Then P′ emb Then, we pass through a 3×3 convolutional layer with a stride of 2, a batch normalization layer, and a ReLu activation function layer to obtain the block embedding P″. emb ,in P″ emb =ReLu(BatchNorm2d(Conv2d(P′ emb ))) (35) Finally, P″ emb Combined with position embedding, we get the final block embedding P emb , P emb =P″ emb +PositionEmbedding (36)。 8. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The global fine-grained feature extraction module includes: a first stage, a second stage, a third stage and a high pooling layer connected in sequence; The first stage includes: N1 feature extraction submodules connected in series; The second stage includes: a first downsampling layer and N2 feature extraction submodules connected in series; The third stage includes: a second downsampling layer and N3 feature extraction submodules connected in series; Among them, the feature extraction submodule adopts stacked group convolution and self-attention mechanism, which enables the model to capture features of different granularities; the specific process is to first send the block embedding module or the downsampling layer output image to the feature extraction module, and the feature extraction submodule contains submodules that are alternately connected with stacked group convolution and self-attention; the image is reshaped into a vector form after group convolution, and then reshaped into the original image format after multiple sets of self-attention layers. If it is in the first or second stage, it is sent to the downsampling layer of the next stage to downsample the height; otherwise, it is sent to the height pooling layer to compress the height to 1; For the image P input into the feature extraction submodule i-1 , first send it to the grouped convolutional layer with layer normalization. The specific steps are to normalize the layer to obtain the feature a i The spatial dimension is divided into k parts, each of which is subjected to a 3×3 convolution operation. Finally, the results of each group of convolutions are concatenated and a skip connection is introduced to obtain a i ′: a i =LaterNorm(P i-1 ) (37) After that, a′ i It is sent to the multi-layer perceptron with layer normalization, and the skip connection is introduced again to obtain the feature c extracted by group convolution i : c i =reshape(MLP(LayerNorm(a′ i ))+a′ i ) (40) Next, for c i Perform a reshape operation to convert it into a vector form b i , where reshape is a function used to change the shape of a tensor. The main operation here is to convert c i The last two dimensions of b are combined into one dimension to get b i ; Then b i Feed into the layer normalization module: b i =reshape(c i ) (41) b′ i =LayerNorm(b i ) (42) Then b′ i It is sent to the self-attention layer to obtain the dependency, and then a skip connection is introduced to generate the intermediate variable h i ; A multi-head self-attention mechanism is used in the self-attention layer, and the process is as follows: Q,K,V=b′ i (43) MultiHead(Q,K,V)=Concat(head1,…,head h )W O (45) head i =Attention(QW i Q ,KW i K ,VW i V ) (46) h i =MultiHead(Q,K,V)+b i (47) Among them, d k is the dimension of K, h is the number of self-attention heads, where d model P i-1 The number of channels, d v is the dimension of V, and d k =d v =d model / h; Finally, h i It is sent to the normalized multi-layer perceptron, and the skip connection is introduced again, and finally the reshape operation is performed to obtain P i ′: P′ i =reshape(MLP(LayerNorm(h i ))+h i ) (48) Next, if P i ′ is not in the last stage, it is sent to the downsampling layer of the next stage to downsample the height; P i ′First, it passes through a 3×3 convolution with a step size of 2 in the height dimension and a step size of 1 in the width dimension, which will halve the height dimension; then it is sent to the layer normalization to obtain P i , and will be sent to the next stage: Among them, H i-1 , W i-1 P i-1 The height and width of C i is the number of channels input to the next stage; if P i ′ is in the last stage, it is sent to the height pooling layer, which first compresses the image height to 1 through a global pooling operation, and then sends the compressed image to a 3×3 convolution, changing the number of channels to C out Finally, after a layer of activation function, the output result P of the global fine-grained feature extraction module is obtained. out : Among them, W i-1 P i-1 width, and then P out Send it to the output head module to get the prediction result of the recognition model; The output head module includes: an output P extracted using global fine-grained features out As input, we first use the squeeze operation to compress the height dimension, and then pass it through a linear layer to get pred′, whose number of channels is C char , then use CTC decoding to guide the model output to align with the label by calculating the CTC loss; finally, the text sequence pred is obtained: pred = CTC(pred′) (53).

9. The handwritten Chinese recognition method based on text detection and text recognition as claimed in claim 1, characterized in that: The text detection result is input into the trained text recognition model to obtain the text recognition result. The training process of the trained text recognition model includes: Constructing a second training set, wherein the second training set is text region images of known text recognition results; Input the second training set into the text recognition model to train the text recognition model. When the loss function value of the text recognition model no longer decreases or the iteration reaches a set number of times, stop the training to obtain a trained text detection model. The loss function L of the text recognition model is as follows: L=L ctc +λL seq (54) Where L ctc is the CTC decoding loss, λ is the weight parameter, L seq is the sequence loss function, and the overall loss function L drives the recognition model training; where L ctc The specific formula is as follows: in, Indicates that at time step t there is a character π t The probability of π represents the entire prediction path, and the prediction result of each time step is Multiplying them together, we get the probability p(π|x) of the entire predicted path π; π∈B -1 (l) represents the predicted path π of the final sequence l after B transformation. The probability p(l|x) of the predicted result l is expressed as the sum of the probabilities p(π|x) of all the predicted paths π that are transformed to l after B transformation. Finally, the objective function L ctc It is the negative log-likelihood of the conditional probability p(l|x) of the label sequence l; Sequence loss function L seq The specific formula is as follows: k=(k1,k2,…,k |D| )(59) Among them, l is the label, D is the dictionary sequence file, and y is the prediction result of the recognition model; ∑ |l| y represents the sum; k means that if the guide exists in the label, the guide position value is 0, otherwise it is 1.

10. A handwritten Chinese recognition system based on text detection and text recognition, characterized in that: include: An acquisition module is configured to: acquire a document image to be recognized; The text detection module is configured to: input the document image to be identified into the trained text detection model to obtain the text detection result; wherein the trained text detection model first performs a preprocessing operation on the document image to be identified, then performs feature extraction on the preprocessing operation result, then performs feature fusion on the extracted features, then performs row regression operation and probability map prediction on the fused features, and finally generates a text box based on the predicted probability map; The text recognition module is configured to: input the text detection result into the trained text recognition model to obtain the text recognition result; perform preprocessing operation on the text area image in the text box, perform image augmentation processing on the preprocessed text area image, perform block embedding operation on the augmented image, perform global fine-grained feature extraction on the image after the block embedding operation, and recognize the extracted global fine-grained features to obtain the text recognition result.

Images