A text recognition method and device, and a storage medium

Abstract

The application discloses a text recognition method and device, and a storage medium. The method comprises the following steps: detecting non-text elements in a to-be-recognized text, determining whether the non-text elements comprise a first element, and extracting the first element from the to-be-recognized text according to the position of the first element in the to-be-recognized text; performing category recognition on each pixel point in an image corresponding to the extracted first element, and at least identifying first-category pixel points and second-category pixel points; performing convolution, pooling and up-sampling processing on a first sub-element corresponding to the first-category pixel points and a second sub-element corresponding to the second-category pixel points, and performing convolution training; recognizing text in the to-be-recognized text, and then splicing the recognized non-text elements with the recognized text according to position information before recognition to generate a recognized text. The application improves the accuracy of table recognition in a text file and improves the accuracy of text recognition.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to image text recognition technology, and in particular, to a text recognition method and device, and a storage medium. BACKGROUND

[0002] With the rapid generation of electronic documents generated by digital documents, photographing and scanning and other document methods, structured analysis and content extraction of electronic documents have become a hot research topic. Electronic documents include scanned and photographed image pieces of paper documents and digital documents. These electronic documents are not editable and are stored in image form, which is very inconvenient. At present, optical character recognition (OCR) technology mainly detects and recognizes text, and lacks recognition and processing of layout.

[0003] There are many elements in electronic documents, such as pictures, tables, signatures, seals, and two-dimensional codes. The detection and structure recognition of tables have always been an important branch of document OCR. Due to the diversity of types and styles and the complexity of structures of tables, table analysis has always been a research difficulty in the field of document recognition. SUMMARY

[0004] Therefore, embodiments of the present application provide a text recognition method and device, and a storage medium.

[0005] According to a first aspect of the present application, a text recognition method is provided, comprising:

[0006] detecting non-text elements in the text to be recognized, and determining that the non-text elements include a first element, and extracting the first element from the text to be recognized according to the position of the first element in the text to be recognized;

[0007] performing category recognition on each pixel point in the image corresponding to the extracted first element, and at least identifying first category pixel points and second category pixel points;

[0008] For the first type of pixel point corresponding to the first element in the first type of sub-element, the first convolution, the rectified linear unit (ReLU) function processing are performed layer by layer according to the set number of layers of hierarchical processing for the first type of sub-element, and then the pooling processing is performed; after the first convolution and the ReLU processing of the first type of sub-element reach the bottom layer, the second convolution processing is performed, and the upsampling, the first convolution processing and the ReLU processing are performed layer by layer from the bottom layer to the top layer according to the set hierarchical layer; the second convolution processing is performed on the processing data of the top layer to obtain the first type of sub-element in the first element; wherein, the first convolution processing and the ReLU in the upsampling process of the first type of sub-element include: the feature map in the downsampling process is cropped, connected with the feature map of the same level in the upsampling process, and the connected feature map is subjected to the first convolution and the ReLU processing;

[0009] For the second type of sub-element in the first element corresponding to the first type of pixel point, the first convolution and the ReLU function processing are performed layer by layer according to the set number of layers of hierarchical processing for the second type of sub-element, and then the pooling processing is performed; after the first convolution and the ReLU processing of the first type of sub-element reach the bottom layer, the first convolution processing and the ReLU processing are performed; then the upsampling, the first convolution and the ReLU processing are performed layer by layer from the last layer to the top layer according to the set hierarchical layer; the second convolution processing is performed on the processing data of the top layer to obtain the second type of sub-element in the first element; wherein, the first convolution and the ReLU processing of the upsampling of the second type of sub-element include: the feature map in the downsampling process is cropped, connected with the feature map of the same level in the current upsampling process; and when the feature map corresponding to the same level contains the first type of sub-element of the upsampling processing, the feature map of the same level of the upsampling of the first type of sub-element is cropped, connected with the feature map in the current upsampling process; and the connected feature map is subjected to the first convolution and the ReLU processing;

[0010] The non-text elements in the to-be-recognized text are recognized, and the recognized non-text elements are spliced with the recognized text according to the position information before recognition to generate a recognized text.

[0011] As an implementation manner, the detecting the non-text elements in the to-be-recognized text comprises:

[0012] The whole image features of the to-be-recognized text are extracted by a sampling convolution network to predict the region of the non-text element;

[0013] The features of the non-text element region are classified to determine the non-text element.

[0014] As an implementation form, the non-text element includes at least one of the following: a picture, a table, a signature, a two-dimensional code, and a seal.

[0015] As an implementation form, the first element includes a table.

[0016] The first type of sub-element includes visible horizontal lines and vertical lines, and the second type of sub-element includes invisible horizontal lines and vertical lines; wherein the number of down-sampling processes for the visible horizontal lines and vertical lines is less than that for the invisible horizontal lines and vertical lines.

[0017] As an implementation form, the first convolution processing includes 3*3 convolution processing, the second convolution processing includes 1*1 convolution processing, the pooling processing includes 2*2 maximum pooling, and the up-sampling includes 2*2 bilinear up-sampling.

[0018] According to a second aspect of the present application, a text recognition device is provided, comprising:

[0019] An extraction unit is configured to detect a non-text element in a text to be recognized, and when it is determined that the non-text element includes a first element, extract the first element from the text to be recognized according to a position of the first element in the text to be recognized.

[0020] A first recognition unit is configured to perform category recognition on each pixel point in an image corresponding to the extracted first element, and at least recognize first category pixel points and second category pixel points.

[0021] the first element corresponding to the first category pixel point, performing first convolution, ReLU function processing layer by layer respectively according to the number of hierarchical processing layers set for the first category sub-element, and then performing pooling processing; after the first convolution and ReLU processing of the bottom layer are reached, performing second convolution processing, and performing upsampling, first convolution and ReLU processing layer by layer upwards according to the set hierarchical layers from the bottom layer until the top layer is reached; performing second convolution processing on the processing data of the top layer to obtain the first category sub-element in the first element; wherein the first convolution and ReLU processing in the upsampling process of the first category sub-element includes: cropping the feature map in the downsampling process, connecting with the feature map of the same level in the upsampling process, and then performing first convolution and ReLU processing on the connected feature map; and for the second category sub-element in the first element corresponding to the first category pixel point, performing first convolution, ReLU function processing layer by layer respectively according to the number of hierarchical processing layers set for the second category sub-element, and then performing pooling processing; after the first convolution and ReLU processing of the bottom layer are reached, performing first convolution processing and ReLU processing; then performing upsampling, first convolution and ReLU processing layer by layer respectively from the last layer to the top layer according to the set hierarchical layers; performing second convolution processing on the processing data of the top layer to obtain the second category sub-element in the first element; wherein the first convolution and ReLU processing of the upsampling of the second category sub-element includes: cropping the feature map in the downsampling process, connecting with the feature map of the same level in the current upsampling process; and when the feature map corresponding to the same level contains the upsampling processing of the first category sub-element, cropping the feature map of the same level of the upsampling of the first category sub-element, and connecting with the feature map in the current upsampling process; and then performing first convolution and ReLU processing on the connected feature map;

[0022] The text recovery unit is configured to recognize the characters in the to-be-recognized text, and then splice the recognized non-character elements with the recognized characters according to the position information before recognition to generate a recognized text.

[0023] As an implementation manner, the extraction unit is further configured to:

[0024] extract the whole image features of the to-be-recognized text by using a sampling convolution network, predict the region of the non-character element, and classify the features of the non-character element region to determine the non-character element.

[0025] As an implementation manner, the non-character element includes at least one of the following: a picture, a table, a signature, a two-dimensional code and a seal;

[0026] The first element includes a table.

[0027] The first type of sub-element includes visible horizontal and vertical lines, and the second type of sub-element includes invisible horizontal and vertical lines; wherein the number of down-sampling processes for the visible horizontal and vertical lines is less than the number of down-sampling processes for the invisible horizontal and vertical lines.

[0028] As an implementation manner, the first convolution processing includes 3*3 convolution processing; the second convolution processing includes 1*1 convolution processing; the pooling processing includes 2*2 maximum pooling; and the up-sampling includes 2*2 bilinear up-sampling.

[0029] According to a third aspect of the present application, a storage medium is provided, which stores an executable program, and the executable program, when executed by a processor, implements the steps of the text recognition method.

[0030] The text recognition method and device provided by the embodiments of the present application, and the storage medium, first detect non-text elements in the text to be recognized, obtain image blocks, QR codes, seal elements, signature elements and position information thereof in the document image, and also obtain table information and text information, reconstruct the document and design a multi-branch deep learning model for table analysis, so that the visible lines and invisible lines in the table can be accurately recognized, so that the table can be reconstructed, and the detected non-text elements are inserted into the recognized text information according to their positions, so that the document image can be effectively reconstructed, and the reconstruction effect of a complex table is better. The embodiments of the present application improve the accuracy of text recognition, especially for tables in the text, and the recognition is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 A flowchart of the text recognition method provided by the embodiments of the present application is shown in the figure.

[0032] Figure 2 A flowchart of the text recognition method provided by the embodiments of the present application is shown in the figure.

[0033] Figure 3 A table recognition flowchart of the embodiments of the present application is shown in the figure.

[0034] Figure 4 A composition structure diagram of the text recognition device provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0035] The essence of the technical solutions of the embodiments of the present application will be described in detail below in combination with examples.

[0036] Figure 1 A flowchart of the text recognition method provided by the embodiments of the present application is shown in the figure. Figure 1 As shown in the figure, the text recognition method of the embodiments of the present application includes the following processing steps:

[0037] In step 101, a non-text element in the to-be-recognized text is detected, when it is determined that the non-text element includes a first element, the first element is extracted from the to-be-recognized text according to a position of the first element in the to-be-recognized text.

[0038] In an embodiment of the present application, the non-text element includes at least one of the following: a picture, a table, a signature, a two-dimensional code, and a seal.

[0039] The first element includes a table, the first type of sub-element includes a visible horizontal line and a vertical line, and the second type of sub-element includes an invisible horizontal line and a vertical line.

[0040] The non-text element in the to-be-recognized text is detected, that is, the non-text element in the to-be-recognized text is subjected to object detection (Object Detection), which is to find all objects of interest in an image, including object positioning and object classification two sub-tasks, while determining the category and position of the object. Specifically, it includes:

[0041] The features of the entire image of the to-be-recognized text are extracted by a sampling convolutional network to predict the region of the non-text element; the features of the region of the non-text element are classified to determine the non-text element.

[0042] In an embodiment of the present application, when the non-text element is recognized, it is extracted. In an embodiment of the present application, when the table is recognized, the table is extracted and processed.

[0043] In step 102, each pixel point in an image corresponding to the extracted first element is subjected to category recognition, and at least a first category pixel point and a second category pixel point are recognized.

[0044] In an embodiment of the present application, the table is mainly recognized, and in addition to the visible line, many tables currently also include invisible lines. For example, in many table structures, invisible lines are often used to represent table structures. In an embodiment of the present application, the pixel points in the table image are recognized and divided into first category pixel points and second category pixel points, that is, at least the table formed by the invisible lines and the table structure formed by the visible lines are respectively recognized.

[0045] In step 103, a first type of sub-element corresponding to the first category pixel point and a second type of sub-element corresponding to the second category pixel point are respectively subjected to convolution, pooling, and up-sampling processing, and convolution training is performed.

[0046] In the embodiment of the present application, for the first type of sub-element in the first element corresponding to the first type of pixel point, first convolution, ReLU function processing are performed layer by layer according to the set number of hierarchical processing layers of the first type of sub-element, and then pooling processing is performed; after the first convolution and ReLU processing of the bottom layer, second convolution processing is performed, and up-sampling, first convolution processing and ReLU processing are performed layer by layer from the bottom layer to the top layer according to the set hierarchy, until the top layer is reached; the processing data of the top layer is subjected to second convolution processing to obtain the first type of sub-element in the first element; wherein, the first convolution processing and ReLU in the up-sampling process of the first type of sub-element include: cropping the feature map in the down-sampling process, connecting the feature map of the same level in the up-sampling process, and then performing first convolution and ReLU processing on the connected feature map;

[0047] For the second type of sub-element in the first element corresponding to the first type of pixel point, first convolution, ReLU function processing are performed layer by layer according to the set number of hierarchical processing layers of the second type of sub-element, and then pooling processing is performed; after the first convolution and ReLU processing of the bottom layer, first convolution processing and ReLU processing are performed; then up-sampling, first convolution and ReLU processing are performed layer by layer from the last layer to the top layer according to the set hierarchy, until the top layer is reached; the processing data of the top layer is subjected to second convolution processing to obtain the second type of sub-element in the first element; wherein, the first convolution and ReLU processing of the up-sampling of the second type of sub-element include: cropping the feature map in the down-sampling process, connecting the feature map of the same level in the current up-sampling process; and when the feature map corresponding to the same level contains the up-sampling processing of the first type of sub-element, the feature map of the same level of the up-sampling of the first type of sub-element is cropped, and connected with the feature map in the current up-sampling process; and then first convolution and ReLU processing are performed on the connected feature map.

[0048] In the embodiment of the present application, the number of down-sampling processes for the visible horizontal and vertical lines is less than that for the invisible horizontal and vertical lines. The first convolution processing includes 3x3 convolution processing; the second convolution processing includes 1x1 convolution processing; the pooling processing includes 2x2 maximum pooling; and the up-sampling includes 2x2 bilinear up-sampling.

[0049] In step 104, the characters in the to-be-recognized text are recognized, and the recognized non-character elements are spliced with the recognized characters according to the position information before recognition to generate a recognized text.

[0050] In the embodiment of the present application, after the characters in the to-be-recognized text are recognized, the non-character elements and the character content are recombined into the recognized text according to the position information of the characters and the previously recognized non-character elements in the to-be-recognized text, and the recognized text is output.

[0051] In the embodiment of the present application, different convolution training methods are used for the visible lines and the invisible lines in the table respectively, so that the accuracy of table recognition is higher. Therefore, the recognition accuracy of the entire text is higher, and the practicability is better.

[0052] The embodiment of the present application will be further described in detail in combination with specific examples.

[0053] In view of the practical problem of difficult table analysis in the document OCR process, the embodiment of the present application provides a document OCR recognition method, and a multi-branch deep learning model is designed for table analysis. The deep learning model can effectively reconstruct the document image, and the reconstruction effect of the complex table is ideal.

[0054] Figure 2 The flowchart of the text recognition method provided by the embodiment of the present application is shown in FIG. 1. Figure 2 The text recognition method of the embodiment of the present application includes the following processing steps.

[0055] The scanned copy, the photographed image, and the PDF scanned copy of the to-be-recognized document are input into the text recognition system of the embodiment of the present application in the form of an image as the to-be-recognized text of the embodiment of the present application.

[0056] First, the special elements such as non-character elements in the to-be-recognized text are recognized, and the non-character elements including image blocks, tables, two-dimensional codes, signatures, and seals in the image of the to-be-recognized text are detected, and the position information corresponding to the non-character elements is obtained. The detected non-character elements are extracted, and the corresponding position information is saved.

[0057] If the document image contains a table, the table will be processed by a convolutional neural network. First, the table is subjected to semantic segmentation processing to segment the table lines and obtain table structure information. Then, the characters in the table cells are subjected to character detection and character recognition processing to obtain the character information of the cells.

[0058] After the non-character elements in the image of the to-be-recognized text are extracted, only characters are left in the remaining document image. The remaining document image is subjected to character detection and character recognition processing to obtain the position and content information of the text lines.

[0059] The image blocks, two-dimensional codes, seal elements, and position information in the document image obtained above, as well as the table information and character information, can reconstruct the document.

[0060] Specifically, the detection of special elements such as non-text elements in the embodiments of the present application can adopt target detection technology, and the target detection model such as a lightweight model yolov4 can be adopted in the embodiments of the present application. The special elements in the embodiments of the present application include five categories of pictures, tables, signatures, two-dimensional codes, and seals. The pixel size of the image to be recognized is 608x608, and the output of the target detection is the category and corresponding position information of the special element.

[0061] In the embodiments of the present application, both the text detection and the text recognition use open-source deep learning models. The text detection can use the DBNet method, and the text recognition can use the convolutional recurrent neural network structure CRNN technology, which is mainly used for end-to-end recognition of indefinite-length text sequences, without cutting individual characters first, but converting text recognition into a time-dependent sequence learning problem, that is, sequence recognition based on images.

[0062] In the embodiments of the present application, a separate recognition method is designed for table recognition in the text to be recognized, that is, a semantic segmentation technology is used to segment the table lines, which are mainly divided into four categories of visible horizontal lines, visible vertical lines, invisible horizontal lines, and invisible vertical lines. According to the characteristics of the table, a multi-branch semantic segmentation model is designed in the embodiments of the present application.

[0063] Since the segmentation of invisible lines of the table needs to combine the layout information of the table rows and columns for segmentation, a larger receptive field is needed to obtain the row and column information, so the number of down-sampling layers is relatively large. However, for the visible horizontal and vertical lines, since the line segments are relatively thin, if the number of down-sampling is too large, the line segment information will be lost, affecting the segmentation of the visible lines, so the number of down-sampling required for the visible lines is relatively small. In summary, a multi-branch semantic segmentation model is proposed in the embodiments of the present application, which uses a U-shaped structure to sample different numbers of up-sampling and down-sampling for visible lines and invisible lines. For details of the recognition method of the table structure, please refer to Figure 3 , Figure 3 A table recognition flowchart of the embodiments of the present application is shown, and the table recognition of the embodiments of the present application specifically includes:

[0064] In the embodiments of the present application, Figure 3 each rectangular line represents a feature map (feature map), and the number above the rectangular line represents the number of feature maps, which is also the number of convolution kernels in the previous convolution operation. The number of feature maps at the input is 3, representing the channel number of the image. Figure 3 The operation explanation of each arrow line is shown in the following table:

[0065] 1. conv3x3, ReLU, which represents performing 3x3 convolution and then performing nonlinear activation function ReLU operation;

[0066] 2. crop, which represents cropping the feature map, and the cropped feature map is used to connect with the subsequent up-sampled feature map;

[0067] 3. conv1x1, which represents performing 1x1 convolution;

[0068] 4. max pool2x2, which represents performing 2x2 maximum pooling.

[0069] 5. Up bilnear2x2, which represents performing 2x2 bilinear up-sampling.

[0070] The table recognition model of the embodiment of the application has four branches, which are visible horizontal lines, visible vertical lines, invisible horizontal lines and invisible vertical lines. Each branch uses binary classification, and the pixel-wise softmax output by each branch is expressed as formula (1):

[0071]

[0072] where x is a pixel on a two-dimensional plane Ω, a k (x) is the value of the kth channel corresponding to the x pixel, K is the total number of classes, p k (x) is the probability that the pixel x belongs to the k class.

[0073] The loss function uses cross entropy. The expression of cross entropy is formula (2):

[0074]

[0075] where p l (x) is the probability of pixel x on the real label. A weight w(x) is added in the cross entropy to give more attention to the boundary between objects.

[0076] w(x) is formula (3):

[0077]

[0078] where w c (x) is the class weight, d1(x) and d2(x) represent the boundary distance of the pixel to the nearest and second nearest cell respectively, w0=10 and σ=5.

[0079] The losses of the four branches of the table recognition segmentation model in this application embodiment are E1, E2, E3, and E4, respectively. The total loss of the table recognition model is defined as loss = E1 + E2 + E3 + E4.

[0080] like Figure 3 As shown, the left branch represents the convolutional training process for recognizing the visible horizontal and vertical lines in the table. Specifically, for the pixels of the visible horizontal and vertical lines in the table, the convolutional training performs the set number of processing layers, performing 3×3 convolution and ReLU function processing layer by layer, followed by 2×2 max pooling. After reaching the bottom layer of 3×3 convolution and ReLU function processing, 1×1 convolution processing is performed, and upsampling, 3×3 convolution, and ReLU function processing are performed layer by layer from the bottom layer upwards until the top layer is reached. The top layer data is then processed with 1×1 convolution to achieve the recognition of the visible horizontal and vertical lines in the table. The 3×3 convolution and ReLU function processing in the upsampling process during the recognition of invisible horizontal and vertical lines in the table includes: cropping the feature map from the downsampling process, concatenating it with the feature map of the same level from the upsampling process, and then performing 3×3 convolution and ReLU function processing on the concatenated feature map, as shown below. Figure 3 As shown.

[0081] like Figure 3 As shown, the branch on the right is the convolutional training process for recognizing the pixels of invisible horizontal and vertical lines in the table. Specifically, for the pixels of visible horizontal and vertical lines in the table, the convolutional training is performed according to the set number of processing layers. Each layer is processed with 3×3 convolution and ReLU function, and then 2×2 max pooling is performed. After the bottom layer pooling is reached, 3×3 convolution and ReLU function are performed. Then, according to the set layering, upsampling, 3×3 convolution and ReLU function are performed layer by layer from the last layer until the top layer. The data processed in the top layer is processed with 1×1 convolution to realize the recognition of invisible horizontal and vertical lines in the table. The upsampling process for invisible horizontal and vertical lines during recognition involves 3×3 convolution and ReLU function processing, including: cropping the feature map from the downsampling process and concatenating it with the feature map of the same level from the upsampling process; cropping the feature map from the downsampling process and concatenating it with the feature map of the same level from the current upsampling process; and, when the feature map contains visible horizontal and vertical lines at the same level, cropping the feature map of the same level from the upsampling of the visible horizontal and vertical lines and concatenating it with the feature map from the current upsampling process of invisible horizontal and vertical lines; then performing 3×3 convolution and ReLU function processing on the concatenated feature map, such as... Figure 3 As shown.

[0082] Figure 4A component structure diagram of a text recognition device provided by an embodiment of the present application is shown in Figure 4 The text recognition device of the embodiment of the present application comprises:

[0083] The extraction unit 40 is configured to detect non-text elements in the text to be recognized, and when the non-text elements include a first element, extract the first element from the text to be recognized according to a position of the first element in the text to be recognized.

[0084] The first recognition unit 41 is configured to perform category recognition on each pixel point in an image corresponding to the extracted first element, and at least identify first-category pixel points and second-category pixel points.

[0085] The convolution training unit 42 is configured to, for a first sub-element in the first element corresponding to the first-category pixel points, perform first convolution, ReLU function processing, and then perform pooling processing layer by layer according to a set number of layers of hierarchical processing for the first sub-element. After first convolution and ReLU processing of the bottommost layer, second convolution processing is performed, and up-sampling, first convolution, and ReLU processing are performed layer by layer upwards according to a set hierarchical layer from the bottommost layer until the topmost layer is reached. The processing data of the topmost layer is subjected to second convolution processing to obtain the first sub-element in the first element. The first convolution and ReLU processing in the up-sampling process of the first sub-element include: cropping the feature map in the down-sampling process, connecting the feature map in the same level in the up-sampling process, and performing first convolution and ReLU processing on the connected feature map. For a second sub-element in the first element corresponding to the first-category pixel points, first convolution and ReLU function processing are performed layer by layer according to a set number of layers of hierarchical processing for the second sub-element, and then pooling processing is performed. After first convolution and ReLU processing of the bottommost layer, first convolution processing and ReLU processing are performed. Then, up-sampling, first convolution, and ReLU processing are performed layer by layer from the last layer until the topmost layer. The processing data of the topmost layer is subjected to second convolution processing to obtain the second sub-element in the first element. The first convolution and ReLU processing in the up-sampling of the second sub-element include: cropping the feature map in the down-sampling process, connecting the feature map in the same level in the current up-sampling process, and performing first convolution and ReLU processing on the connected feature map. When the feature map corresponds to the same level that includes up-sampling processing of the first sub-element, the feature map in the same level of the up-sampling of the first sub-element is cropped, and connected with the feature map in the current up-sampling process.

[0086] The text recovery unit 43 is configured to recognize the characters in the to-be-recognized text, and splice the recognized non-character elements with the recognized characters according to the position information before recognition to generate the recognized text.

[0087] As an implementation form, the extraction unit 40 is further configured to:

[0088] extract the whole image features of the to-be-recognized text by using a sampling convolutional network to predict the region of the non-character element, and classify the features of the region of the non-character element to determine the non-character element.

[0089] As an implementation form, the non-character element includes at least one of the following: a picture, a table, a signature, a two-dimensional code, and a seal.

[0090] The first element includes a table.

[0091] The first type of sub-element includes visible horizontal lines and vertical lines, and the second type of sub-element includes invisible horizontal lines and vertical lines; wherein the number of down-sampling processes for the visible horizontal lines and vertical lines is less than that for the invisible horizontal lines and vertical lines.

[0092] As an implementation form, the first convolutional processing includes 3*3 convolutional processing, the second convolutional processing includes 1*1 convolutional processing, the pooling processing includes 2*2 maximum pooling, and the up-sampling includes 2*2 bilinear up-sampling.

[0093] In the example embodiments, the above-mentioned processing units of the text recognition apparatus can be implemented by one or more of a central processing unit (CPU), a graphics processing unit (GPU), a base processor (BP), an application specific integrated circuit (ASIC), a DSP, a programmable logic device (PLD), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a general-purpose processor, a controller, a micro controller unit (MCU), a microprocessor, or other electronic elements.

[0094] In the embodiments of the present disclosure, Figure 4The specific manner in which the various processing units in the illustrated text recognition apparatus perform operations has been described in detail in the embodiments relating to the method, and will not be described in detail here.

[0095] The embodiments of the present application also disclose a storage medium, and the storage medium stores an executable program which, when executed by a processor, implements the steps of the text recognition method of the embodiments.

[0096] It should be understood that every technical feature mentioned in the specification refers to a specific feature of the embodiments and is included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that the sequence of the above processes does not mean the execution sequence, and the execution sequence of the processes should be determined according to the function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The sequence of the above embodiments of the present application is only for description, and does not represent the advantages or disadvantages of the embodiments.

[0097] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0098] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The above-described device embodiments are only illustrative, for example, the division of the units is only a logical functional division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling or communication connection between each component part shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0099] The units described as separate parts above can or can not be physically separate, and the parts displayed as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0100] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; and the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0101] The above is only an embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A text recognition method, characterized by, The method comprises: detecting non-text elements in the to-be-recognized text, determining that the non-text elements include a first element, extracting the first element from the to-be-recognized text according to a position of the first element in the to-be-recognized text, and the first element including a table; performing category recognition on each pixel point in an image corresponding to the extracted first element, and at least identifying first-category pixel points and second-category pixel points; for a first sub-element in the first element corresponding to the first-category pixel points, the first sub-element including visible horizontal lines and vertical lines, performing first convolution, ReLU function processing, and then performing pooling processing layer by layer according to a set number of layers of hierarchical processing of the first sub-element; after first convolution and ReLU processing of the bottommost layer, performing second convolution processing, and performing upsampling, first convolution processing, and ReLU processing layer by layer upwards according to a set hierarchical layer from the bottommost layer until the topmost layer is reached; performing second convolution processing on the processing data of the topmost layer to obtain the first sub-element in the first element; wherein the first convolution processing and ReLU in the upsampling process of the first sub-element include: cropping a feature map in the downsampling process, connecting the feature map in the same level in the upsampling process, and then performing first convolution and ReLU processing on the connected feature map; for a second sub-element in the first element corresponding to the first-category pixel points, the second sub-element including invisible horizontal lines and vertical lines, performing first convolution, ReLU function processing, and then performing pooling processing layer by layer according to a set number of layers of hierarchical processing of the second sub-element; after first convolution and ReLU processing of the bottommost layer, performing first convolution processing and ReLU processing; then performing upsampling, first convolution, and ReLU processing layer by layer from the last layer to the topmost layer; performing second convolution processing on the processing data of the topmost layer to obtain the second sub-element in the first element; wherein the first convolution and ReLU processing of the upsampling of the second sub-element include: cropping a feature map in the downsampling process, connecting the feature map in the same level in the current upsampling process, and when the feature map corresponds to the same level containing the first sub-element of the upsampling processing, cropping the feature map in the same level of the upsampling of the first sub-element, and connecting the feature map in the current upsampling process; and then performing first convolution and ReLU processing on the connected feature map; recognizing text in the to-be-recognized text, and then splicing the recognized non-text elements according to position information before recognition and the recognized text to generate a recognized text.

2. The method of claim 1, wherein, The detection of the non-text elements in the to-be-recognized text comprises: extracting whole image features of the to-be-recognized text through a sampling convolution network, and predicting a region of a non-text element; classifying features of the non-text element region, and determining the non-text element.

3. The method according to claim 1 or 2, characterized in that, The non-text element includes at least one of the following: a picture, a table, a signature, a two-dimensional code, and a seal.

4. The method of claim 3, wherein, wherein The number of downsampling processes for the visible horizontal and vertical lines is less than that for the invisible horizontal and vertical lines.

5. The method according to claim 1 or 2, characterized in that, The first convolution processing includes 3*3 convolution processing, the second convolution processing includes 1*1 convolution processing, the pooling processing includes 2*2 maximum pooling, and the upsampling includes 2*2 bilinear upsampling.

6. A text recognition apparatus characterized by comprising: The apparatus comprises: The extraction unit is configured to detect non-text elements in the to-be-identified text, determine whether the non-text elements include a first element, and extract the first element from the to-be-identified text according to a position of the first element in the to-be-identified text when the first element is included in the non-text elements, wherein the first element includes a table. The identification unit is configured to perform category identification on each pixel point in an image corresponding to the extracted first element, and identify at least first-category pixel points and second-category pixel points. The convolution training unit is configured to, for a first sub-element in the first element corresponding to the first-category pixel points, perform first convolution and ReLU function processing on the first sub-element layer by layer according to a set number of hierarchical processing layers of the first sub-element, and then perform pooling processing; after first convolution and ReLU processing on the bottommost layer, perform second convolution processing, and perform upsampling, first convolution, and ReLU processing layer by layer upwards from the bottommost layer according to a set hierarchy until the topmost layer is reached; and perform second convolution processing on processing data of the topmost layer to obtain the first sub-element in the first element; wherein the first convolution and ReLU processing in the upsampling process of the first sub-element include: cropping a feature map in a downsampling process, connecting the cropped feature map with a feature map at the same level in the upsampling process, and performing first convolution and ReLU processing on the connected feature map; and for a second sub-element in the first element corresponding to the first-category pixel points, the second sub-element includes invisible horizontal and vertical lines, and first convolution and ReLU function processing are performed on the second sub-element layer by layer according to a set number of hierarchical processing layers of the second sub-element, and then pooling processing is performed; after first convolution and ReLU processing on the bottommost layer, first convolution processing and ReLU processing are performed; then, upsampling, first convolution, and ReLU processing are performed layer by layer from the last layer to the topmost layer according to a set hierarchy; and second convolution processing is performed on processing data of the topmost layer to obtain the second sub-element in the first element; wherein the first convolution and ReLU processing of the upsampling of the second sub-element include: cropping a feature map in a downsampling process, and connecting the cropped feature map with a feature map at the same level in the current upsampling process; and when the feature map at the same level includes the upsampling processing of the first sub-element, the feature map at the same level of the upsampling of the first sub-element is cropped, and the cropped feature map is connected with a feature map in the current upsampling process; and first convolution and ReLU processing are performed on the connected feature map. The text recovery unit is configured to recognize the characters in the to-be-recognized text, and splice the recognized non-character elements with the recognized characters according to the position information before recognition to generate the recognized text.

7. The apparatus of claim 6, wherein, The extraction unit is further configured to: extract the whole image features of the to-be-recognized text by using a sampling convolution network to predict the region of the non-character element, and classify the features of the region of the non-character element to determine the non-character element.

8. The apparatus of claim 6 or 7, wherein, The non-character element includes at least one of a picture, a table, a signature, a two-dimensional code, and a seal. The number of down-sampling processes for the visible horizontal and vertical lines is less than the number of down-sampling processes for the invisible horizontal and vertical lines.

9. The apparatus of claim 6 or 7, wherein, The first convolution process includes a 3*3 convolution process, the second convolution process includes a 1*1 convolution process, the pooling process includes a 2*2 max pooling, and the up-sampling includes a 2*2 bilinear up-sampling. 10.A storage medium having stored thereon an executable program, which, when executed by a processor, implements the steps of the text recognition method according to any one of claims 1 to 5.

Images