Text recognition method and text recognition device

Abstract

Embodiments of the present disclosure provide a text recognition method and a text recognition device. In this text recognition method, an image including text is acquired. The text includes multiple characters. A shallow convolutional neural network is used to extract features of the image to generate an original feature map. Then, an attention module is used to determine first and second attention regions respectively for the width dimension and the height dimension of the original feature map. The overlapping region of the first and second attention regions is determined as the attention selection region. Next, a candidate feature matrix is generated based on the original feature map and the attention selection region. After that, the candidate feature matrix is multiplied by the original feature map to generate a feature interaction matrix. The feature interaction matrix is normalized to obtain a two-dimensional attention weight matrix. The original feature map is dot-multiplied with the two-dimensional attention weight matrix to obtain an enhanced feature matrix. Then, unrecognized characters in the text are recognized based on the enhanced feature matrix and the recognized characters in the text.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to a text recognition method and a text recognition device. Background Art

[0002] The digitization of shopping receipts is a trend in the development of the sales industry. A shopping receipt is a shopping voucher provided by a merchant to a consumer for retention during an offline consumption process, which truly records the details of a transaction behavior, including key contents such as the sales unit, transaction time, commodity information, and transaction amount. These key information truly and effectively reflect the sales situation of the merchant and the behavior pattern of the consumer. Due to the decentralization of the offline retail industry, the production and sales data of different commodities often only circulate within the brand internal system. The offline retail industry lacks a cross-brand, cross-category, cross-region data platform for effective sharing and analysis of production information, scheduling information, sales information, etc. The above situation has led to the formation of "information islands" in the offline retail industry with brands and / or regions as the fulcrums. The massive offline consumer behavior data is like a stagnant pool, and the huge value contained in the data itself is obscured. Shopping receipts record most of the offline consumer behavior in daily scenarios. After information desensitization, the receipts can provide real consumption data for the platform operator. Therefore, the digitization of shopping receipts is the key to breaking the information barrier of commodity brands, and a text recognition method for shopping receipts has emerged as the times require. Summary of the Invention

[0003] Embodiments described herein provide a text recognition method, a text recognition device, and a computer-readable storage medium storing a computer program.

[0004] According to a first aspect of the present disclosure, a text recognition method is provided. In this method, an image including text is acquired. The text includes multiple characters. A shallow convolutional neural network is used to extract features of the image to generate an original feature map. Among them, the resolution of the original feature map is lower than the resolution of the image. Then, an attention module is used to determine a first attention area and a second attention area for the width dimension and the height dimension of the original feature map respectively. The overlapping area of the first attention area and the second attention area is determined as the attention selection area. Next, a candidate feature matrix is generated according to the original feature map and the attention selection area. Among them, the resolution of the candidate feature matrix is the same as that of the original feature map. After that, the candidate feature matrix is multiplied by the original feature map to generate a feature interaction matrix. The feature interaction matrix is normalized to obtain a two-dimensional attention weight matrix. The original feature map and the two-dimensional attention weight matrix are subjected to a dot product operation to obtain an enhanced feature matrix. Then, according to the enhanced feature matrix and the recognized characters in the text, the unrecognized characters in the text are recognized.

[0005] In some embodiments of the present disclosure, determining the first attention region and the second attention region for the width dimension and the height dimension of the original feature map respectively through the attention module includes: generating width - continuous position intervals and height - continuous position intervals symmetrically about the center point of the original feature map in the width dimension and the height dimension respectively; mapping the width - continuous position intervals to width - discrete position intervals through the first embedding layer in the attention module; mapping the height - continuous position intervals to height - discrete position intervals through the second embedding layer in the attention module; determining the width - position offset of the width - discrete position intervals through the first fully - connected layer in the attention module; determining the height - position offset of the height - discrete position intervals through the second fully - connected layer in the attention module; determining the first attention region according to the width - discrete position intervals and the width - position offset; and determining the second attention region according to the height - discrete position intervals and the height - position offset.

[0006] In some embodiments of the present disclosure, the width - continuous position intervals are calculated as (P w , Y w ), where Y w ∈[0, H], and the height - continuous position intervals are calculated as (X h , P h ), where X h ∈[0, W]. Where, w represents the initial value of attention focus in the width dimension, h represents the initial value of attention focus in the height dimension, W represents the width of the original feature map, and H represents the height of the original feature map.

[0007] In some embodiments of the present disclosure, the first attention region is calculated as (X w , Y w ), where X w = W w ×f e (P w ) + b w , and Y w ∈[0, H]. The second attention region is calculated as (X h , Y h ), where X h ∈[0, W], and Y h = W h ×f e (P h ) + b h . Where, f e represents the coordinate embedding process, W w represents the weight matrix of the first fully - connected layer, W h represents the weight matrix of the second fully - connected layer, b w represents the width - position offset, bh Indicates the height position offset.

[0008] In some embodiments of the present disclosure, generating a candidate feature matrix based on an original feature map and an attention selection region includes: generating a blank matrix with the same resolution as the original feature map; copying elements corresponding to the attention selection region in the original feature map to corresponding positions in the blank matrix; assigning zero values to boundary elements of the blank matrix; and calculating values of elements in unassigned regions of the blank matrix through linear interpolation.

[0009] In some embodiments of the present disclosure, a shallow convolutional neural network includes a basic module; wherein, the basic module includes: a convolutional layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer; wherein, the kernel size of the convolutional layer is 3×3, and the kernel size of the max pooling layer is 2×2 and its stride is 2.

[0010] In some embodiments of the present disclosure, the number of basic modules in the shallow convolutional neural network is 2.

[0011] In some embodiments of the present disclosure, identifying unrecognized characters in a text based on an enhanced feature matrix and recognized characters in the text includes: taking each character in the unrecognized characters as a target character respectively, and performing the following operations on the target character: obtaining an embedding vector corresponding to the recognized characters in the text, where the embedding vector is the sum of the word vector and the position vector of the recognized characters; generating a historical text vector from the embedding vector through a multi-head self-attention module; performing an interaction operation on the historical text vector and the enhanced feature matrix to generate a sequence decoding feature; predicting the classification probability of the target character according to the sequence decoding feature; and identifying the target character according to the classification probability.

[0012] In some embodiments of the present disclosure, the shapes of characters in the picture are irregular.

[0013] In some embodiments of the present disclosure, the picture is a picture of a bill.

[0014] According to a second aspect of the present disclosure, a text recognition device is provided. The device includes at least one processor; and at least one memory storing a computer program. When the computer program is executed by the at least one processor, the device is caused to: obtain a picture including text, the text including a plurality of characters; extract features of the picture through a shallow convolutional neural network to generate an original feature map, wherein the resolution of the original feature map is lower than the resolution of the picture; determine a first attention region and a second attention region for the width dimension and the height dimension of the original feature map respectively through an attention module; determine the overlapping region of the first attention region and the second attention region as the attention selection region; generate a candidate feature matrix according to the original feature map and the attention selection region, wherein the candidate feature matrix has the same resolution as the original feature map; multiply the candidate feature matrix by the original feature map to generate a feature interaction matrix; perform normalization processing on the feature interaction matrix to obtain a two-dimensional attention weight matrix; perform a dot product operation on the original feature map and the two-dimensional attention weight matrix to obtain an enhanced feature matrix; and recognize unrecognized characters in the text according to the enhanced feature matrix and the recognized characters in the text.

[0015] In some embodiments of the present disclosure, when the computer program is executed by the at least one processor, the device is caused to determine the first attention region and the second attention region for the width dimension and the height dimension of the original feature map respectively through an attention module by the following operations: symmetrically generate a width continuous position interval and a height continuous position interval about the center point of the original feature map in the width dimension and the height dimension respectively; map the width continuous position interval to a width discrete position interval through a first embedding layer in the attention module; map the height continuous position interval to a height discrete position interval through a second embedding layer in the attention module; determine a width position offset of the width discrete position interval through a first fully connected layer in the attention module; determine a height position offset of the height discrete position interval through a second fully connected layer in the attention module; determine the first attention region according to the width discrete position interval and the width position offset; and determine the second attention region according to the height discrete position interval and the height position offset.

[0016] In some embodiments of the present disclosure, when the computer program is executed by the at least one processor, the device is caused to generate a candidate feature matrix according to the original feature map and the attention selection region by the following operations: generate a blank matrix having the same resolution as the original feature map; copy the elements corresponding to the attention selection region in the original feature map to the corresponding positions in the blank matrix; assign the boundary elements of the blank matrix to zero; and calculate the values of the unassigned regions in the blank matrix through linear interpolation.

[0017] In some embodiments of the present disclosure, when executed by at least one processor, a computer program causes a device to identify unrecognized characters in a text based on an enhanced feature matrix and recognized characters in the text by performing the following operations: taking each character in the unrecognized characters as a target character respectively, and performing the following operations on the target character: obtaining an embedding vector corresponding to the recognized characters in the text, where the embedding vector is the sum of the word vector of the recognized character and the position vector of the recognized character; generating a historical text vector from the embedding vector through a multi-head self-attention module; performing an interaction operation on the historical text vector and the enhanced feature matrix to generate a sequence decoding feature; predicting the classification probability of the target character according to the sequence decoding feature; and identifying the target character according to the classification probability.

[0018] According to a third aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program, wherein the computer program realizes the steps of the method according to the first aspect of the present disclosure when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To illustrate the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the following described drawings only relate to some embodiments of the present disclosure and do not limit the present disclosure, where:

[0020] Figure 1 is an exemplary flowchart of a text recognition method according to an embodiment of the present disclosure;

[0021] Figure 2 is an exemplary block diagram of a model for implementing the text recognition method according to an embodiment of the present disclosure;

[0022] Figure 3 is an exemplary schematic diagram of a process for determining an attention selection area in a picture;

[0023] Figure 4 is in Figure 1 is an exemplary flowchart of a process for generating a candidate feature matrix according to an original feature map and an attention selection area in the embodiment shown;

[0024] Figure 5 is in Figure 1 is an exemplary flowchart of a process for identifying unrecognized characters in a text according to an enhanced feature matrix and recognized characters in the text in the embodiment shown; and

[0025] Figure 6 is a schematic block diagram of a text recognition device according to an embodiment of the present disclosure.

[0026] It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. All other embodiments obtained by those skilled in the art based on the described embodiments of the present disclosure without creative efforts also fall within the scope of protection of the present disclosure.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the subject matter of the present disclosure pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal form unless clearly defined herein otherwise. Terms such as "first" and "second" are only used to distinguish one component (or a part of a component) from another component (or another part of a component).

[0029] Traditional text recognition methods usually group them into a word or a sequence by locating and predicting the probabilities of individual characters. Such methods may include: (1) Manually designing visual feature operators to extract information from an image to obtain visual features; (2) Using the obtained visual features for character-level classification and recognition with a classifier. The above methods perform excellently on scanned document images, but are limited by the expressive ability of the manually designed visual feature operators for images and usually cannot handle problems such as complex backgrounds and depth-of-view changes existing in bill images captured by handheld devices.

[0030] The development of deep learning technology has greatly promoted the development of scene text recognition. Deep learning-based methods regard text recognition as a sequence-to-sequence problem and usually use a Convolutional Neural Network (CNN for short) to automatically extract two-dimensional image visual features. After compressing the two-dimensional image visual features into one-dimensional sequence features, the one-dimensional sequence features are fed into a Recurrent Neural Network (RNN for short) to extract sequence context relationships. Finally, a decoding module based on the Connectionist Temporal Classification (CTC for short) loss function or a decoding module based on the attention mechanism is used to decode each character in the sequence. Among them, the CTC-based method uses pre-filled blank spaces to achieve the alignment of the predicted sequence and the target sequence, and uses the method of dynamic programming to find the maximum probability sum that conforms to the target sequence as the loss function of the maximum entropy, so as to achieve supervised training at the sequence level. The forward-backward algorithm of CTC is very complex to implement, which makes this type of method require a large amount of computation. In addition, CTC is difficult to apply to two-dimensional prediction. For non-horizontal distributed text, the recognition performance of this type of method is poor. The attention-based method usually combines the attention module with the RNN structure as the sequence prediction module. By referring to the history and encoded features of the target decoding sequence, the alignment between the input image and the output text sequence is learned, so as to predict the current character classification probability to form a predicted sequence. However, this type of method has the problem of attention drift, that is, the attention module cannot accurately associate each character prediction feature vector with the corresponding target area in the input image. Similar to CNN, the compression process from two-dimensional visual features to one-dimensional sequence features loses the local correlation of features in the two-dimensional space, and also loses the key information for locating the spatial position of the next character, making it difficult to process shopping receipt images with irregular shapes.

[0031] Embodiments of the present disclosure propose a text recognition method. This text recognition method can effectively recognize the text in shopping receipt images with irregular shapes. Figure 1 FIG. shows an exemplary flowchart of a text recognition method 100 according to an embodiment of the present disclosure. Figure 2 FIG. shows an exemplary block diagram of a model for implementing the text recognition method according to an embodiment of the present disclosure. The model for implementing the text recognition method 100 may include an encoder 210 (for example, having a Transformer structure) and a decoder 220. The role of the encoder 210 is to embed the image into the visual two-dimensional feature representation space, and the role of the decoder 220 is to map the visual features generated by the encoder and the context sequence features encoded by itself into sequence decoding features. The following combines Figure 1 and Figure 2 to describe the process of recognizing the text in the picture.

[0032] In method 100, at block S102, an image including text is obtained. The text includes multiple characters. In some embodiments of the present disclosure, the shapes of the characters in the image are irregular. In some embodiments of the present disclosure, the image is an irregularly shaped shopping receipt image. The image can be a real shopping receipt image captured by a handheld device. These shopping receipts can be from different brands and cover most common receipt formats. The characters in the image can include Chinese, English, and / or Japanese, etc. In some embodiments of the present disclosure, the characters included in the image are on the same line. In the case where the characters in the image are on multiple lines, the image can be cropped to obtain multiple images including single-line characters. In some embodiments of the present disclosure, the size of the image including single-line characters can be adjusted to 32×100.

[0033] In some embodiments of the present disclosure, the obtained images can be used to construct a training dataset. During the process of constructing the training dataset, the collected images can be annotated. The annotation content includes the position information of the text included in the image and its text information. The image can be cropped according to the position information of the text (usually represented in the form of four vertex coordinates or the length, width, and center point coordinates), and the cropped image is associated with the text information corresponding to the image to generate a training dataset.

[0034] At block S104, features of the image are extracted through a shallow convolutional neural network (shallow CNN) to generate an original feature map. The resolution of the original feature map is lower than the resolution of the image. This can reduce the attention to redundant low-level semantic information in the image during the text recognition process. As Figure 2 shown, the original feature map is input into the encoder 210. The Transformer structure of the encoder 210 can enhance the visual semantics of images with low resolution.

[0035] In some embodiments of the present disclosure, the shallow convolutional neural network can include, for example, 2 basic modules. Among them, each basic module includes: a convolutional layer, a batch normalization layer, a ReLU activation layer, and a max pooling layer. Among them, the kernel size of the convolutional layer is, for example, 3×3, and the kernel size of the max pooling layer is, for example, 2×2 and its stride is 2.

[0036] Figure 3 shows that an original feature map 303 can be generated from an image 301 through a shallow CNN. The original feature map 303 can be a matrix of W×H (8×8 in the Figure 3 example).

[0037] Back to Figure 1, at box S106, the first attention region and the second attention region are determined respectively for the width dimension and the height dimension of the original feature map through the attention module. In some embodiments of the present disclosure, the attention module is an adaptive two-dimensional spatial attention module. The adaptive two-dimensional spatial attention module includes two learnable network component blocks. For example, the first network component block is used for the width dimension of the original feature map. The second network component block is used for the height dimension of the original feature map. Each network block includes an embedding layer and a fully-connected neural network layer (which can also be simply referred to as a fully-connected layer). The embedding layer maps the selected continuous coordinate range to a discrete feature map scale space range. The fully-connected layer adds an additional position offset to the discrete spatial coordinates. The discretized coordinate represents the focus after the attention is dispersed. In some embodiments of the present disclosure, the number of hidden units of the adaptive two-dimensional spatial attention module can be set to 256.

[0038] In some embodiments of the present disclosure, a width continuous position interval and a height continuous position interval can be symmetrically generated about the center point of the original feature map in the width dimension and the height dimension respectively. In some embodiments of the present disclosure, the coordinates of the width continuous position interval can be expressed as (P w , Y w ), where, Y w ∈[0, H]. The coordinates of the height continuous position interval can be expressed as (X h , P h ), where, X h ∈[0, W], where, w represents the initial value of attention focus in the width dimension, h represents the initial value of attention focus in the height dimension, W represents the width of the original feature map, and H represents the height of the original feature map. w and h can be empirical values. Then, the width continuous position interval is mapped to a width discrete position interval through the first embedding layer of the first network component block. The height continuous position interval is mapped to a height discrete position interval through the second embedding layer of the second network component block. The width position offset of the width discrete position interval is determined through the first fully-connected layer of the first network component block. The height position offset of the height discrete position interval is determined through the second fully-connected layer of the second network component block. Then, the first attention region is determined according to the width discrete position interval and the width position offset, and the second attention region is determined according to the height discrete position interval and the height position offset.

[0039] In some embodiments of the present disclosure, the first attention region can be calculated as (X w , Y w ), where, X w = W w ×f e (P w ) + bw , Y w ∈ [0, H]. The second attention area can be calculated as (X h , γ h ), where X h ∈ [0, W], Y h = W h × f e (P h ) + b h . Among them, f e represents the coordinate embedding process, W w represents the weight matrix of the first fully connected layer, W h represents the weight matrix of the second fully connected layer, b w represents the width position offset, b h represents the height position offset. W w , W h , b w and b h can be obtained through training.

[0040] In the example of Figure 3 , the first attention area in the width dimension (shown as the shaded part in Feature Map 304) and the second attention area in the height dimension (shown as the shaded part in Feature Map 306) can be determined from the original feature map 303 through the adaptive two-dimensional spatial attention module. The elements in Feature Map 304 and Feature Map 306 are the same as those in the original feature map 303. The shaded part in Feature Map 304 corresponds to the character in the width dimension. In the example of Figure 3 , in the width dimension, assuming that the ":" in Picture 301 corresponds to the 3rd column element in Feature Map 304, and the "2" in Picture 301 corresponds to the 5th column element in Feature Map 304, then the 3rd column element and the 5th column element are included in the first attention area, while the 4th column element corresponding to the space between the ":" and "2" in Picture 301 is excluded from the first attention area. Similarly, in the height dimension, the characters in Picture 301 are located in the middle area, so the second attention area is located in the middle area of Feature Map 306.

[0041] In Figure 1 , at the box S108, the overlapping area of the first attention area and the second attention area is determined as the attention selection area. In the example of Figure 3 , the overlapping area of the shaded part in Feature Map 304 and the shaded part in Feature Map 306 can be determined as the attention selection area, that is, the shaded part in Feature Map 308. Feature Map 308 can correspond to the two-dimensional attention heat map of the original feature map.

[0042] In Figure 1At the box S110, a candidate feature matrix is generated based on the original feature map and the attention selection region. Among them, the resolution of the candidate feature matrix is the same as that of the original feature map. Figure 4 An exemplary flowchart showing the process of generating the candidate feature matrix.

[0043] At the box S402, a blank matrix with the same resolution as the original feature map is generated. When the resolution of the original feature map is W×H, a matrix with a size of W×H is generated, and each element in this matrix is zero.

[0044] At the box S404, the elements corresponding to the attention selection region in the original feature map are copied to the corresponding positions in the blank matrix. In Figure 3 the example of, the elements of the shaded part in the feature map 308 are copied to the corresponding positions in the blank matrix generated at the box S402. Here, the corresponding positions refer to the positions with the same coordinates.

[0045] At the box S406, the boundary elements of the blank matrix are assigned zero values. Here, the boundary elements refer to the elements in the first row, the last row, the first column, and the last column of the blank matrix.

[0046] At the box S408, the values of the elements in the unassigned area of the blank matrix are calculated by linear interpolation. Linear interpolation can smooth the transition from the area with values to the area with zero values.

[0047] After the above operations, a candidate feature matrix can be generated from the blank matrix according to the original feature map and the attention selection region. Compared with the original feature map, the elements corresponding to the characters in the candidate feature matrix are retained, and other elements are weakened.

[0048] Return to Figure 1 , at the box S112, the candidate feature matrix is multiplied by the original feature map to generate a feature interaction matrix. In the context, can be used to represent the candidate feature matrix, and F can be used to represent the original feature map.

[0049] At the box S114, the feature interaction matrix is normalized to obtain a two-dimensional attention weight matrix. In the context, S can be used to represent the two-dimensional attention weight matrix.

[0050]

[0051] Among them, f Softmax represents the Soffmax normalization operation. By using the two-dimensional spatial attention weight matrix, the adaptive two-dimensional spatial attention module can intuitively locate the position of the next character of the current character.

[0052] At block S116, a dot product operation is performed on the original feature map and the two-dimensional attention weight matrix to obtain an enhanced feature matrix. In the context, the enhanced feature matrix can be denoted as F'.

[0053] F' = S · F

[0054] In Figure 2 the example of, the enhanced feature matrix is input into the multi-head attention module in decoder 220 after being processed by the feed-forward layer. The feed-forward layer may include a ReLU activation function and a linear activation function.

[0055] After the above operations, the adaptive two-dimensional spatial attention module can extract the two-dimensional correlation information of image features and perform feature self-interaction enhancement, thereby alleviating the attention drift problem encountered when aligning pictures and texts during the decoding process.

[0056] At block S118, unrecognized characters in the text are recognized based on the enhanced feature matrix and the recognized characters in the text. In some embodiments of the present disclosure, each character in the unrecognized characters can be used as a target character respectively, and the operations shown in Figure 5 are performed on the target character.

[0057] At block S502, an embedding vector corresponding to the recognized characters in the text is obtained. The embedding vector is the sum of the word vector of the recognized character and the position vector of the recognized character. When recognizing the first character in the text, the recognized character is empty. The position vector PE can be generated according to the following formula.

[0058]

[0059]

[0060]

[0061] where, t represents the position of the recognized character in the entire text, k represents the k-th element in the position vector, and d model represents the dimension of the position vector.

[0062] At block S504, a historical text vector is generated from the embedding vector through the multi-head self-attention module. In Figure 2 the example of, in the training phase, a historical text vector can be generated from the embedding vector through a masked multi-head self-attention module. As described above, the pictures in the training dataset include annotation content. The annotation content can be input as the recognized characters. The unrecognized characters in the text information of the annotation content are masked. This can avoid the influence of the unrecognized characters in the annotation content on the decoding process through the mask.

[0063] At block S506, an interaction operation is performed on the historical text vector and the enhanced feature matrix to generate a sequence decoding feature. In Figure 2 the example of

[0064] Figure 2 At block S508, the classification probability of the target character is predicted based on the sequence decoding feature. In the example of

[0065] At block S510, the target character is identified based on the classification probability. In some embodiments of the present disclosure, the candidate character with the maximum probability may be determined as the target character.

[0066] Although not shown in Figure 2 the example of

[0067] Figure 6 FIG. shows a schematic block diagram of a text recognition device 600 according to an embodiment of the present disclosure. As Figure 6 shown, the text recognition device 600 may include a processor 610 and a memory 620 storing a computer program. When the computer program is executed by the processor 610, the text recognition device 600 is enabled to execute as Figure 1Steps of the method 100 shown. In one example, the text recognition device 600 can be a computer device or a cloud computing node. The text recognition device 600 can obtain a picture including text. The text includes multiple characters. The text recognition device 600 can extract features of the picture through a shallow convolutional neural network to generate an original feature map. Among them, the resolution of the original feature map is lower than the resolution of the picture. The text recognition device 600 can determine a first attention region and a second attention region for the width dimension and the height dimension of the original feature map respectively through an attention module. The text recognition device 600 can determine the overlapping region of the first attention region and the second attention region as the attention selection region. The text recognition device 600 can generate a candidate feature matrix according to the original feature map and the attention selection region. Among them, the resolution of the candidate feature matrix is the same as that of the original feature map. The text recognition device 600 can multiply the candidate feature matrix by the original feature map to generate a feature interaction matrix. The text recognition device 600 can perform normalization processing on the feature interaction matrix to obtain a two-dimensional attention weight matrix. The text recognition device 600 can perform a dot product operation on the original feature map and the two-dimensional attention weight matrix to obtain an enhanced feature matrix. The text recognition device 600 can identify unrecognized characters in the text according to the enhanced feature matrix and the recognized characters in the text.

[0068] In some embodiments of the present disclosure, the text recognition device 600 can symmetrically generate a width continuous position interval and a height continuous position interval about the center point of the original feature map in the width dimension and the height dimension respectively. The text recognition device 600 can map the width continuous position interval to a width discrete position interval through a first embedding layer in the attention module. The text recognition device 600 can map the height continuous position interval to a height discrete position interval through a second embedding layer in the attention module. The text recognition device 600 can determine the width position offset of the width discrete position interval through a first fully connected layer in the attention module. The text recognition device 600 can determine the height position offset of the height discrete position interval through a second fully connected layer in the attention module. The text recognition device 600 can determine the first attention region according to the width discrete position interval and the width position offset. The text recognition device 600 can determine the second attention region according to the height discrete position interval and the height position offset.

[0069] In some embodiments of the present disclosure, the text recognition device 600 can generate a blank matrix with the same resolution as the original feature map. The text recognition device 600 can copy the elements corresponding to the attention selection region in the original feature map to the corresponding positions in the blank matrix. The text recognition device 600 can assign the boundary elements of the blank matrix to zero. The text recognition device 600 can calculate the values of the elements in the unassigned region of the blank matrix through linear interpolation.

[0070] In some embodiments of the present disclosure, the text recognition device 600 may obtain the embedding vectors corresponding to the recognized characters in the text. The embedding vector is the sum of the word vector of the recognized character and the position vector of the recognized character. The text recognition device 600 may generate a historical text vector from the embedding vectors through a multi-head self-attention module. The text recognition device 600 may perform an interaction operation on the historical text vector and the enhanced feature matrix to generate a sequence decoding feature. The text recognition device 600 may predict the classification probability of the target character based on the sequence decoding feature. The text recognition device 600 may recognize the target character based on the classification probability.

[0071] In an embodiment of the present disclosure, the processor 610 may be, for example, a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a processor based on a multi-core processor architecture, etc. The memory 620 may be any type of memory implemented using data storage technology, including but not limited to random access memory, read-only memory, semiconductor-based memory, flash memory, disk memory, etc.

[0072] In addition, in an embodiment of the present disclosure, the text recognition device 600 may also include an input device 630, such as a microphone, a keyboard, a mouse, etc., for inputting pictures including text. Additionally, the text recognition device 600 may further include an output device 640, such as a loudspeaker, a display, etc., for outputting the recognized text.

[0073] In other embodiments of the present disclosure, there is also provided a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is capable of implementing the steps of the method as Figure 1 , Figure 4 and Figure 5 shown.

[0074] In summary, the text recognition method according to an embodiment of the present disclosure proposes an adaptive spatial two-dimensional attention mechanism. The adaptive spatial two-dimensional attention module obtains a two-dimensional attention weight matrix in the width and height dimensions through width-height embedding discretization, linear projection, and candidate feature interaction. The adaptive spatial two-dimensional attention module can utilize the two-dimensional attention weight matrix to extract two-dimensional correlation information of image features and perform feature self-interaction enhancement, thereby alleviating the attention drift problem encountered when aligning pictures and texts during the decoding process. Further, the text recognition method according to an embodiment of the present disclosure can use the two-dimensional attention weight matrix to locate the position of the next character according to the position order of the previous character. This method can effectively utilize the two-dimensional correlation of images to extract image information, better utilize the image features and semantic features of texts for sequence decoding, reduce the model calculation complexity, and improve the overall inference speed of the model. The text recognition method according to an embodiment of the present disclosure can accurately recognize texts of arbitrary shapes in shopping receipts, and requires fewer training parameters and lower calculation complexity.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of apparatuses and methods according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, and the module, the segment of the program, or the part of the instruction includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the block may occur in a different order than noted in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0076] Unless the context clearly indicates otherwise, the singular forms of words used in this specification and the appended claims include the plural, and vice versa. Thus, when referring to the singular, the corresponding plural is usually included. Similarly, the terms "comprising" and "including" will be interpreted as inclusive rather than exclusive. Likewise, the term "including" and "or" should be interpreted as inclusive, unless such an interpretation is explicitly prohibited in this specification. Where the term "example" is used in this specification, particularly when it is placed after a list of terms, the "example" is merely exemplary and explanatory, and should not be considered exclusive or extensive.

[0077] Further aspects and scope of adaptability will become apparent from the description provided herein. It should be understood that various aspects of the present application may be implemented alone or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of the present application.

[0078] The above has described several embodiments of the present disclosure in detail. However, obviously, those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the appended claims.

Claims

1. A text recognition method, comprising: Acquire a picture including text, wherein the text includes a plurality of characters; Extracting features of the image using a shallow convolutional neural network to generate an original feature map, wherein the resolution of the original feature map is lower than the resolution of the image; Determining, by an attention module, a first attention region and a second attention region for the width dimension and the height dimension of the original feature map respectively; determining an overlapping area of the first attention area and the second attention area as an attention selection area; generating a candidate feature matrix according to the original feature map and the attention selection area, wherein the candidate feature matrix has the same resolution as the original feature map; multiplying the candidate feature matrix by the original feature map to generate a feature interaction matrix; Normalizing the feature interaction matrix to obtain a two-dimensional attention weight matrix; Performing a dot product operation on the original feature map and the two-dimensional attention weight matrix to obtain an enhanced feature matrix; and identifying unrecognized characters in the text based on the enhanced feature matrix and recognized characters in the text; The determining of the first attention region and the second attention region for the width dimension and the height dimension of the original feature map by the attention module includes: Generating a width-continuous position interval and a height-continuous position interval symmetrically about the center point of the original feature map in the width dimension and the height dimension respectively; Mapping the width-continuous position interval to the width-discrete position interval through a first embedding layer in the attention module; Mapping the highly continuous position interval to a highly discrete position interval via a second embedding layer in the attention module; Determining the width position offset of the width discrete position interval by the first fully connected layer in the attention module; Determining the height position offset of the highly discrete position interval by a second fully connected layer in the attention module; determining the first attention area according to the width discrete position interval and the width position offset; and determining the second attention area according to the height discrete position interval and the height position offset; Generating a candidate feature matrix according to the original feature map and the attention selection area includes: Generate a blank matrix with the same resolution as the original feature map; Copying the elements in the original feature map corresponding to the attention selection area to corresponding positions in the blank matrix; Assigning zero to the boundary elements of the blank matrix; and The values of the elements in the unassigned regions of the blank matrix are calculated by linear interpolation.

2. The text recognition method according to claim 1, wherein: The width of the continuous position interval is calculated as (P w , Y w ),in, Y w ∈[0,H], the highly continuous position interval is calculated as (X h , P h ), where X h ∈[0,W], Wherein, w represents the initial value of attention in the width dimension, h represents the initial value of attention in the height dimension, W represents the width of the original feature map, and H represents the height of the original feature map; The first attention region is calculated as (X w , Y w ), where X w =W w ×f e (P w )+b w , Y w ∈[0,H], the second attention region is calculated as (X h , Y h ), where X h ∈[0,W],Y h =W h ×f e (P h )+b h , where f e represents the coordinate embedding process, W w represents the weight matrix of the first fully connected layer, W h represents the weight matrix of the second fully connected layer, b w Indicates the width position offset, b h Indicates the height position offset.

3. The text recognition method according to claim 1, wherein: The shallow convolutional neural network includes a basic module; The basic module includes: convolution layer, batch normalization layer, ReLU activation layer, and maximum pooling layer; The kernel size of the convolutional layer is 3×3, the kernel size of the maximum pooling layer is 2×2 and its stride is 2.

4. The text recognition method according to claim 3, wherein: The number of basic modules in the shallow convolutional neural network is 2.

5. The text recognition method according to claim 1, wherein: Recognizing unrecognized characters in the text according to the enhanced feature matrix and recognized characters in the text includes: Each of the unrecognized characters is used as a target character, and the following operations are performed on the target character: Obtaining an embedding vector corresponding to a recognized character in the text, the embedding vector being the sum of a word vector of the recognized character and a position vector of the recognized character; Generate a historical text vector from the embedding vector through a multi-head self-attention module; Performing interactive operations on the historical text vector and the enhanced feature matrix to generate sequence decoding features; Predicting the classification probability of the target character according to the sequence decoding features; and The target character is identified according to the classification probability.

6. The text recognition method according to claim 1, wherein: The shapes of the characters in the picture are irregular.

7. A text recognition device comprising: at least one processor; as well as at least one memory storing a computer program; Wherein, when the computer program is executed by the at least one processor, the text recognition device is enabled to perform the steps of the text recognition method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the computer program implements the steps of the text recognition method according to any one of claims 1 to 6.

Images