Multi-posture text recognition method, device and electronic equipment

Abstract

The present invention discloses a multi-pose text recognition method, device, and electronic device, which are applied to the field of recognition technology. The method extracts two-dimensional image features of a target text image through a feature extraction network; divides the two-dimensional image features into multiple feature segments of equal length; inputs the multiple feature segments of equal length into an encoder network created based on an attention mechanism for encoding, thereby obtaining feature segments; and fuses the encoded features of the multiple feature segments of equal length. The resulting fused encoded feature sequence is then input into a decoder network for decoding, thereby obtaining the text information in the target text image. The present invention achieves accurate recognition of multi-pose variable-length text.

Description

Technical Field

[0001] The embodiments of the present invention belong to the field of recognition technology, and in particular to a multi-gesture text recognition method, device, and electronic device. Background Art

[0002] When extracting editable text from images using OCR (Optical Character Recognition) technology, the text poses in the images are diverse. That is, the text layout and text form in natural scenes are diverse, and due to differences in shooting angles or target entities, the text length is also diverse.

[0003] After extracting the image's features, an RNN (Recurrent Neural Network) is used to predict the extracted features and derive editable text content based on the prediction results. However, the RNN model is prone to errors when recognizing variable-length text, resulting in low recognition accuracy for variable-length text. Summary of the Invention

[0004] In view of the technical problem that the existing technology has low accuracy in recognizing variable-length text, the embodiments of the present invention provide a multi-posture text recognition method, device and electronic device to improve the recognition accuracy of variable-length text in multi-posture text when recognizing multi-posture text from an image.

[0005] In a first aspect, an embodiment of the present invention provides a multi-pose text recognition method, comprising:

[0006] Acquire a target text image to be recognized, and extract two-dimensional image features of the target text image through a feature extraction network;

[0007] Cutting the two-dimensional image feature into a plurality of feature segments of equal length;

[0008] Inputting the plurality of feature segments of equal length into an encoder network for encoding, and obtaining encoding features corresponding to the feature segments, respectively, wherein the encoder network is established based on an attention mechanism;

[0009] The coding features of the multiple feature segments of equal length are fused, and the fused coding feature sequence obtained by fusion is input into a decoder network for decoding to obtain the text information in the target text image.

[0010] Optionally, the encoder network includes a two-dimensional attention network and a parallel attention network cascaded with the two-dimensional attention network, and the inputting the plurality of feature segments of equal length into the encoder network for encoding to obtain encoding features corresponding to the feature segments respectively includes:

[0011] fusing position codes for the plurality of feature segments of equal length to obtain a two-dimensional fused feature segment sequence;

[0012] Inputting the fused feature segment sequence into the two-dimensional attention network for self-attention calculation to obtain a two-dimensional first output segment sequence;

[0013] The first output segment sequence and the multiple feature segments of equal length are input into the parallel attention network for parallel encoding to obtain a second output segment sequence, where the second output segment sequence contains the encoding features corresponding to each feature segment.

[0014] Optionally, the two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer includes multiple self-attention units;

[0015] The step of inputting the fused feature segment sequence into the two-dimensional attention network for self-attention calculation to obtain a two-dimensional first output segment sequence includes:

[0016] The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

[0017] Optionally, sequentially processing the fused feature segment sequence through the multiple layers of two-dimensional attention layers to obtain the first output segment sequence includes:

[0018] Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer;

[0019] Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer;

[0020] Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

[0021] Optionally, before extracting the two-dimensional image features of the target text image through the feature extraction network, the method further includes:

[0022] The feature extraction network, the encoder network and the decoder network are jointly trained in advance using variable-length text image samples.

[0023] Optionally, the number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

[0024] In a second aspect, an embodiment of the present invention provides a multi-gesture text recognition method, which is applied to a server, comprising:

[0025] Upon receiving a target text image captured by a user device, extracting two-dimensional image features of the target text image through a feature extraction network;

[0026] Cutting the two-dimensional image feature into a plurality of feature segments of equal length;

[0027] Inputting the plurality of feature segments of equal length into an encoder network for encoding, and obtaining encoding features corresponding to the feature segments, respectively, wherein the encoder network is established based on an attention mechanism;

[0028] Fusing the coding features of the plurality of feature segments of equal length, and inputting the fused coding feature sequence obtained by fusion into a decoder network for decoding to obtain text information in the target text image;

[0029] Feedback the text information to the user equipment so that the user equipment outputs the text information.

[0030] In a third aspect, an embodiment of the present invention provides a multi-pose text recognition device, comprising:

[0031] An image acquisition unit is used to acquire a target text image to be recognized and extract two-dimensional image features of the target text image through a feature extraction network;

[0032] A feature segmentation unit, configured to segment the two-dimensional image feature into a plurality of feature segments of equal length;

[0033] an encoding unit, configured to input the plurality of feature segments of equal length into an encoder network for encoding, and obtain encoding features corresponding to the feature segments, respectively, wherein the encoder network is established based on an attention mechanism;

[0034] a feature fusion unit, configured to fuse the coding features of the plurality of feature segments of equal length;

[0035] The decoding unit is used to input the fused coded feature sequence into the decoder network for decoding to obtain the text information in the target text image.

[0036] Optionally, the encoder network includes a two-dimensional attention network and a parallel attention network cascaded with the two-dimensional attention network, and the encoding unit includes:

[0037] a position fusion subunit, configured to fuse position codes for the plurality of feature segments of equal length to obtain a two-dimensional fused feature segment sequence;

[0038] An attention calculation subunit, configured to input the fused feature segment sequence into the two-dimensional attention network to perform self-attention calculation, and obtain a two-dimensional first output segment sequence;

[0039] A parallel encoding unit is used to input the first output segment sequence and the multiple feature segments of equal length into the parallel attention network for parallel encoding to obtain a second output segment sequence, wherein the second output segment sequence contains the encoding features corresponding to each feature segment.

[0040] Optionally, the two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer includes multiple self-attention units; the attention calculation subunit is specifically used to:

[0041] The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

[0042] Optionally, the attention calculation subunit is specifically configured to include:

[0043] Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer;

[0044] Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer;

[0045] Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

[0046] Optionally, the device further comprises:

[0047] A training unit is used to jointly train the feature extraction network, the encoder network and the decoder network using variable-length text image samples in advance.

[0048] Optionally, the number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

[0049] In a fourth aspect, an embodiment of the present invention provides a multi-gesture text recognition device, applied to a server, characterized in that the device includes:

[0050] An image receiving unit, configured to receive a target text image captured by a user device;

[0051] A feature extraction unit, configured to extract two-dimensional image features of the target text image through a feature extraction network;

[0052] A feature segmentation unit, configured to segment the two-dimensional image feature into a plurality of feature segments of equal length;

[0053] an encoding unit, configured to input the plurality of feature segments of equal length into an encoder network for encoding, and obtain encoding features corresponding to the feature segments, respectively, wherein the encoder network is established based on an attention mechanism;

[0054] a feature fusion unit, configured to fuse the coding features of the plurality of feature segments of equal length;

[0055] A decoding unit, configured to input the fused coded feature sequence obtained by fusion into a decoder network for decoding to obtain text information in the target text image;

[0056] The text sending unit is used to feed back the text information to the user equipment so that the user equipment outputs the text information.

[0057] In a fifth aspect, an embodiment of the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect or the second aspect when executing the program.

[0058] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect or the second aspect.

[0059] The one or more technical solutions provided by the embodiments of the present invention achieve at least the following technical effects or advantages.

[0060] The embodiment of the present invention combines a feature extraction network with two-dimensional image feature segmentation to segment the two-dimensional image features extracted from the target text image by the feature extraction network into multiple feature segments of equal length, encodes the segmented feature segments using an encoder network of an attention mechanism, fuses the encoding features of the multiple feature segments of equal length, and inputs the fused fused encoding feature sequence into a decoder network for decoding to obtain text information in the target text image, instead of directly using the extracted two-dimensional image features for prediction. The encoding-decoding of the attention mechanism is effectively utilized to recognize multi-pose text, and the problem of poor support of the attention mechanism for variable length is effectively addressed, thereby improving the recognition accuracy of variable-length text images under the attention mechanism, and further realizing accurate recognition of multi-pose variable-length text. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0062] Figure 1 A schematic diagram of the structure of a text recognition model used in an embodiment of the present invention;

[0063] Figure 2 A flowchart of a multi-gesture text recognition method provided by an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of the structure of an encoder network based on the attention mechanism provided in an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of the structure of a two-dimensional attention network provided by an embodiment of the present invention;

[0066] Figure 5 for Figure 4 Schematic diagram of the structure of the self-attention unit of the two-dimensional attention network;

[0067] Figure 6 A schematic diagram of the functional modules of a multi-gesture text recognition device provided by an embodiment of the present invention;

[0068] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0070] In the embodiments of the present invention, the term "plurality" means "more than two", that is, including two or more than two situations; the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0071] An embodiment of the present invention provides a multi-gesture text recognition method for recognizing editable text information from a text image.

[0072] The multi-gesture text recognition method provided in embodiments of the present invention can be directly applied to user devices, such as reading pens and dictionary pens, or to servers, enabling multi-gesture text recognition based on interaction between the user device and the server. In embodiments of the present invention, the user device includes a camera module that captures an image of the target text. Specifically, the user device can be a smartphone, tablet computer, or smart wearable device.

[0073] It should be noted that if the multi-gesture text recognition method provided in an embodiment of the present invention is applied to a user device, the trained text recognition model needs to be stored on the user device. If the multi-gesture text recognition method provided in an embodiment of the present invention is applied to a server, the user device does not need to store the trained text recognition model, but the server stores the trained text recognition model.

[0074] Specifically, refer to Figure 1 As shown, the text recognition model stored in the user device or server is composed of a cascade of a feature extraction network, an encoder network created based on the attention mechanism, and a decoder network. The feature extraction network is used to extract two-dimensional image features from the target text image, wherein a segmentation layer is also included between the feature extraction network and the encoder network, which is used to segment the extracted two-dimensional image features into multiple feature segments of equal length; the encoder network encodes each feature segment to obtain corresponding encoding features, and a fusion layer is also included between the encoder network and the decoder network, which is used to fuse the encoding features after encoding multiple feature segments of equal length; the decoder network is used to decode the fused encoding feature sequence obtained by fusion to obtain the text information in the target text image.

[0075] Combined with reference Figure 1 and Figure 2 As shown, the multi-gesture text recognition method provided by the embodiment of the present invention includes the following steps

[0076] S201: Obtain a target text image to be recognized, and extract two-dimensional image features of the target text image through a feature extraction network.

[0077] Specifically, step S201 may be that after the user device collects the target text image to be recognized, the user device extracts the two-dimensional image features of the target text image through the feature extraction network, and the user device executes the subsequent steps To obtain the text information in the target text image.

[0078] Specifically, step S201 may also be performed by directly collecting the target text image to be recognized through the user device and uploading it to the server. After the server receives the target text image uploaded by the user device, the server extracts the two-dimensional image features of the target text image through the feature extraction network, and the server performs the subsequent steps. To obtain the text information in the target text image, the server feeds the obtained text information back to the user device.

[0079] In embodiments of the present invention, the feature extraction network is selected based on practical circumstances and may specifically be a convolutional neural network, such as a lightweight convolutional neural network or a classic convolutional neural network. To reduce the amount of image compression when extracting two-dimensional image features and avoid loss of detail information, the number of convolutional layers with a stride of 2 in the convolutional neural network is less than a preset threshold. Only two convolutional layers in the convolutional neural network are configured with a stride of 2; the remaining convolutional layers are configured with a stride of 1.

[0080] Specifically, the stride of the second and fourth convolutional layers is set to 2, and the stride of other convolutional layers is set to 1. This allows the length and width of the two-dimensional image features extracted from the target text image to be compressed to only 1 / 4 of the target text image. That is, assuming the width and height of the target text image are w and h respectively; the width and height of the two-dimensional image features are w / 4 and h / 4 respectively.

[0081] In specific implementation, the classic convolutional neural network can be modified by retaining the first two convolutional layers with an original stride of 2, and changing the stride of the remaining convolutional layers with an original stride of 2 to 1. For example, assuming that in the original convolutional neural network: the original stride of the 2nd, 4th, 6th, 8th, and 10th convolutional layers is 2, and the original stride of the 1st, 3rd, 5th, 7th, and 9th convolutional layers is 1, then the stride of the 6th, 8th, and 10th convolutional layers is modified to 1.

[0082] After the two-dimensional image features of the target text image are extracted in step S201, there is no need to convert the two-dimensional image features into a one-dimensional feature sequence. Instead, the extracted two-dimensional image features are directly applied to the step S202. This enables encoding and decoding directly in two-dimensional space, and can automatically learn weights to capture the hidden states of the encoder network and the decoder network, thereby implicitly modeling character-level features and the relationship between characters.

[0083] After step S201 , step S202 is performed: dividing the two-dimensional image features into a plurality of feature segments of equal length.

[0084] Since the feature extraction network extracts two-dimensional image features in a two-dimensional space, these features actually consist of feature maps from multiple channels. The encoder network then expands these input feature maps to produce a two-dimensional image feature sequence. This sequence is then segmented according to a preset number of segments, yielding multiple feature segments of equal length in the two-dimensional space. In practice, the specific value of the preset number of segments is determined based on actual conditions and is not a limitation here.

[0085] S203. Input the multiple feature segments of equal length into an encoder network for encoding, and obtain encoding features corresponding to each feature segment of the multiple feature segments of equal length, wherein the encoder network is constructed based on an attention mechanism.

[0086] Among them, the two-dimensional image feature sequence obtained by expanding the two-dimensional image features of multiple channels is m*c dimensional, where m is the total length of the two-dimensional image feature and c is the number of channels of the two-dimensional image feature sequence, that is, the number of channels of the feature map. The two-dimensional image feature sequence is then divided into multiple feature segments of equal length, with a shape of k*c. Each feature segment can be represented as {I1,I2,I3,……I k-2 ,I k-1 ,I k Each feature segment is input into the encoder network for encoding, and the encoding feature corresponding to each feature segment is obtained. Thus, the encoder network outputs the encoding feature G after each feature segment is encoded. i That is, the coding feature sequence {G1, G2, G3, ... G c-2 ,G c-1 ,G c}.

[0087] In an embodiment of the present invention, the encoder network used introduces a two-dimensional attention mechanism to achieve attention-based encoding in two-dimensional space for each feature segment segmented from a two-dimensional image feature sequence.

[0088] Specifically, in the two-dimensional space, the attention-based encoding Figure 3 As shown, the structure of the encoder network used in the embodiment of the present invention is a two-dimensional attention network and a parallel attention network cascaded with the two-dimensional attention network. Figure 1 As shown in Figure 2, the output of the two-dimensional attention network and the segmented feature segments of equal length are used as inputs to the parallel attention network.

[0089] Next, we will first introduce the structure of the two-dimensional attention network in an embodiment of the present invention:

[0090] refer to Figure 4As shown in Figure 2, the structure of the two-dimensional attention network uses multiple layers of two-dimensional attention layers based on the attention mechanism in series. Each two-dimensional attention layer contains multiple self-attention units. Figure 4 Each circle "0" in the figure represents a self-attention unit. In practice, the number of self-attention units in each two-dimensional attention layer matches the number of feature vectors in each feature segment. In practice, the self-attention unit can be a transfermer unit, such as the BERT (Bidirectional Encoder Representation from Transformers) unit. For more information, see the BERT architecture. Figure 5 As shown, no further details are given here.

[0091] Next, we describe the process of processing a two-dimensional image feature sequence using a two-dimensional attention network:

[0092] Step 1: For each feature segment i (i∈[1,k]) in the plurality of feature segments of equal length, position encoding is fused to form a position-sensitive fused feature segment F, thereby obtaining a fused feature segment sequence to be input into the two-dimensional attention network. Figure 4 As shown, each fusion feature segment F can be expressed as F = {F1, F2, F3, ... F k-2 ,F k-1 ,F k}.

[0093] Specifically, for each feature segment, the i-th feature vector in the feature segment is added to the position code fused with the i-th feature vector, where i is 1 to k in sequence, to obtain a position-sensitive fused feature segment = {F1, F2, F3, ... F k-2 ,F k-1 ,F k}, establish the order relationship between each feature vector in the feature segment, and thus establish the order relationship between character features. The reference of position coding is as follows:

[0094] If i is an odd number: If i is even:

[0095] Where i∈[1, k] represents the position index of the feature vector, p represents the feature dimension index of the feature segment, y is the position code, and k is the length of the feature segment.

[0096] Of course, in specific implementation, it is not limited to the above position encoding method, and absolute position encoding can also be used: directly randomly initialize a position vector for different feature vectors, add it to the feature vector, and obtain a fused feature segment.

[0097] Step 2: Input the fused feature fragment sequence into the two-dimensional attention network for self-attention calculation to obtain the first two-dimensional output fragment sequence.

[0098] Specifically, each fused feature segment in the fused feature segment sequence is sequentially processed by multiple layers of two-dimensional attention layers to obtain a first output segment sequence. The first output segment sequence represents the latent state of the character feature.

[0099] The following two-dimensional attention network includes reference Figure 4 Taking the three-layer two-dimensional attention layer shown in the figure as an example, the process of sequentially processing each fused feature fragment in the fused feature fragment sequence by multiple layers of two-dimensional attention layers is described in detail:

[0100] Step 2A: Input the fused feature segment sequence serially into the first two-dimensional attention layer, and perform self-attention calculation on each fused feature segment through each self-attention unit of the first two-dimensional attention layer;

[0101] Step 2B: Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer;

[0102] Step 2C: Perform self-attention calculation on the weighted output of the second two-dimensional attention layer through each self-attention unit of the third two-dimensional attention layer to obtain a first output fragment sequence.

[0103] For the i-th fused feature fragment, each self-attention unit of the first two-dimensional attention layer performs self-attention calculation on the i-th fused feature fragment; each self-attention unit of the second two-dimensional attention layer performs self-attention calculation on the weighted output of the first two-dimensional attention layer; each self-attention unit of the third two-dimensional attention layer performs self-attention calculation on the weighted output of the second two-dimensional attention layer, and obtains the first output fragment corresponding to the i-th fused feature fragment, i takes 1 to C in sequence, that is, the first output fragment containing the i-th fused feature fragment is obtained. A fused feature fragment sequence of fused feature fragments.

[0104] refer to Figure 4 , the first output segment is specifically represented as {O1, O2, O3, ... k-2 ,O k-1 ,O k}, where the self-attention calculation of each self-attention unit is combined Figure 5For the i-th self-attention unit in the first layer, it is necessary to obtain the query vector as input Key Vector Sum value vector V l i : Query vector by applying query Key Vector Sum value vector V l i The output of the self-attention unit is calculated by weighted sum operation, i∈[l,k], l takes 1, 2, 3. Among them, each self-attention unit query vector in the first two-dimensional attention layer The calculation depends on the feature vector F in the input fusion feature segment i , key vector Sum value vector V l i The calculation depends on the input fusion feature fragment F; each self-attention unit in the remaining two-dimensional attention layer: query vector The calculation of depends on the query vector corresponding to the i-th self-attention unit in the previous attention layer The key vector Sum value vector V l i The calculation of depends on the weighted sum output of each self-attention unit in the previous attention layer. is the query vector of the i-th self-attention unit in the l-th 2D attention layer, and its shape is l×c. and V l i are the key vector and the value vector, both of shape k × c.

[0105] In an optional implementation, the two-dimensional attention network always works in a serial manner and can be integrated with the RNN. The RNN uses the output of the two-dimensional attention network for serial encoding processing. However, the RNN using the output of the two-dimensional attention network for serial encoding processing is limited by the previous steps, and therefore the operation encoding speed is limited.

[0106] In order to improve the computing speed and thus accelerate the efficiency of text recognition, the present invention provides another optional implementation method, which uses a parallel attention network, that is, performing step 3 after step 2: inputting the first output fragment sequence and the multiple feature fragments of equal length into the parallel attention network for parallel encoding to obtain a second output fragment sequence, which contains the encoding features corresponding to each feature fragment.

[0107] Specifically, the number of output nodes of the parallel attention network is n, and the first output segment sequence 0 is k×c dimensional. The specific processing process is as follows:

[0108] The parallel attention network will output the weight matrix coefficient a through the following process,

[0109] a=softmax(W2tanh(W1O T ))

[0110] Among them, W1 and W2 are the network parameters of the parallel attention network, softmax and tanh are activation functions, and O T Represents the transpose of the first output segment sequence.

[0111] Next, based on the weight coefficient a and the multiple feature segments of equal length, each output node outputs the encoding features of the corresponding feature segment:

[0112]

[0113] Among them, i is the index of the output node, j is the feature vector index in the feature segment, G i is the encoding feature of the jth feature segment output by the i-th output node, a ij is the weight corresponding to the jth eigenvector of the i-th output node.

[0114] In the above embodiment, the output of the two-dimensional attention network is encoded in parallel with the help of a parallel attention network, which removes the dependency between the output nodes. The calculation result of the current output node does not need to depend on the calculation result of the previous moment, and can be encoded and calculated in parallel, thereby improving the calculation speed of the encoder network.

[0115] The embodiment of the present invention effectively implicitly models the relationship between character-level features and characters by introducing a two-dimensional attention network and a parallel attention network based on the attention mechanism, which helps to quickly and effectively recognize multi-pose text.

[0116] S204: Fusing the coding features of the plurality of feature segments of equal length, and inputting the fused coding feature sequence obtained by fusion into a decoder network for decoding to obtain text information in the target text image.

[0117] Specifically, feature segments belonging to the same feature map are fused to obtain corresponding fused coding features, and the fused coding features corresponding to multiple feature maps constitute a fused coding feature sequence.

[0118] In specific implementation, any existing feature fusion method can be used to fuse the coding features of each feature fragment from the same feature map. For example, the coding features of each feature fragment from the same feature map can be added or connected to obtain a fused coding feature sequence, which is not limited here.

[0119] Specifically, the encoded feature sequence {G1, G2, G3, ... G C-2 ,G C-1 ,G C} are fused to obtain the fused coding features corresponding to the two-dimensional image features. The fused coding features corresponding to the two-dimensional image features of multiple channels form a fused coding feature sequence to be input into the decoder network.

[0120] After the fused encoding feature sequence is input into the decoder network, the decoder network predicts character information based on the encoding feature sequence. Specifically, the decoder network comprises a two-dimensional attention network and a decoding module. The structure and operating principle of the two-dimensional attention network in the decoder network are identical to those in the encoder network, and for the sake of brevity, they are not further described here. The decoder network then uses the attention mechanism to decode the second output feature sequence, obtaining a character probability matrix; based on this character probability matrix, the text information is obtained.

[0121] Specifically, first, each output node of the decoder network predicts the probability of the corresponding output character in the following way:

[0122] P i =softmax(wG i +b)

[0123] Among them, P i is the encoding feature G i The probability of the output character, softmax is the activation function, w and b are the network learning parameters of the decoder network, and the encoding feature of the corresponding j-th feature segment output by the i-th output node.

[0124] Next, a character probability matrix is obtained according to the character probability output by each output node. Finally, the text information in the target text image is obtained based on the character probability matrix.

[0125] Through the above steps The technical solution is to first segment the extracted two-dimensional image features and then encode the feature segments, and then fuse the encoded features of each feature segment, rather than directly encoding the extracted two-dimensional image features. It can match the soft alignment features of the attention mechanism, thereby effectively solving the problem of inaccurate variable-length text recognition, and then, can improve the accuracy of text recognition.

[0126] In an optional embodiment, before extracting the two-dimensional image features of the target text image through the feature extraction network, the method further includes: pre-training a text recognition model formed by cascading the feature extraction network, the encoder network, and the decoder network using variable-length text image samples, thereby achieving a technical effect of more accurately recognizing characters in variable-length text images.

[0127] Based on the same inventive concept, the embodiment of the present invention provides a multi-gesture text recognition device, referring to Figure 6 Shown, including:

[0128] The image acquisition unit 601 is used to acquire a target text image to be recognized and extract two-dimensional image features of the target text image through a feature extraction network;

[0129] A feature segmentation unit 602 is used to segment the two-dimensional image features into a plurality of feature segments of equal length;

[0130] An encoding unit 603 is configured to input a plurality of feature segments of equal length into an encoder network for encoding, and obtain encoding features corresponding to the feature segments, wherein the encoder network is built based on an attention mechanism;

[0131] A feature fusion unit 604 is used to fuse the coding features of multiple feature segments of equal length;

[0132] The decoding unit 605 is used to input the fused coded feature sequence obtained by fusion into the decoder network for decoding to obtain the text information in the target text image.

[0133] In an optional embodiment, the encoder network includes a two-dimensional attention network and a parallel attention network cascaded with the two-dimensional attention network, and the encoding unit includes:

[0134] The position fusion subunit is used to fuse the position codes of multiple feature segments of equal length to obtain a two-dimensional fused feature segment sequence;

[0135] The attention calculation subunit is used to input the fused feature segment sequence into the two-dimensional attention network for self-attention calculation to obtain a two-dimensional first output segment sequence;

[0136] A parallel encoding unit is used to input the first output segment sequence and the multiple feature segments of equal length into a parallel attention network for parallel encoding to obtain a second output segment sequence, where the second output segment sequence contains the encoding features corresponding to each feature segment.

[0137] In an optional embodiment, the two-dimensional attention network uses multiple two-dimensional attention layers, wherein each two-dimensional attention layer includes multiple self-attention units; the attention calculation subunit is specifically used to:

[0138] The fused feature fragment sequence is sequentially processed through multiple layers of two-dimensional attention layers to obtain the first output fragment sequence.

[0139] In an optional embodiment, the attention calculation subunit is specifically configured to include:

[0140] Inputting the fused feature segment sequence into the first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer;

[0141] Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer;

[0142] Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

[0143] In an optional embodiment, the device further includes:

[0144] The training unit is used to jointly train the feature extraction network, encoder network and decoder network using variable-length text image samples in advance.

[0145] In an optional implementation, the number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset threshold.

[0146] The specific functions of the various units in the above-mentioned device have been described in detail in the embodiment of the multi-gesture text recognition method provided by the embodiment of the present invention, and will not be elaborated here.

[0147] Based on the same inventive concept, an embodiment of the present invention provides a multi-gesture text recognition device, which is applied to a server. The device includes:

[0148] An image receiving unit, configured to receive a target text image captured by a user device;

[0149] A feature extraction unit, configured to extract two-dimensional image features of a target text image through a feature extraction network;

[0150] A feature segmentation unit, used to segment the two-dimensional image features into multiple feature segments of equal length;

[0151] The encoding unit is used to input multiple feature segments of equal length into the encoder network for encoding, and obtain the encoding features corresponding to the feature segments respectively, wherein the encoder network is built based on the attention mechanism;

[0152] A feature fusion unit is used to fuse the encoding features of multiple feature segments of equal length;

[0153] A decoding unit is used to input the fused coded feature sequence into a decoder network for decoding to obtain the text information in the target text image;

[0154] The text sending unit is used to feed back text information to the user equipment so that the user equipment outputs the text information.

[0155] The specific functions of each intra-palace unit in the above-mentioned device have been described in detail in the embodiment of the multi-pose text recognition method provided in the embodiment of the present invention, and will not be elaborated here.

[0156] Based on the same inventive concept as the aforementioned multi-gesture text recognition method embodiment, the embodiment of the present invention further provides an electronic device, such as Figure 7 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0157] Reference Figure 7 , electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0158] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 702 may include one or more modules to facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.

[0159] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0160] The power component 706 provides power to the various components of the electronic device 700. The power component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and integrating power for the electronic device 700.

[0161] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0162] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0163] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0164] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect changes in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and temperature changes of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0165] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0166] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0167] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0169] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope of the present invention. The scope of the present invention is limited only by the appended claims. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A multi-pose text recognition method, characterized in that: include: Acquire a target text image to be recognized, and extract two-dimensional image features of the target text image through a feature extraction network; Expanding the feature maps of multiple channels contained in the two-dimensional image features to obtain a two-dimensional image feature sequence, and dividing the two-dimensional image feature sequence into multiple feature segments of equal length through a segmentation layer between the feature extraction network and the encoder network; fusing position codes for the plurality of feature segments of equal length to obtain a two-dimensional fused feature segment sequence; Inputting the fused feature segment sequence into the two-dimensional attention network of the encoder network for self-attention calculation to obtain a two-dimensional first output segment sequence; Inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding to obtain a second output segment sequence, wherein the second output segment sequence includes encoding features corresponding to each feature segment; wherein the encoder network is established based on the attention mechanism, the two-dimensional attention network is cascaded with the parallel attention network, and inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding includes: obtaining a weight matrix coefficient a according to a network parameter of the parallel attention network, and outputting encoding features of the corresponding feature segments based on the weight matrix coefficient a and the multiple feature segments of equal length; The coding features of the multiple feature segments of equal length are fused, and the fused coding feature sequence obtained by fusion is input into the decoder network for decoding to obtain the text information in the target text image, wherein the fused coding feature is obtained by fusion of feature segments belonging to the same feature map.

2. The method according to claim 1, wherein The two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer contains multiple self-attention units; The step of inputting the fused feature segment sequence into the two-dimensional attention network for self-attention calculation to obtain a two-dimensional first output segment sequence includes: The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

3. The method according to claim 2, wherein The sequentially processing the fused feature segment sequence through the multi-layer two-dimensional attention layer to obtain the first output segment sequence includes: Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer; Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer; Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

4. The method according to any one of claims 1 to 3, wherein: Before extracting the two-dimensional image features of the target text image through the feature extraction network, the method further includes: The feature extraction network, the encoder network and the decoder network are jointly trained in advance using variable-length text image samples.

5. The method according to any one of claims 1 to 3, wherein: The number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

6. A multi-gesture text recognition method, applied to a server, characterized in that: The method comprises: Upon receiving a target text image captured by a user device, extracting two-dimensional image features of the target text image through a feature extraction network; Expanding the feature maps of multiple channels contained in the two-dimensional image features to obtain a two-dimensional image feature sequence, and dividing the two-dimensional image feature sequence into multiple feature segments of equal length through a segmentation layer between the feature extraction network and the encoder network; fusing position codes for the plurality of feature segments of equal length to obtain a two-dimensional fused feature segment sequence; Inputting the fused feature segment sequence into the two-dimensional attention network of the encoder network for self-attention calculation to obtain a two-dimensional first output segment sequence; Inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding to obtain a second output segment sequence, wherein the second output segment sequence includes encoding features corresponding to each feature segment, wherein the encoder network is created based on the attention mechanism, and the two-dimensional attention network is cascaded with the parallel attention network; inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding includes: obtaining a weight matrix coefficient a according to a network parameter of the parallel attention network, and outputting encoding features of the corresponding feature segments based on the weight matrix coefficient a and the multiple feature segments of equal length; Fusing the coding features of the plurality of feature segments of equal length, and inputting the fused coding feature sequence obtained by fusion into a decoder network for decoding, thereby obtaining text information in the target text image, wherein the fused coding feature is obtained by fusing feature segments belonging to the same feature map; Feedback the text information to the user equipment so that the user equipment outputs the text information.

7. The method according to claim 6, wherein The two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer contains multiple self-attention units; The step of inputting the fused feature segment sequence into the two-dimensional attention network for self-attention calculation to obtain a two-dimensional first output segment sequence includes: The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

8. The method according to claim 7, wherein The sequentially processing the fused feature segment sequence through the multi-layer two-dimensional attention layer to obtain the first output segment sequence includes: Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer; Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer; Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

9. The method according to any one of claims 6 to 8, wherein: Before extracting the two-dimensional image features of the target text image through the feature extraction network, the method further includes: The feature extraction network, the encoder network and the decoder network are jointly trained in advance using variable-length text image samples.

10. The method according to any one of claims 6 to 8, wherein: The number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

11. A multi-gesture text recognition device, characterized in that: include: An image acquisition unit is used to acquire a target text image to be recognized and extract two-dimensional image features of the target text image through a feature extraction network; a feature segmentation unit, configured to expand the feature maps of multiple channels contained in the two-dimensional image features to obtain a two-dimensional image feature sequence, and segment the two-dimensional image feature sequence into a plurality of feature segments of equal length via a segmentation layer between the feature extraction network and the encoder network; The encoding unit includes: a position fusion subunit, which is used to fuse the position codes for the multiple feature segments of equal length to obtain a two-dimensional fused feature segment sequence; an attention calculation subunit, which is used to input the fused feature segment sequence into the two-dimensional attention network of the encoder network for self-attention calculation to obtain a two-dimensional first output segment sequence; a parallel encoding unit, which is used to input the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding to obtain a second output segment sequence, wherein the second output segment sequence contains the encoding features corresponding to each feature segment, wherein the encoder network is created based on the attention mechanism, and the two-dimensional attention network is cascaded with the parallel attention network; the step of inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding includes: obtaining a weight matrix coefficient a according to the network parameters of the parallel attention network, and outputting the encoding features of the corresponding feature segments based on the weight matrix coefficient a and the multiple feature segments of equal length; a feature fusion unit, configured to fuse the coding features of the plurality of feature segments of equal length; The decoding unit is used to input the fused coded feature sequence obtained by fusion into the decoder network for decoding to obtain the text information in the target text image, wherein the fused coded feature is obtained by fusion of feature segments belonging to the same feature map.

12. The device according to claim 11, wherein The two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer contains multiple self-attention units; The attention calculation subunit is specifically used to: The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

13. The device according to claim 12, wherein The attention calculation subunit is specifically used to: Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer; Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer; Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

14. The device according to any one of claims 11 to 13, characterized in that The device further comprises: A training unit is used to jointly train the feature extraction network, the encoder network and the decoder network using variable-length text image samples in advance.

15. The device according to any one of claims 11 to 13, characterized in that The number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

16. A multi-gesture text recognition device, applied to a server, characterized in that: The device comprises: An image receiving unit, configured to receive a target text image captured by a user device; A feature extraction unit, configured to extract two-dimensional image features of the target text image through a feature extraction network; a feature segmentation unit, configured to expand the feature maps of multiple channels contained in the two-dimensional image features to obtain a two-dimensional image feature sequence, and segment the two-dimensional image feature sequence into a plurality of feature segments of equal length via a segmentation layer between the feature extraction network and the encoder network; The encoding unit includes: a position fusion subunit, which is used to fuse position codes for the multiple feature segments of equal length to obtain a two-dimensional fused feature segment sequence; an attention calculation subunit, which is used to input the fused feature segment sequence into the two-dimensional attention network of the encoder network for self-attention calculation to obtain a two-dimensional first output segment sequence; a parallel encoding unit, which is used to input the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding to obtain a second output segment sequence, wherein the second output segment sequence contains encoding features corresponding to each feature segment, wherein the encoder network is created based on the attention mechanism, and the two-dimensional attention network is cascaded with the parallel attention network; the step of inputting the first output segment sequence and the multiple feature segments of equal length into the parallel attention network of the encoder network for parallel encoding includes: obtaining a weight matrix coefficient a according to the network parameters of the parallel attention network, and outputting encoding features of the corresponding feature segments based on the weight matrix coefficient a and the multiple feature segments of equal length; a feature fusion unit, configured to fuse the coding features of the plurality of feature segments of equal length; A decoding unit, configured to input the fused coded feature sequence obtained by fusion into a decoder network for decoding to obtain text information in the target text image; The text sending unit is used to feed back the text information to the user equipment so that the user equipment outputs the text information. The fused coding feature is obtained by fusing feature segments belonging to the same feature graph.

17. The device according to claim 16, wherein The two-dimensional attention network adopts multiple two-dimensional attention layers, wherein each two-dimensional attention layer contains multiple self-attention units; The attention calculation subunit is specifically used to: The fused feature segment sequence is sequentially processed through the multi-layer two-dimensional attention layer to obtain the first output segment sequence.

18. The device according to claim 17, wherein The attention calculation subunit is specifically used to: Inputting the fused feature segment sequence into a first two-dimensional attention layer, and performing self-attention calculation on the fused feature segment sequence through the first two-dimensional attention layer; Perform self-attention calculation on the weighted output of the first two-dimensional attention layer through each self-attention unit of the second two-dimensional attention layer; Through each self-attention unit of the third two-dimensional attention layer, self-attention calculation is performed on the weighted output of the second two-dimensional attention layer to obtain the first output fragment sequence.

19. The device according to any one of claims 16 to 18, characterized in that The device further comprises: The feature extraction network, the encoder network and the decoder network are jointly trained in advance using variable-length text image samples.

20. The device according to any one of claims 16 to 18, characterized in that The number of convolutional layers with a stride of 2 in the feature extraction network is less than a preset number threshold.

21. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 10 when executing the program.

22. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images