Character recognition method, device and electronic device based on progressive coding

Through the progressive encoding-decoding network model of cascaded encoding module and decoding module, image features are extracted layer by layer and recognition results are integrated, which solves the problem of low accuracy in word recognition in the prior art, and improves the contaminated characters and high frequency words recognition ability.

CN112712079BActive Publication Date: 2024-11-12BEIJING SANKUAI ONLINE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110019931.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-07
Publication Date
2024-11-12
Estimated Expiration
2041-01-07

AI Technical Summary

Technical Problem

When characters are contaminated or words with high frequency appear in the image, the recognition accuracy is reduced and it relies too much on context information, resulting in inaccurate recognition results.

Method used

Using a progressive encoding method, visual features and semantic features are extracted layer by layer through the encoding-decoding network model of at least three encoding modules and corresponding decoding modules, and decoded by decoding modules, and finally recognized by fusion. The unit determines the text recognition result.

Benefits of technology

It improves the recognition accuracy of words in the image, reduces the impact of background noise, and improves the recognition ability of high-frequency words.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112712079B_ABST
    Figure CN112712079B_ABST
Patent Text Reader

Abstract

The present application discloses a text recognition method based on progressive coding, which belongs to the field of computer technology and helps to improve the accuracy of text recognition in images. The text recognition method includes: pre-training an encoding-decoding network model including: at least three cascaded encoding modules, and a decoding module corresponding to each encoding module, and in the prediction stage, sequentially executing the at least three encoding modules, performing progressive depth feature encoding on the target image, and obtaining the encoding output of each encoding module respectively; executing each decoding module respectively, decoding the encoding output of the corresponding encoding module, and obtaining the text recognition result of each decoding module on the target image; fusing the text recognition results of each decoding module on the target image, determining the text recognition result of the encoding-decoding network model on the target image, and improving the accuracy of text recognition in the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a text recognition method, device, electronic device and computer-readable storage medium based on progressive coding. Background Art

[0002] As a common technology in machine vision, text recognition is widely used in application scenarios such as document analysis, license recognition, street scene recognition, and automatic information entry. In the prior art, the mainstream methods of text recognition are all composed of encoding-decoding structures. Among them, the encoding process is to convert the input image information into a representation vector; the decoding process is to convert the representation vector generated by the encoding process into an output target sequence. In the prior art, the encoding process is usually implemented using a convolutional neural network, or a network structure of a convolutional neural network + a recursive neural network. Correspondingly, the decoding process is implemented using a sequence connection classification decoding technology or a decoding technology based on an attention mechanism to recognize contextual semantics by sequence-to-sequence decoding.

[0003] The text recognition methods in the prior art have at least the following defects: 1. The encoding-decoding process method of convolutional neural network + sequence connection classification decoding technology only considers the visual features of the image. If a character in the image is contaminated (such as occlusion), the recognition accuracy will be greatly reduced; 2. The encoding-decoding process method of convolutional neural network + recursive neural network + self-attention mechanism is too dependent on contextual information. For example, if the frequency of occurrence of a word or sentence in the training sample is too high, the words similar to the word will be recognized as too sensitive to the distribution of training data, which will also reduce the recognition accuracy and recognize the word or sentence with too high frequency.

[0004] In summary, the text recognition method in the prior art still needs to be improved. Summary of the invention

[0005] The embodiment of the present application provides a text recognition method based on progressive encoding, which helps to improve the accuracy of text recognition in images.

[0006] In a first aspect, an embodiment of the present application provides a text recognition method based on progressive encoding, wherein a pre-trained encoding-decoding network model comprises: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules, and the method comprises:

[0007] Executing the at least three encoding modules in sequence, performing feature encoding of a target image with progressive depth, and obtaining encoding outputs of each encoding module respectively;

[0008] Executing each of the decoding modules respectively, decoding the coded output of the corresponding coding module, and obtaining the text recognition results of each decoding module on the target image;

[0009] The text recognition results of the target image by each decoding module are fused to determine the text recognition result of the target image by the encoding-decoding network model.

[0010] In a second aspect, an embodiment of the present application provides a text recognition device based on progressive encoding, wherein a pre-trained encoding-decoding network model includes: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules, and the device includes:

[0011] An encoding control unit, configured to sequentially execute the at least three encoding modules, perform feature encoding of progressive depth on the target image, and obtain encoding outputs of each encoding module respectively;

[0012] A decoding control unit, used to execute each of the decoding modules respectively, decode the coded output of the corresponding coding module, and obtain the text recognition result of each decoding module on the target image;

[0013] The fusion recognition unit is used to fuse the text recognition results of the target image by each decoding module to determine the text recognition result of the encoding-decoding network model for the target image.

[0014] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the text recognition method based on progressive coding described in the embodiment of the present application is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the text recognition method based on progressive coding disclosed in an embodiment of the present application.

[0016] The text recognition method based on progressive coding disclosed in the embodiment of the present application is pre-trained, including: at least three cascaded coding modules, and an encoding-decoding network model of a decoding module corresponding to each of the coding modules. In the prediction stage, the at least three coding modules are executed in sequence to perform progressive depth feature encoding on the target image to obtain the encoding output of each of the coding modules respectively; each of the decoding modules is executed respectively to decode the encoding output of the corresponding coding module to obtain the text recognition result of each of the decoding modules on the target image; the text recognition results of each of the decoding modules on the target image are fused to determine the text recognition result of the encoding-decoding network model on the target image, which helps to improve the accuracy of recognizing text in the image.

[0017] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Figure 1 is a flow chart of a text recognition method based on progressive coding according to the first embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the structure of the encoding-decoding network model in Example 1 of the present application;

[0021] Figure 3 is a schematic diagram of a training sample image of Embodiment 1 of the present application;

[0022] Figure 4 This is a schematic diagram of the second coding network structure of Embodiment 1 of the present application;

[0023] Figure 5 is a structural diagram of a text recognition device based on progressive coding according to Embodiment 2 of the present application;

[0024] Figure 6 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0025] Figure 7 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Embodiment 1

[0028] The embodiment of the present application discloses a text recognition method based on progressive coding, such as Figure 1 As shown, the method includes: steps 110 to 130.

[0029] The text recognition method based on progressive encoding described in the embodiment of the present application is implemented by executing a pre-trained encoding-decoding network model, wherein the encoding network of the pre-trained encoding-decoding network model includes: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules. Figure 2 Taking the encoding-decoding network model shown as an example, the encoding-decoding network model includes: an encoding module group 210, a decoding module group 220, and a fusion recognition module 230. The encoding module group 210 further includes three cascaded encoding modules, namely: a first encoding module 2101, a second encoding module 2102, and a third encoding module 2103; the decoding module group 220 further includes three parallel decoding modules, namely: a first decoding module 2201 corresponding to the first encoding module 2101, a second decoding module 2202 corresponding to the second encoding module 2102, and a third decoding module 2203 corresponding to the third encoding module 2103. A group of encoding modules and decoding modules corresponding to each other constitute an encoding-decoding network branch of the encoding-decoding network model, which is used to perform feature encoding and decoding processing of different depths on the image input to the encoding-decoding network model; the fusion recognition module 230 is used to perform fusion processing on the output results of all the encoding-decoding network branches to obtain a text recognition result or text prediction result of the image input to the encoding-decoding network model.

[0030] In some embodiments of the present application, the encoding-decoding network model is trained by the following method: for each training sample image, the following encoding and decoding operations are performed respectively: the at least three encoding modules are executed in sequence, and the feature encoding of the training sample image with progressive depth is performed, and the encoding output of each encoding module for the training sample image is obtained respectively; each decoding module is executed respectively, and the encoding output of the training sample image by the corresponding encoding module is decoded to obtain the text prediction result of each decoding module for the training sample image; for each training sample image, the prediction error of the training sample image is calculated according to the difference between the text prediction result of each decoding module for the training sample image and the true value of the text corresponding to the training sample image, and the loss value of the encoding-decoding network model is determined according to the prediction errors of all the training sample images; by optimizing the model parameters of the encoding-decoding network model, the encoding-decoding network model is iteratively trained to optimize the loss value.

[0031] The training sample image described in the embodiment of the present application is a text image annotated with a true value of the text, and the true value of the text can be used as a sample label of the training sample image.

[0032] In the process of training the encoding-decoding network model, the encoding-decoding network model learns the encoding mapping relationship between each training sample image and the text true value corresponding to the training sample image by encoding and decoding all the training sample images, thereby learning the optimal network parameters of each encoding module and decoding module in the encoding-decoding network model for subsequent online image text recognition applications. The encoding-decoding network model performs encoding and decoding operations on each training sample image respectively to obtain the text prediction results corresponding to each training sample image, and then calculates the loss value of the encoding-decoding network model according to the error between the text prediction results of all training sample images and the text true value as the sample label; then, by adjusting the network model parameters in each encoding module and decoding module of the encoding-decoding network model, iteratively trains the encoding-decoding network model, optimizes the loss value of the encoding-decoding network model, until the loss value reaches a preset value, or satisfies the preset iterative training termination condition, ends the training process of the encoding-decoding network model, and applies the encoding-decoding network model to the online image text recognition task.

[0033] The process of parameter adjustment and iterative training of the encoding-decoding network model can be referred to in the prior art and will not be described in detail in the embodiments of the present application.

[0034] In some embodiments of the present application, the loss value of the encoding-decoding network model can be obtained by fusion calculation based on the loss values ​​of all encoding-decoding network branches in the encoding-decoding network model. Figure 2 The encoding-decoding network model shown, the loss value of the encoding-decoding network model can be obtained according to the weighted sum of the loss value of the first network branch composed of the first encoding module and the first decoding module, the loss value of the second network branch composed of the second encoding module and the second decoding module, and the loss value of the third network branch composed of the third encoding module and the third decoding module. Among them, the loss value of the first network branch composed of the first encoding module and the first decoding module is calculated based on the error between the decoding result of the current training sample image of the encoding-decoding network model by the first decoding module (that is, the text prediction result of the current training sample image) and the sample label of the current training sample image (that is, the true value of the text of the current training sample image); the loss value of the second network branch composed of the second encoding module and the second decoding module is calculated based on the error between the decoding result of the current training sample image of the encoding-decoding network model by the second decoding module (that is, the text prediction result of the current training sample image) and the sample label of the current training sample image (that is, the true value of the text of the current training sample image); the loss value of the third network branch composed of the third encoding module and the third decoding module is calculated based on the error between the decoding result of the current training sample image of the encoding-decoding network model by the third decoding module (that is, the text prediction result of the current training sample image) and the sample label of the current training sample image (that is, the true value of the text of the current training sample image).

[0035] In some embodiments of the present application, the inverse of the similarity between the true value of the text and the text prediction result can be used as the loss value of each network branch. In other embodiments of the present application, other methods can be used to calculate the loss value of the network branch based on the true value of the text and the text prediction result. The present application does not limit the specific implementation method of calculating the loss value of each network branch.

[0036] For each of the aforementioned network branches, the encoding operation and decoding operation are related and corresponding. The following describes the specific scheme for each network branch to obtain text prediction results from the two operation processes of encoding operation and decoding operation, combined with the structure of each encoding module and decoding module.

[0037] In some embodiments of the present application, the at least three encoding modules are executed in sequence to perform feature encoding of the training sample image with progressive depth, and the encoding output of the training sample image by each encoding module is obtained respectively, including: encoding step S1 to encoding step S3, wherein the encoding step S1 to encoding step S3 gradually deepens the hierarchical depth of feature extraction of the training sample image.

[0038] The encoding step S1 is to execute the first encoding module to encode the visual features in the training sample image and output a first feature vector corresponding to the training sample image.

[0039] The first encoding module is used to extract visual features from the input image to obtain the first feature vector of the current input image of the encoding-decoding network model, and the first feature vector represents the visual features of the current input image, that is, the image features. For example, in the model training phase, the visual features of the training sample images input to the encoding-decoding network model are extracted by executing the program code of the first encoding module; in the testing phase, the visual features of the target image input to the encoding-decoding network model are extracted by executing the program code of the first encoding module. The visual features may include, for example, key image information such as text contours in the current input image.

[0040] In some embodiments of the present application, the first encoding module is constructed based on a convolutional neural network. The first encoding module includes: a convolutional layer, a pooling layer, and a fully connected layer. Among them, the step size of the pooling layer is generally transformed to [2,1] to maintain a higher resolution in the horizontal direction. Among them, the first encoding module can adopt any of the following typical convolutional neural network structures: VGG (a new deep convolutional neural network developed by researchers from the Visual Geometry Group of the University of Oxford and Google DeepMind), ResNet (residual network) or InceptionNet (a classic convolutional neural network), etc. In some embodiments of the present application, an attention module can be added to the convolutional neural network, such as CBAM (Convolutional Block Attention Module, which represents the attention mechanism module of the convolution module, which is an attention mechanism module that combines space and channels), BAM (Bottleneck Attention Module, bottleneck attention module) or SE block (Sequeze and Excitation block, a commonly used attention module) to enhance the feature extraction capability of the convolutional neural network.

[0041] The encoding step S2 executes the second encoding module to perform feature alignment processing on the first feature vector corresponding to the training sample image, and outputs the second feature vector corresponding to the training sample image.

[0042] The input of the second encoding module is connected to the output of the first encoding module, and is used to align the visual features output by the first encoding module. Figure 3Taking the training sample image shown as an example, the training sample image usually includes a text image and a background image (for example, the gap between words or the gap between the upper and lower edges of the words and the edge of the image). In order to reduce the interference of the noise introduced by the background image on the model training, in some embodiments of the present application, the high-level image features of the training samples extracted by the first encoding module are aligned by the second encoding module.

[0043] In some embodiments of the present application, the second encoding module 2102 further includes: a self-attention module for deleting residual links, such as Figure 4 As shown, the self-attention module with residual links removed further includes: a multi-head attention network 21021, a normalization layer 21022, a feed-forward network 21023 with residual links removed, and a normalization layer 21024 connected in sequence, wherein the execution of the second encoding module performs feature alignment processing on the first feature vector corresponding to the training sample image, and outputs the second feature vector corresponding to the training sample image, including: aligning the vector representing the visual features of the text area in the current input image and the vector representing the visual features of the non-text area in the first feature vector corresponding to the current input image through the second encoding module, and outputting the second feature vector after alignment processing of the current input image. Wherein, the current input image is the training sample image; the feed-forward network 21023 is a typical feed-forward network structure after residual links are removed in the prior art, the normalization layers 21022 and 21024 adopt the typical normalization network structure in the prior art, and the multi-head attention network 21021 adopts an attention network with residual links removed. The multi-head attention network 21021 converts the first feature vector into a feature vector in an aligned form, i.e., a second feature vector, through the attention weight matrix learned in the training phase. In the second feature vector obtained after the alignment process, the vectors representing the visual features of the text area (such as a text block image) in the training sample image are aligned within a specified dimensional range, which is convenient for subsequent visual feature extraction of the text in the decoding phase, and is also convenient for filtering out noise in the second feature vector in the decoding phase.

[0044] The self-attention module that deletes the residual link can prevent low-level background information from being transmitted to high-level features, effectively reducing or even removing the noise introduced by the background part of the image during the training process, and can further improve the accuracy of text recognition in the image.

[0045] The encoding step S3 is to execute the third encoding module, extract semantic features from the second feature vector corresponding to the training sample image, and output a third feature vector corresponding to the training sample image.

[0046] In some embodiments of the present application, the third encoding module is implemented based on a self-attention network. The self-attention mechanism generally adopts a multi-head encoding method, that is, multi-head self-attention encoding. The self-attention process is repeated multiple times and combined together, and the encoding representation of the feature information is further deepened by providing multiple "representation subspaces" of the attention layer. In some embodiments of the present application, by adopting a network with multiple attention heads, that is, by multiple query / key / value weight matrix sets, the semantic association between the characters in the feature sequence input to the third encoding module is learned. Each weight in the query / key / value weight matrix set is randomly initialized, and after training, each set is projected into a different representation subspace. When the model encodes or decodes the current text image (such as a training sample image), the self-attention mechanism can help the model view other positions in the input sequence and find relevant clues to achieve better encoding effects. By adopting a self-attention network, when encoding the current text image, information of other related characters can be incorporated to improve the accuracy of the decoding result.

[0047] For each training sample image, during the encoding operation, after being encoded by the first encoding module, the second encoding module, and the third encoding module respectively, the first feature vector, the second feature vector, and the third feature vector of different levels of depth are obtained; then, during the decoding operation, the program codes of the first decoding module, the second decoding module, and the third decoding module are executed in parallel to perform decoding operations of feature vectors at each level.

[0048] Each decoding module described in the embodiment of the present application corresponds to a coding module. For example, the first decoding module corresponds to the first coding module, the second decoding module corresponds to the second coding module, and the third decoding module corresponds to the third coding module. The decoding module decodes the feature vector obtained after encoding the corresponding coding module by adopting a network structure matching the corresponding coding module, and obtains the text prediction result of the current input image (such as the training sample image) of the coding-decoding network model by the coding-decoding network branch composed of each group of coding modules and decoding modules.

[0049] The decoding module described in the embodiments of the present application can simultaneously decode the encoding results of different levels of depth of a certain training sample image, or can start to perform decoding operations on the encoding results after the corresponding encoding module outputs the encoding results. In some embodiments of the present application, the execution of each decoding module separately, decoding the encoding output of the training sample image by the corresponding encoding module, and obtaining the text prediction results of the training sample image by each decoding module includes: decoding steps S4 to decoding step 6, each decoding step can be executed after the corresponding encoding step is executed.

[0050] Decoding step S4, executing the first decoding module, decoding the first feature vector corresponding to the training sample image, and obtaining a first text prediction result.

[0051] The first decoding module is used to decode the high-level image features obtained after the convolutional neural network encodes the image. In some embodiments of the present application, the first decoding module is constructed based on a connectionist temporal classification model. For example, the first decoding module can adopt a structure of a fully connected layer + softmax classification network + CTC sequence connection classification model. In some other embodiments of the present application, the first decoding module can also be implemented by other methods, and the embodiments of the present application do not limit the specific structure of the first decoding module.

[0052] Since semantics and self-attention are not introduced, the decoding operation only considers the expression of image texture features. The first decoding module mainly relies on the characteristics of the characters themselves for decoding, and finally outputs the text sequence recognition result of the input image, that is, each character category and corresponding probability.

[0053] Decoding step S5, executing the second decoding module to decode the second feature vector corresponding to the training sample image to obtain a second text prediction result.

[0054] The second decoding module is used to perform masking according to the length of the true value of the text of the training sample image, extract valid coding information, and calculate the loss value of the model only within the effective length. In some embodiments of the present application, the second decoding module is implemented based on mask cross entropy. The second decoding module performs full connection processing and classification mapping on the second feature vector after the alignment processing of the self-attention encoding output of the second encoding module, and based on the classification result, performs masking according to the length of the true value of the text of the training sample image, extracts the valid coding information in the second feature vector, and enables the prediction result to achieve the effect of automatic left alignment. When calculating the loss value of the encoding-decoding network model (for example, the loss value of the aforementioned second network branch), the error of the text prediction result is calculated only within the length of the true value of the text.

[0055] Decoding step S6, executing the third decoding module to decode the third feature vector corresponding to the training sample image to obtain a third text prediction result.

[0056] The third decoding module is used to decode the semantic features. As described above, the first encoding module, the second encoding module, and the structure of the third encoding module, since the third feature vector input to the third decoding module is a deeper feature obtained after two multi-head self-attention encodings, it is necessary to adopt a decoder structure symmetrical to the self-attention. In some embodiments of the present application, the third decoding module can be implemented using a Transformer decoder structure.

[0057] In some embodiments of the present application, in order to improve the accuracy of the model training results, position information is introduced into the third decoding module during the decoding process to utilize the absolute or relative position information of the input text sequence. For example, the third decoding module is implemented based on the structure of the position encoding module + Transformer decoder. In the period, the position information further includes: text encoding and text position encoding. Accordingly, the third decoding module is executed to decode the third feature vector corresponding to the training sample image to obtain the third text prediction result, including: determining the text encoding of the true value of the text corresponding to the training sample image and the text position encoding corresponding to the true value of the text; using the fusion encoding of the text encoding and the text position encoding as the first input of the third decoding module, and using the third feature vector as the second input of the third decoding module, and the first input and the second input are decoded by the third decoding module to obtain the third text prediction result. That is, the text encoding of the current character in the text true value (i.e., sample label) of the currently input training sample image is obtained through the text vector encoding submodule, and the text position encoding of the current character is obtained. Then, the text encoding and the text position encoding are concatenated into a fusion encoding as a query input of the Transformer decoder, and the third feature vector output by the third encoding module is used as another input of the Transformer decoder to perform a decoding operation.

[0058] In some embodiments of the present application, the third decoding module is constructed based on an attention mechanism network, and the fusion code of the text vector code and the position code is used as the first input of the third decoding module, and the third feature vector is used as the second input of the third decoding module. The first input and the second input are decoded by the third decoding module to obtain a third text prediction result, including: using the fusion code as the query (Q) of the third decoding module, and the third feature vector as the key (K) and value (V) of the third decoding module, and the first input and the second input are decoded by the attention mechanism network to obtain a third text prediction result.

[0059] In some embodiments of the present application, the third decoding module may also be implemented using other network structures corresponding to the third encoding module, which will not be listed one by one in the embodiments of the present application.

[0060] After completing the training of the encoding-decoding network model according to the above method, the encoding-decoding network model can be applied to perform online text recognition.

[0061] The following describes in detail the technical solutions for each step of the text recognition method based on progressive coding.

[0062] Step 110 , executing the at least three encoding modules in sequence, performing feature encoding of a target image with progressive depth, and obtaining encoding outputs of each encoding module respectively.

[0063] like Figure 2 As shown, each encoding module included in the encoding-decoding network model described in the embodiment of the present application encodes the input image with progressive depth. In some embodiments of the present application, taking the encoding-decoding network model including three encoding modules and corresponding three decoding modules as an example, the at least three encoding modules are executed in sequence to perform progressive depth feature encoding on the target image, and the encoding output of each encoding module is obtained respectively, including: executing the first encoding module, encoding the visual features in the target image, and outputting a first feature vector; executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector; executing the third encoding module, performing semantic feature encoding on the second feature vector, and outputting a third feature vector.

[0064] Execute the first encoding module, encode the visual features in the target image, and output the first feature vector. For the specific implementation method, refer to the model training phase. The specific implementation method of encoding the visual features in the training sample image and outputting the first feature vector corresponding to the training sample image will not be repeated here.

[0065] In some embodiments of the present application, as described above, the second encoding module further includes: a self-attention module for deleting residual links, the self-attention module for deleting residual links further includes: a multi-head attention network for deleting residual links, a normalization layer, a feedforward network for deleting residual links, and a normalization layer connected in sequence, the execution of the second encoding module, the feature alignment processing of the first feature vector, and the output of the second feature vector, including: through the second encoding module, the vector of the visual features of the text area and the vector of the visual features of the non-text area in the current input image of the encoding-decoding network model in the first feature vector are respectively aligned, and the second feature vector after the alignment processing is output. Wherein, the current input image is the target image; the self-attention module for deleting residual links can prevent the low-level background area information from being transmitted to the high-level features.

[0066] Through the second encoding module, the vectors of visual features of the text area and the vectors of visual features of the non-text area in the current input image of the encoding-decoding network model in the first feature vector are aligned respectively, and the specific implementation method of outputting the second feature vector after the alignment processing refers to the model training stage, and the specific implementation method of performing feature alignment processing on the first feature vector corresponding to the training sample image and outputting the second feature vector corresponding to the training sample image is not repeated here.

[0067] Execute the third encoding module, perform semantic feature encoding on the second feature vector, and output the specific implementation method of the third feature vector. Refer to the model training phase. Perform semantic feature encoding on the second feature vector corresponding to the training sample image, and output the specific implementation method of the third feature vector corresponding to the training sample image, which will not be repeated here.

[0068] Step 120 , executing each of the decoding modules respectively, decoding the encoded output of the corresponding encoding module, and obtaining the text recognition results of each decoding module on the target image.

[0069] After sequentially executing the first encoding module, the second encoding module, and the third encoding module to perform layer-by-layer deepening feature encoding on the target image, a first feature vector output by the first encoding module, a second feature vector output by the second encoding module, and a third feature vector output by the third encoding module are obtained in sequence. In some embodiments of the present application, after obtaining a feature vector output by a certain encoding module, a decoding module corresponding to the encoding module can be executed to perform a decoding operation on the feature vector of the encoding module data. For example: after executing the first encoding module, encoding the visual features in the target image, and outputting the first feature vector, executing each of the decoding modules respectively, decoding the encoded output of the corresponding encoding module, and obtaining the text recognition results of each decoding module on the target image, including: executing the first decoding module to decode the first feature vector to obtain the first text recognition result; after executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting the second feature vector, executing each of the decoding modules respectively, decoding the encoded output of the corresponding encoding module, and obtaining the text recognition results of each decoding module on the target image, including: executing the second decoding module to decode the second feature vector, and obtaining the second text recognition result; after executing the third encoding module, performing semantic feature encoding on the second feature vector, and outputting the third feature vector, executing each of the decoding modules respectively, decoding the encoded output of the corresponding encoding module, and obtaining the text recognition results of each decoding module on the target image, including: executing the third decoding module to decode the third feature vector, and obtaining the third text recognition result.

[0070] In some embodiments of the present application, after all the above encoding modules are executed, each decoding module can be started in parallel to decode the encoding output of the corresponding encoding module. For example, executing each decoding module separately, decoding the encoding output of the corresponding encoding module, and obtaining the text recognition results of each decoding module on the target image include: executing the first decoding module, decoding the first feature vector, and obtaining the first text recognition result; executing the second decoding module, decoding the second feature vector, and obtaining the second text recognition result; and executing the third decoding module to decode the third feature vector to obtain the third text recognition result.

[0071] Since the decoding modules are arranged in parallel and do not interfere with each other, a parallel processor can be used for decoding processing, which helps to improve the efficiency of text recognition.

[0072] For a specific implementation method of executing the first decoding module to decode the first feature vector to obtain a first text recognition result, refer to the specific implementation method of executing the first decoding module to decode the first feature vector corresponding to the training sample image to obtain a first text prediction result during the encoding-decoding network model training phase, which will not be repeated here.

[0073] Execute the second decoding module to decode the second feature vector to obtain the specific implementation method of the second text recognition result. Refer to the specific implementation method of executing the second decoding module to decode the second feature vector corresponding to the training sample image in the encoding-decoding network model training phase to obtain the second text prediction result, which will not be repeated here.

[0074] The specific implementation method of executing the third decoding module to decode the third feature vector to obtain the third text recognition result refers to the specific implementation method of executing the third decoding module to decode the third feature vector corresponding to the training sample image to obtain the third text prediction result during the encoding-decoding network model training phase, which will not be repeated here.

[0075] Step 130 , fusing the text recognition results of the target image by each of the decoding modules to determine the text recognition result of the target image by the encoding-decoding network model.

[0076] After obtaining the text recognition results output by all the above decoding modules, the text recognition results output by all the decoding modules are further fused, and the text recognition result of the target image by the encoding-decoding network model is determined according to the result obtained by the fusion process. Still taking the encoding-decoding network model including: a first decoding module, a second decoding module and a third decoding module, as an example, the text recognition result with the highest probability can be determined as the text recognition result of the target image according to the probabilities of the text recognition results output by the three decoding modules.

[0077] In other embodiments of the present application, other methods can also be used to fuse the text recognition results output by each decoding module to determine the text recognition result of the encoding-decoding network model for the target image, which will not be listed one by one in the embodiments of the present application.

[0078] The text recognition method based on progressive coding disclosed in the embodiment of the present application is pre-trained, including: at least three cascaded coding modules, and an encoding-decoding network model of a decoding module corresponding to each of the coding modules. In the prediction stage, the at least three coding modules are executed in sequence to perform progressive depth feature encoding on the target image to obtain the encoding output of each of the coding modules respectively; each of the decoding modules is executed respectively to decode the encoding output of the corresponding coding module to obtain the text recognition result of each of the decoding modules on the target image; the text recognition results of each of the decoding modules on the target image are fused to determine the text recognition result of the encoding-decoding network model on the target image, which helps to improve the accuracy of recognizing text in the image.

[0079] The text recognition method based on progressive coding disclosed in the embodiment of the present application pre-trains a plurality of cascaded coding modules for respectively extracting image features at different levels, extracts features of the input image at multiple levels of depth such as visual features and semantic features, and performs subsequent feature encoding and decoding respectively, thereby obtaining text recognition results based on features at different levels; further, by fusing the text recognition results based on features at different levels, the final text recognition result of the input image is obtained, thereby effectively improving the accuracy of text recognition in the image.

[0080] Furthermore, by using an improved self-attention network to perform feature alignment processing on visual features, the noise introduced by the background part of the image during the training and recognition processes can be effectively reduced or even removed, which can further improve the accuracy of text recognition in the image.

[0081] Embodiment 2

[0082] The embodiment of the present application discloses a text recognition device based on progressive coding, such as Figure 2 As shown, the pre-trained encoding-decoding network model executed by the device includes: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules. Figure 5 As shown, the device comprises:

[0083] The encoding control unit 510 is used to execute the at least three encoding modules in sequence, perform feature encoding of progressive depth on the target image, and obtain encoding outputs of each encoding module respectively;

[0084] A decoding control unit 520 is used to execute each of the decoding modules respectively, decode the encoded output of the corresponding encoding module, and obtain the text recognition result of each decoding module on the target image;

[0085] The fusion recognition unit 530 is used to perform fusion processing on the text recognition results of the target image by each of the decoding modules, and determine the text recognition result of the encoding-decoding network model for the target image.

[0086] In some embodiments of the present application, the at least three encoding modules include: a first encoding module, a second encoding module and a third encoding module, the decoding module includes: a first decoding module corresponding to the first encoding module, a second decoding module corresponding to the second encoding module, and a third decoding module corresponding to the third encoding module; the encoding control unit 510 is further used to:

[0087] Executing the first encoding module to encode the visual features in the target image and output a first feature vector;

[0088] executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector;

[0089] The third encoding module is executed to perform semantic feature encoding on the second feature vector, and output a third feature vector.

[0090] In some embodiments of the present application, the encoding-decoding network model is trained by the following method:

[0091] For each training sample image, the following encoding and decoding operations are performed respectively:

[0092] Executing the at least three encoding modules in sequence, performing feature encoding of the training sample image with progressive depth, and obtaining encoding output of the training sample image by each encoding module;

[0093] Executing each of the decoding modules respectively, decoding the encoding output of the corresponding encoding module on the training sample image, and obtaining the text prediction results of each decoding module on the training sample image;

[0094] For each of the training sample images, the prediction error of the training sample image is calculated according to the difference between the text prediction result of each decoding module on the training sample image and the true value of the text corresponding to the training sample image, and the loss value of the encoding-decoding network model is determined according to the prediction errors of all the training sample images;

[0095] The encoding-decoding network model is iteratively trained by optimizing the model parameters of the encoding-decoding network model to optimize the loss value.

[0096] In some embodiments of the present application, the at least three encoding modules are executed in sequence to perform feature encoding of the training sample image with progressive depth, and the encoding output of the training sample image by each encoding module is obtained respectively, including:

[0097] Executing the first encoding module to encode the visual features in the training sample image and output a first feature vector corresponding to the training sample image;

[0098] The respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image includes:

[0099] The first decoding module is executed to decode the first feature vector corresponding to the training sample image to obtain a first text prediction result.

[0100] In some embodiments of the present application, the step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining encoding output of the training sample image by each encoding module, further includes:

[0101] executing the second encoding module to perform feature alignment processing on the first feature vector corresponding to the training sample image, and outputting a second feature vector corresponding to the training sample image;

[0102] The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises:

[0103] The second decoding module is executed to decode the second feature vector corresponding to the training sample image to obtain a second text prediction result.

[0104] In some embodiments of the present application, the step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining encoding output of the training sample image by each encoding module, further includes:

[0105] Executing the third encoding module, performing semantic feature extraction on the second feature vector corresponding to the training sample image, and outputting a third feature vector corresponding to the training sample image;

[0106] The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises:

[0107] The third decoding module is executed to decode the third feature vector corresponding to the training sample image to obtain a third text prediction result.

[0108] In some embodiments of the present application, the second encoding module further includes: a self-attention module for deleting residual links, and executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector, includes:

[0109] Through the second encoding module, the vector of visual features of the text area and the vector of visual features of the non-text area in the current input image of the encoding-decoding network model in the first feature vector are aligned respectively, and the second feature vector after the alignment is output.

[0110] Among them, the current input image is the target image.

[0111] The text recognition device based on progressive coding disclosed in the embodiment of the present application is used to implement the text recognition method based on progressive coding described in the first embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here, and reference can be made to the specific implementation methods of the corresponding steps in the method embodiment.

[0112] The text recognition device based on progressive coding disclosed in the embodiment of the present application is pre-trained to include: at least three cascaded coding modules, and an encoding-decoding network model of a decoding module corresponding to each of the coding modules. In the prediction stage, the at least three coding modules are executed in sequence to perform progressive depth feature encoding on the target image to obtain the encoding output of each of the coding modules respectively; each of the decoding modules is executed respectively to decode the encoding output of the corresponding coding module to obtain the text recognition result of each of the decoding modules on the target image; the text recognition results of each of the decoding modules on the target image are fused to determine the text recognition result of the encoding-decoding network model on the target image, which helps to improve the accuracy of recognizing text in the image.

[0113] The text recognition device based on progressive coding disclosed in the embodiment of the present application, through pre-training of multiple cascaded coding modules that respectively extract image features at different levels, extracts features of the input image at multiple levels such as visual features and semantic features, and performs subsequent feature encoding and decoding respectively, thereby obtaining text recognition results based on features at different levels; further, by fusing the text recognition results based on features at different levels, the final text recognition result of the input image is obtained, thereby effectively improving the accuracy of text recognition in the image.

[0114] Furthermore, by using an improved self-attention network to perform feature alignment processing on visual features, the noise introduced by the background part of the image during the training and recognition processes can be effectively reduced or even removed, which can further improve the accuracy of text recognition in the image.

[0115] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] The above is a detailed introduction to a text recognition method and device based on progressive encoding provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0117] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0118] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0119] For example, Figure 6An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 610 and a memory 620 and a program code 630 stored on the memory 620 and can be run on the processor 610, and the processor 610 implements the method described in the above embodiment when executing the program code 630. The memory 620 can be a computer program product or a computer-readable medium. The memory 620 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 620 has a storage space 6201 for the program code 630 of a computer program for executing any method step in the above method. For example, the storage space 6201 for the program code 630 can include various computer programs for implementing the various steps in the above method respectively. The program code 630 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0120] The embodiment of the present application further discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the text recognition method based on progressive encoding as described in the first embodiment of the present application are implemented.

[0121] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Figure 6 The memory 620 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Figure 7 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 630', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0122] The term "one embodiment", "embodiment" or "one or more embodiments" herein means that a particular feature, structure or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that the examples of the term "in one embodiment" here do not necessarily all refer to the same embodiment.

[0123] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0124] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A character recognition method based on progressive coding, characterized in that: The pre-trained encoding-decoding network model includes: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules, and the method includes: Executing the at least three encoding modules in sequence, performing feature encoding of a target image with progressive depth, and obtaining encoding outputs of each encoding module respectively; Executing each of the decoding modules respectively, decoding the coded output of the corresponding coding module, and obtaining the text recognition results of each decoding module on the target image; Performing fusion processing on the text recognition results of the target image by each of the decoding modules to determine the text recognition result of the target image by the encoding-decoding network model; The at least three encoding modules include: a first encoding module, a second encoding module and a third encoding module, and the decoding module includes: A first decoding module corresponding to the first encoding module, a second decoding module corresponding to the second encoding module, and a third decoding module corresponding to the third encoding module; the step of sequentially executing the at least three encoding modules, performing progressive depth feature encoding on the target image, and obtaining the encoding output of each encoding module respectively, includes: Executing the first encoding module to encode the visual features in the target image and output a first feature vector; executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector; Executing the third encoding module to perform semantic feature encoding on the second feature vector and output a third feature vector; The encoder-decoder network model is trained by the following method: For each training sample image, the following encoding and decoding operations are performed respectively: Executing the at least three encoding modules in sequence, performing feature encoding of the training sample image with progressive depth, and obtaining encoding output of the training sample image by each encoding module; Executing each of the decoding modules respectively, decoding the encoding output of the corresponding encoding module on the training sample image, and obtaining the text prediction results of each decoding module on the training sample image; For each of the training sample images, the prediction error of the training sample image is calculated according to the difference between the text prediction result of each decoding module on the training sample image and the true value of the text corresponding to the training sample image, and the loss value of the encoding-decoding network model is determined according to the prediction errors of all the training sample images; Iteratively training the encoding-decoding network model by optimizing model parameters of the encoding-decoding network model to optimize the loss value; The step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining the encoding output of the training sample image by each encoding module, comprises: Executing the first encoding module to encode the visual features in the training sample image and output a first feature vector corresponding to the training sample image; The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises: The first decoding module is executed to decode the first feature vector corresponding to the training sample image to obtain a first text prediction result.

2. The method according to claim 1, characterized in that The step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining the encoding output of the training sample image by each encoding module, further includes: executing the second encoding module to perform feature alignment processing on the first feature vector corresponding to the training sample image, and outputting a second feature vector corresponding to the training sample image; The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises: The second decoding module is executed to decode the second feature vector corresponding to the training sample image to obtain a second text prediction result.

3. The method according to claim 2, characterized in that The step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining the encoding output of the training sample image by each encoding module, further includes: Executing the third encoding module, performing semantic feature extraction on the second feature vector corresponding to the training sample image, and outputting a third feature vector corresponding to the training sample image; The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises: The third decoding module is executed to decode the third feature vector corresponding to the training sample image to obtain a third text prediction result.

4. The method according to any one of claims 1 to 3, characterized in that: The second encoding module further includes: a self-attention module for deleting residual links, and the step of executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector includes: Through the second encoding module, the vectors of visual features of the text area and the vectors of visual features of the non-text area in the current input image of the encoding-decoding network model in the first feature vector are aligned respectively, and the aligned second feature vector is output.

5. A character recognition device based on progressive coding, characterized in that: The pre-trained encoding-decoding network model comprises: at least three cascaded encoding modules, and a decoding module corresponding to each of the encoding modules, and the device comprises: An encoding control unit, configured to sequentially execute the at least three encoding modules, perform feature encoding of progressive depth on the target image, and obtain encoding outputs of each encoding module respectively; A decoding control unit, used to execute each of the decoding modules respectively, decode the coded output of the corresponding coding module, and obtain the text recognition result of each decoding module on the target image; A fusion recognition unit, used for fusing the text recognition results of the target image by each of the decoding modules, and determining the text recognition result of the target image by the encoding-decoding network model; The at least three encoding modules include: a first encoding module, a second encoding module and a third encoding module, and the decoding module includes: a first decoding module corresponding to the first encoding module, a second decoding module corresponding to the second encoding module, and a third decoding module corresponding to the third encoding module; the encoding control unit is further used to: Executing the first encoding module to encode the visual features in the target image and output a first feature vector; executing the second encoding module, performing feature alignment processing on the first feature vector, and outputting a second feature vector; Executing the third encoding module to perform semantic feature encoding on the second feature vector and output a third feature vector; The encoding-decoding network model is trained by the following method: For each training sample image, the following encoding and decoding operations are performed respectively: Executing the at least three encoding modules in sequence, performing feature encoding of the training sample image with progressive depth, and obtaining encoding output of the training sample image by each encoding module; Executing each of the decoding modules respectively, decoding the encoding output of the corresponding encoding module on the training sample image, and obtaining the text prediction results of each decoding module on the training sample image; For each of the training sample images, the prediction error of the training sample image is calculated according to the difference between the text prediction result of each decoding module on the training sample image and the true value of the text corresponding to the training sample image, and the loss value of the encoding-decoding network model is determined according to the prediction errors of all the training sample images; Iteratively training the encoding-decoding network model by optimizing model parameters of the encoding-decoding network model to optimize the loss value; The step of sequentially executing the at least three encoding modules to perform feature encoding of the training sample image with progressive depth, and obtaining the encoding output of the training sample image by each encoding module, comprises: Executing the first encoding module to encode the visual features in the training sample image and output a first feature vector corresponding to the training sample image; The step of respectively executing each of the decoding modules to decode the coded output of the corresponding encoding module on the training sample image to obtain the text prediction result of each decoding module on the training sample image comprises: The first decoding module is executed to decode the first feature vector corresponding to the training sample image to obtain a first text prediction result.

6. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, characterized in that: When the processor executes the program code, the text recognition method based on progressive coding described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the text recognition method based on progressive coding described in any one of claims 1 to 4 are implemented.