Character recognition method and device, storage medium and electronic equipment
By performing sub-image feature extraction and convolution processing on text images to obtain position information, combined with the semantic feature extraction and decoding processing of Transformer architecture, the problem of low accuracy of text images at different scales in the prior art is solved, and high accuracy and variable-length text recognition capabilities are achieved.
Patent Information
- Application Number
- CN202311188878.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-14
- Publication Date
- 2025-03-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing text recognition method based on Transformer architecture is not very accurate in processing text images of different scales, and cannot effectively capture the scale characteristics of text.
By dividing the preprocessed text images, the embedding features of the sub-image are extracted and merged into an image embedding feature matrix, and then the matrix is convolutional to obtain the position matrix and add it to the image embedding feature matrix to obtain the position embedding feature matrix. Then semantic feature extraction and decoding processing are performed to obtain text recognition results.
This method can effectively adapt to text images of different scales, improve the accuracy of text recognition, and has variable-length text recognition capabilities.
Smart Images
Figure CN119625763A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to image processing technology, and in particular to a text recognition method, device, storage medium and electronic device. Background Art
[0002] With the continuous development of neural network technology, neural network technology has been more and more widely used in various fields such as image processing. Among them, the Transformer architecture has been a research hotspot in recent years.
[0003] At present, the research enthusiasm based on the Transformer architecture has gradually extended from the field of natural language processing to the field of computer vision, effectively improving the results of tasks such as image classification, object detection, and object segmentation. However, the Transformer structure originated in the field of natural language processing is not fully adapted to text recognition tasks. The existing Transformer-based text recognition method adds Gaussian bias to the self-attention layer of the Transformer model, alleviating the problem of weak local dependency capture ability of the self-attention layer. However, the above method ignores a feature of the text recognition task, that is, text often has different scales. The above method only modifies the self-attention layer, and cannot effectively recognize text images of different scales, and the recognition accuracy is not high. Summary of the invention
[0004] The present application provides a text recognition method, device, storage medium and electronic device, which can effectively recognize text images of different scales and improve the recognition accuracy.
[0005] To achieve the above objectives, this application adopts the following technical solutions:
[0006] A text recognition method, comprising:
[0007] Perform text detection on the target image and extract the text image for image preprocessing;
[0008] Dividing the preprocessed text image into a plurality of sub-images, and merging the embedding features of each sub-image obtained by extracting features from each sub-image into an image embedding feature matrix of the text image;
[0009] Performing convolution processing on the image embedding feature matrix to obtain a position matrix for representing relative position information inside the image embedding feature matrix, and adding the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix;
[0010] Extracting semantic features from the position embedding feature matrix to obtain a semantic feature matrix;
[0011] Decoding is performed based on the semantic feature matrix to obtain a text recognition result.
[0012] Preferably, the pre-processed text image is divided into a plurality of sub-images, including:
[0013] The text image is divided into strip-shaped sub-images, the height of the sub-images is the same as the height of the text image, and the width of the sub-images is N pixels; wherein N is a preset natural number.
[0014] Preferably, the performing convolution processing on the image embedding feature matrix comprises:
[0015] Using a convolution block to perform convolution processing on the image embedding feature matrix;
[0016] or,
[0017] The image embedding feature matrix is input into a convolutional neural network for processing.
[0018] Preferably, said extracting semantic features from said position embedding feature matrix comprises:
[0019] The position embedding feature matrix is processed by a self-attention layer and a feedforward neural network layer in sequence to obtain the semantic feature matrix.
[0020] Preferably, the decoding process based on the semantic feature matrix includes:
[0021] For each current vector in the semantic feature matrix, character estimation is performed on the current vector using the character estimation result of the current vector and the adjacent previous vector to obtain the character corresponding to the corresponding sub-image;
[0022] Output the characters corresponding to all sub-images in sequence;
[0023] Wherein, each vector in the semantic feature matrix is a semantic feature vector corresponding to each of the sub-images.
[0024] A text recognition device, comprising: a preprocessing module, an image embedding module, a position embedding module, a semantic feature extraction module and a sequence decoding module;
[0025] The preprocessing module is used to perform text detection on the target image and extract the text image for image preprocessing;
[0026] The image embedding module is used to divide the preprocessed text image into a plurality of sub-images, and merge the embedding features of each sub-image obtained by extracting features from each sub-image into an image embedding feature matrix of the text image;
[0027] The position embedding module is used to perform convolution processing on the image embedding feature matrix to obtain a position matrix for representing the relative position information inside the image embedding feature matrix, and add the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix;
[0028] The semantic feature extraction module is used to extract semantic features from the position embedding feature matrix to obtain a semantic feature matrix;
[0029] The sequence decoding module is used to perform decoding processing based on the semantic feature matrix to obtain a text recognition result.
[0030] Preferably, in the image embedding module, dividing the pre-processed text image into a plurality of sub-images comprises:
[0031] The text image is divided into strip-shaped sub-images, the height of the sub-images is the same as the height of the text image, and the width of the sub-images is N pixels; wherein N is a preset natural number.
[0032] Preferably, in the mid-position embedding module, the convolution processing of the image embedding feature matrix comprises:
[0033] Using a convolution block to perform convolution processing on the image embedding feature matrix;
[0034] or,
[0035] The image embedding feature matrix is input into a convolutional neural network for processing.
[0036] Preferably, in the semantic feature extraction module, extracting semantic features from the position embedding feature matrix includes:
[0037] The position embedding feature matrix is processed by a self-attention layer and a feedforward neural network layer in sequence to obtain the semantic feature matrix.
[0038] Preferably, the decoding process based on the semantic feature matrix includes:
[0039] For each current vector in the semantic feature matrix, character estimation is performed on the current vector using the character estimation result of the current vector and the adjacent previous vector to obtain the character corresponding to the corresponding sub-image;
[0040] Output the characters corresponding to all sub-images in sequence;
[0041] Wherein, each vector in the semantic feature matrix is a semantic feature vector corresponding to each of the sub-images.
[0042] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement any of the above-mentioned text recognition methods.
[0043] An electronic device comprising at least a computer-readable storage medium and a processor;
[0044] The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement any of the above-mentioned text recognition methods.
[0045] It can be seen from the above technical solution that in this application, after the target image is subjected to text detection, the text image is extracted, and image preprocessing such as resizing is performed; the processed image is cut into multiple sub-images, and then the embedded features of each sub-image are extracted and merged into an image embedding feature matrix; the position matrix is obtained by convolution processing the image embedding feature matrix, and the position embedding feature matrix is obtained by adding it to the image embedding feature matrix. Since the position matrix is obtained by convolution processing the image embedding feature matrix, the position matrix can reflect the relative position information inside the image embedding feature matrix, so that the features have the ability to distinguish different positions. At the same time, the position matrix can also naturally adapt to the dimension of the image embedding feature matrix, and will not be limited by the length limit of fixed position embedding, and can adapt to text images of different scales. Then, based on the above-mentioned position embedding feature matrix containing image embedding features and their position information, semantic feature extraction and decoding processing are performed to obtain text recognition results, so that such text recognition processing can adapt to text images of different scales, have variable length text recognition capabilities, and can effectively recognize text images of different scales, thereby improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of the basic flow of the character recognition method in this application;
[0047] Figure 2 A schematic diagram of a specific flow chart of a text recognition method in an embodiment of the present application;
[0048] Figure 3 A basic structural diagram of a Chinese character recognition device in this application;
[0049] Figure 4 It is a schematic diagram of the basic structure of the electronic device in this application. DETAILED DESCRIPTION
[0050] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.
[0051] Figure 1Schematic diagram of the basic flow of the character recognition method in this application. Figure 1 As shown, the method includes:
[0052] Step 101, perform text detection on the target image, extract the text image and perform image preprocessing.
[0053] This step is used to extract an image including text content, that is, a text image, from the complete target image through text detection. Then, the extracted text image is subjected to image preprocessing operations, such as resizing and pixel padding of the text image.
[0054] Step 102, dividing the preprocessed text image into a plurality of sub-images, and merging the embedding features of each sub-image obtained by performing feature extraction on each sub-image into an image embedding feature matrix of the text image.
[0055] In the transformer architecture, it is necessary to obtain the image features and position features of the text image, and add the position features to the image features for text recognition. This step is used to obtain the image features of the text image.
[0056] Specifically, in this step, the preprocessed text image is first divided into multiple sub-images, and then image features are extracted for each sub-image to obtain the embedding features corresponding to each sub-image, and all the embedded features are merged into a matrix, which is called the image embedding feature matrix of the text image.
[0057] Among them, when dividing sub-images, they can be divided into small blocks, or, considering the processing speed of the entire method, they can also be divided into strip-shaped sub-images. On the one hand, it conforms to the sequential feature form of text and ensures the accuracy of text recognition. On the other hand, it can reduce the number of sub-images and improve the processing speed of the text recognition method.
[0058] The feature extraction for each sub-image may adopt various existing feature extraction methods, for example, feature extraction may be realized through a neural network.
[0059] Step 103, performing convolution processing on the image embedding feature matrix to obtain a position matrix for representing the relative position information inside the image embedding feature matrix, and adding the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix.
[0060] As mentioned above, in the transformer architecture, in addition to the image features of the text image, the position features also need to be obtained. This step is used to obtain the position features of the text image and add them to the image features.
[0061] Specifically, in this step, the image embedding feature matrix is convolved to obtain a position matrix for representing position features. Since the position matrix is obtained by convolving the image embedding feature matrix, the position matrix can represent the relative position information inside the image embedding feature matrix and can naturally match the image embedding matrix dimension, so it can adapt to text images of different scales. Then the image embedding feature matrix is added to the position matrix, and the addition result is called the position embedding feature matrix.
[0062] Step 104: extract semantic features from the position embedding feature matrix to obtain a semantic feature matrix.
[0063] The processing of semantic feature extraction specifically includes attention layer processing and feedforward neural network layer processing, and a semantic feature matrix is obtained after processing. The specific processing is the same as in the traditional transformer architecture. Each vector of the semantic feature matrix is a semantic feature vector corresponding to each sub-image.
[0064] Step 105, performing decoding processing based on the semantic feature matrix to obtain a text recognition result.
[0065] The semantic feature vector corresponding to each sub-image in the semantic feature matrix is decoded to obtain the text recognition result of each semantic feature vector, and the text recognition results are output in sequence according to the text reading direction.
[0066] At this point, the basic process of the text recognition method in this application ends. It can be seen from the above basic method process that in this application, by convolution processing the image embedding feature matrix to obtain position information, it can be adapted to text images of various scales, realize effective text recognition, and improve recognition accuracy.
[0067] The specific implementation of the text recognition method of the present application is described below through a specific embodiment.
[0068] Figure 2 Schematic diagram of the specific flow of the text recognition method in the specific embodiment of this application. Figure 2 As shown, the method includes:
[0069] Step 201, perform text detection on the target image, extract the text image and perform image preprocessing.
[0070] In this step, the original target image can be inferred using a pre-trained text detection model, and the image with text and the location information of the text box can be retained; then, the text image (i.e., the text image) can be extracted using affine transformation based on the location information of the text image and the text box. The obtained text image is generally an image containing a single line of text, and is an image adjusted according to the text reading direction (for example, if the text reading direction is from top to bottom, the image is usually rotated to a horizontal orientation when the vertical image is extracted as a text image).
[0071] Perform image preprocessing on the text image. Common preprocessing may include image scaling and pixel padding. Image scaling refers to scaling the text thumbnail in equal proportions. Generally, the image height can be scaled to a set value M to facilitate subsequent processing. The specific value of M can be determined based on the system's processing speed requirements and text resolution requirements. The higher M is, the slower the processing speed is. The lower M is, the blurrier the text image is. A balance needs to be struck between the two. For example, M can be set to 32. On the one hand, it ensures that the text image is clear and the text can be distinguished. On the other hand, it ensures that the processing speed meets the requirements. For the scaled text image, it is usually possible to continue to pad the pixels at both ends so that the image width reaches a preset pixel multiple to facilitate subsequent processing. For example, the image width can be padded to a multiple of 512 pixels.
[0072] The preprocessed text image can be processed by the subsequent transformer architecture to realize text recognition.
[0073] Step 202: Divide the preprocessed text image into a plurality of sub-images.
[0074] The text image is divided into multiple small sub-images, and text recognition is performed on a sub-image basis.
[0075] Among them, the text image can be divided into block sub-images according to the existing transformer-based text recognition method. For example, a text image with a size of H*W can be cut into H / k*W / k block sub-images with a size of k*k, and then text recognition processing is performed based on the block sub-images.
[0076] However, since the text image is usually a single line of text, the text image itself has a long strip sequence characteristic, and the above-mentioned method of dividing the text image into block sub-images does not take into account the long strip sequence characteristic of the text image, and its block division method will destroy the structure of the text to a certain extent. Based on this, this embodiment provides another strip sub-image division method suitable for text images, which divides the text image with a size of H*W into W / N strip sub-images, where the size of each strip image is H*N. This division method can keep the height of the divided sub-image unchanged and divide the text part in the left and right directions, so that the text structure can be better preserved, adapted to the sequence characteristics of the text recognition task, and the number of divided sub-images is reduced. Under the premise of ensuring recognition accuracy, the processing speed is effectively improved.
[0077] Step 203: extract features from each sub-image to obtain embedded features of each sub-image, and combine all embedded features into an image embedded feature matrix.
[0078] The feature extraction processing of the sub-image can be implemented in various existing ways. For example, the sub-image can be input into a pre-trained feature extraction neural network. The specific neural network structure can be a neural network structure that can realize feature extraction, such as a multi-layer perceptron. When training the feature extraction neural network, the neural network can be randomly initialized, and then the sub-image samples used for training are input into the neural network. After several cycles of training, the feature extraction neural network is generated. In the processing of this step, each sub-image can be input into the feature extraction neural network, and after being processed by the neural network, the output is the embedded feature corresponding to the input sub-image. Then the embedded features corresponding to all sub-images are merged to form a matrix, and the embedded features of the entire text image are obtained, that is, the image embedding feature matrix of the text image.
[0079] Step 204, performing convolution processing on the image embedding feature matrix to obtain a position matrix, and adding the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix.
[0080] In the text recognition method based on the transformer architecture, it is necessary to add position information to the image embedding feature. This position information is used to characterize the position of each sub-image in the text image after division. In the existing processing method, this position information fix_pos can be calculated by formula. Its calculation process is not related to the embedding feature of the sub-image. It is a fixed absolute position information. This position information is added to the image embedding feature matrix f of the entire text image. emb The processing can be shown as formula (1):
[0081] f pos =f emb +fix_pos (1)
[0082] Among them, f pos It represents the result after adding the position information to the image embedding feature, which is called the position embedding feature matrix in this application.
[0083] In the above process, the position information fix_pos is the fixed absolute position information, and its dimension is also fixed. When the image is embedded in the feature matrix f emb When the dimension of changes, the position information fix_pos has the problem that the dimension cannot be directly matched, even if the interpolation method is used to force alignment f emb The dimension has a great impact on the overall text recognition performance, which makes the overall text recognition processing limited by the length limit of fixed position embedding and cannot have the ability to recognize text with variable length.
[0084] Based on this, a new method for obtaining position information is adopted in this application. The image embedding feature matrix generated in step 203 is convolved, and the convolution result is used as the position matrix fix_pos, which is added to the image embedding feature matrix and used for subsequent semantic feature extraction and character recognition.
[0085] Specifically, the convolution processing performed on the image embedding feature matrix can be directly performed by using the convolution block to perform a single convolution processing operation, for example, fix_pos=conv 33 (f emb ), that is, using a 3*3 convolutional block to embed the image into the feature matrix f emb Convolution processing is performed. The size of the convolution block can be selected as needed. Here, 3*3 is selected mainly because the processing speed is faster when implemented in hardware and the processing effect meets the requirements.
[0086] Alternatively, the convolution processing of the image embedding feature matrix can also be performed using a pre-trained convolutional neural network, that is, performing multiple superimposed convolution processing.
[0087] After obtaining the position matrix fix_pos, the image embedding feature is still added to the position matrix to obtain the position embedding feature matrix, where each vector in the position embedding feature matrix corresponds to a sub-image.
[0088] In the position embedding feature matrix obtained by the above method, its position information (i.e., the position matrix) is the convolution processing of the image embedding feature matrix. The convolution processing can capture the image edge information and the relative position information within the image embedding feature matrix, so that the obtained position information and the position embedding feature matrix have the ability to distinguish different positions. This ability is very important for sequence tasks such as text recognition, and can ensure the correctness of the character output order; at the same time, since the position information is the result of the convolution processing of the image embedding feature matrix, the dimension of the position information is consistent with the image embedding feature matrix, and there will be no problem that the dimension cannot be directly matched, so that the entire text recognition method will not be limited by the length limit of the fixed position embedding, and has the ability to recognize text of variable length, which is suitable for the recognition of text images of various scales, and can effectively improve the accuracy of text recognition.
[0089] Step 205, the position embedding feature matrix is processed by the self-attention layer and the feedforward neural network layer in sequence to obtain the semantic feature matrix of the text image.
[0090] Next, this step extracts semantic features from the position embedding feature matrix. The specific semantic feature extraction process includes the processing of the self-attention layer and the feedforward neural network layer.
[0091] The self-attention layer helps the model better understand contextual information by assigning different weights to different sub-image positions. The self-attention layer can be expressed by the following formulas (2) and (3):
[0092] Q, K, V = Linear q，k，v (f pos ) (2)
[0093]
[0094] Among them, Linear q,k,v () represents a fully connected layer, which is used to map the position embedding feature matrix to different feature spaces to obtain matrices Q, K and V. Softmax processing is used to normalize the probability distribution of weights. In the self-attention layer processing of this embodiment, Gaussian bias can no longer be introduced, because the position information obtained by the convolution processing mentioned above has overcome the problem of weak local dependency capture ability of the self-attention layer.
[0095] The feedforward neural network layer consists of two fully connected layers and a relu linear activation function. The first fully connected layer is processed with the relu activation function, and the second fully connected layer is not processed with the activation function. The processing of the above feedforward neural network layer can be used to explore the nonlinear relationship of features and enhance the performance of features. The specific processing of the feedforward neural network layer can be expressed by the following formula (4):
[0096] f ffn =Mlp(relu(Mlp(f att ))) (4)
[0097] Through the processing of the above self-attention layer and feedforward neural network layer, the semantic feature matrix of the entire text image is obtained. Among them, each vector in the semantic feature matrix is a semantic feature vector corresponding to each sub-image.
[0098] Step 206, performing decoding processing based on the semantic feature matrix to obtain a text recognition result.
[0099] This step is used to convert each vector in the semantic feature matrix generated in step 205 into a corresponding character, and output the text content in the text image. Specifically, the decoding process can predict the characters of each semantic feature vector through a fully connected network and a Softmax layer, obtain the probability of the corresponding predicted character, and finally select the character with the highest probability for output as the character recognition result of the corresponding sub-image. Then, the final text recognition result is obtained based on the character recognition results of all sub-images and output.
[0100] Among them, when performing character prediction for each semantic feature vector, each semantic feature vector can be predicted independently, or, in order to better utilize the correlation between characters and further improve the accuracy of text recognition, optionally, for each current vector in the semantic feature matrix, the character estimation results of the current vector and the adjacent previous vector can be used to perform character estimation on the current vector to obtain the predicted character corresponding to the corresponding sub-image.
[0101] In more detail, the predicted character corresponding to the current semantic feature vector can be obtained based on the predicted character corresponding to the previous semantic feature vector (corresponding to the information of the previous sub-image position) combined with the current semantic feature vector (corresponding to the information of the current sub-image position), and the final text recognition result of the entire text image can be obtained by traversing all the semantic feature vectors.
[0102] So far, Figure 2 The text recognition method process in the specific embodiment shown ends.
[0103] From the specific implementation of the above text recognition method, it can be seen that the present application provides a text recognition method, which realizes variable-length Transformer text recognition through a position information acquisition method based on convolution processing, and effectively improves the problem of poor recognition effect in scenes with large text scales. At the same time, in the text recognition method, a strip division method can also be used when dividing sub-images to further adapt to the sequence characteristics of text and improve the processing speed of the text recognition method.
[0104] The above is the specific implementation of the text recognition method in this application. This application also provides a text recognition device, which can be used to implement the above text recognition method. Figure 3 FIG. 1 is a schematic diagram of the basic structure of the text recognition device. Figure 3 As shown, the text recognition device includes: a preprocessing module, an image embedding module, a position embedding module, a semantic feature extraction module and a sequence decoding module.
[0105] Among them, the preprocessing module is used to perform text detection on the target image and extract the text image for image preprocessing;
[0106] An image embedding module is used to divide the preprocessed text image into a plurality of sub-images, and merge the embedding features of each sub-image obtained by extracting features from each sub-image into an image embedding feature matrix of the text image;
[0107] A position embedding module is used to perform convolution processing on the image embedding feature matrix to obtain a position matrix used to represent the relative position information inside the image embedding feature matrix, and to add the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix;
[0108] A semantic feature extraction module is used to extract semantic features from the position embedding feature matrix to obtain a semantic feature matrix;
[0109] The sequence decoding module is used to perform decoding processing based on the semantic feature matrix to obtain the text recognition result.
[0110] Optionally, in the image embedding module, the process of dividing the preprocessed text image into a plurality of sub-images may specifically include:
[0111] The text image is divided into strip-shaped sub-images, the height of the sub-images is the same as the height of the text image, and the width of the sub-images is N pixels; wherein N is a preset natural number.
[0112] Optionally, in the position embedding module, convolution processing is performed on the image embedding feature matrix, which may specifically include:
[0113] Use convolution blocks to convolve the image embedding feature matrix;
[0114] or,
[0115] The image embedding feature matrix is input into the convolutional neural network for processing.
[0116] Optionally, in the semantic feature extraction module, the processing of extracting semantic features from the position embedding feature matrix may specifically include:
[0117] The position embedding feature matrix is processed by the self-attention layer and the feedforward neural network layer in sequence to obtain the semantic feature matrix.
[0118] Optionally, the decoding process is performed based on the semantic feature matrix, which may specifically include:
[0119] For each current vector in the semantic feature matrix, character estimation is performed on the current vector using the character estimation results of the current vector and the adjacent previous vector to obtain the character corresponding to the corresponding sub-image;
[0120] Output the characters corresponding to all sub-images in sequence;
[0121] Wherein, each vector in the semantic feature matrix is a semantic feature vector corresponding to each sub-image.
[0122] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the above-mentioned method for implementing text recognition can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in the computer-readable storage medium, and when the instructions stored in the computer-readable storage medium are executed by a processor, the steps in the above-mentioned method for implementing text recognition can be executed.
[0123] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.
[0124] Figure 4 This application also provides an electronic device. Figure 4 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:
[0125] The electronic device may include a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 402 is executed, the method for text recognition may be implemented.
[0126] Specifically, in practical applications, the electronic device may further include components such as a power supply 403, an input / output unit 404, etc. Those skilled in the art may understand that Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0127] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.
[0128] The memory 402 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.
[0129] The electronic device also includes a power supply 403 for supplying power to various components, which can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, etc.
[0130] The electronic device may also include an input and output unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 404 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.
[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for character recognition, characterized in that: include: Perform text detection on the target image and extract the text image for image preprocessing; Dividing the preprocessed text image into a plurality of sub-images, and merging the embedding features of each sub-image obtained by extracting features from each sub-image into an image embedding feature matrix of the text image; Performing convolution processing on the image embedding feature matrix to obtain a position matrix for representing relative position information inside the image embedding feature matrix, and adding the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix; Extracting semantic features from the position embedding feature matrix to obtain a semantic feature matrix; Decoding is performed based on the semantic feature matrix to obtain a text recognition result.
2. The method according to claim 1, characterized in that: The preprocessed text image is divided into a plurality of sub-images, including: The text image is divided into strip-shaped sub-images, the height of the sub-images is the same as the height of the text image, and the width of the sub-images is N pixels; wherein N is a preset natural number.
3. The method according to claim 1 or 2, characterized in that: The performing convolution processing on the image embedding feature matrix comprises: Using a convolution block to perform convolution processing on the image embedding feature matrix; or, The image embedding feature matrix is input into a convolutional neural network for processing.
4. The method according to claim 1 or 2, characterized in that: The extracting semantic features from the position embedding feature matrix includes: The position embedding feature matrix is processed by a self-attention layer and a feedforward neural network layer in sequence to obtain the semantic feature matrix.
5. The method according to claim 4, characterized in that The decoding process based on the semantic feature matrix includes: For each current vector in the semantic feature matrix, character estimation is performed on the current vector using the character estimation result of the current vector and the adjacent previous vector to obtain the character corresponding to the corresponding sub-image; Output the characters corresponding to all sub-images in sequence; Wherein, each vector in the semantic feature matrix is a semantic feature vector corresponding to each of the sub-images.
6. A text recognition device, characterized in that: include: Preprocessing module, image embedding module, position embedding module, semantic feature extraction module and sequence decoding module; The preprocessing module is used to perform text detection on the target image and extract the text image for image preprocessing; The image embedding module is used to divide the preprocessed text image into a plurality of sub-images, and merge the embedding features of each sub-image obtained by extracting features from each sub-image into an image embedding feature matrix of the text image; The position embedding module is used to perform convolution processing on the image embedding feature matrix to obtain a position matrix for representing the relative position information inside the image embedding feature matrix, and add the image embedding feature matrix to the position matrix to obtain a position embedding feature matrix; The semantic feature extraction module is used to extract semantic features from the position embedding feature matrix to obtain a semantic feature matrix; The sequence decoding module is used to perform decoding processing based on the semantic feature matrix to obtain a text recognition result.
7. The device according to claim 6, characterized in that In the image embedding module, the preprocessed text image is divided into a plurality of sub-images, including: The text image is divided into strip-shaped sub-images, the height of the sub-images is the same as the height of the text image, and the width of the sub-images is N pixels; wherein N is a preset natural number.
8. The device according to claim 6 or 7, characterized in that In the mid-position embedding module, the convolution processing of the image embedding feature matrix includes: Using a convolution block to perform convolution processing on the image embedding feature matrix; or, The image embedding feature matrix is input into a convolutional neural network for processing.
9. The device according to claim 6 or 7, characterized in that In the semantic feature extraction module, the step of extracting semantic features from the position embedding feature matrix includes: The position embedding feature matrix is processed by a self-attention layer and a feedforward neural network layer in sequence to obtain the semantic feature matrix.
10. The device according to claim 9, characterized in that The decoding process based on the semantic feature matrix includes: For each current vector in the semantic feature matrix, character estimation is performed on the current vector using the character estimation result of the current vector and the adjacent previous vector to obtain the character corresponding to the corresponding sub-image; Output the characters corresponding to all sub-images in sequence; Wherein, each vector in the semantic feature matrix is a semantic feature vector corresponding to each of the sub-images.
11. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by a processor, the text recognition method described in any one of claims 1 to 5 can be implemented.
12. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the text recognition method described in any one of claims 1 to 5.