A method, device, equipment and medium for character recognition of image

Through the combination of feature matrix encoding and attention algorithm, the accuracy problem of existing optical character recognition technology in irregular character sequences and curved text verification code recognition is solved, and efficient and accurate character recognition is achieved.

CN113762261BActive Publication Date: 2024-11-08TENCENT CLOUD COMPUTING (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110508541.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-10
Publication Date
2024-11-08
Estimated Expiration
2041-05-10

AI Technical Summary

Technical Problem

The existing optical character recognition technology is not accurate enough in character recognition, especially in the recognition of irregular character sequences and curved text verification code scenarios.

Method used

Feature matrix encoding combined with attention algorithm is used to identify character sequences in the image through feature extraction network, encoder and decoder, and masking processing is used to avoid repeated recognition, improving recognition efficiency and accuracy.

Benefits of technology

Accurate recognition of character sequences in the image is achieved, and recognition efficiency and accuracy are improved, especially in complex shapes and irregular character scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113762261B_ABST
    Figure CN113762261B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a method, device, equipment and medium for character recognition of an image, wherein the method may include: obtaining an image to be recognized, and performing feature analysis on the image to be recognized to obtain a feature matrix, wherein the feature matrix includes multiple eigenvalues; encoding each eigenvalue in the feature matrix to obtain eigenvalue encoding information; recognizing the first character in the character sequence contained in the image to be recognized based on the eigenvalue encoding information in combination with an attention algorithm, and after predicting the first character, masking the attention map corresponding to the eigenvalue encoding information obtained based on the attention algorithm to obtain a masked attention map; based on the first character, predicting the second character from the masked attention map to obtain a character recognition result for the image to be recognized based on the first character and the second character. The embodiments of the present application can more accurately recognize characters in the image to be recognized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and medium for character recognition of an image. Background Art

[0002] Character recognition is involved in many application scenarios, including document recognition, card number recognition, license plate recognition, and bill recognition, etc. There are also some special scenarios for irregular character sequence recognition, such as seal content recognition, curved text verification code recognition, etc. Among them, in the seal content recognition scenario, effective recognition of each character contained in the seal or the character sequence composed of each character can help determine the owner of the seal, and can provide convenience for scenarios such as contract or bill review; and curved text verification code recognition scenarios can automatically identify the required text verification code for users, so that users can quickly perform verification processing in scenarios such as registration and payment.

[0003] Currently, most methods for character recognition in various images are based on OCR (Optical Character Recognition), but OCR recognition is sometimes not accurate enough. Summary of the invention

[0004] The embodiments of the present application provide a method, device and apparatus for character recognition in an image, which can more accurately recognize characters in an image.

[0005] On the one hand, an embodiment of the present application provides a method for character recognition of an image, the method comprising:

[0006] Acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes a plurality of eigenvalues;

[0007] Encoding each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix;

[0008] Based on the feature value encoding information combined with the attention algorithm, a first character in the character sequence contained in the image to be identified is identified, and after predicting the first character, an attention map corresponding to the feature value encoding information obtained based on the attention algorithm is masked to obtain an attention map after masking;

[0009] Based on the first character, a second character is predicted from the attention map after the mask processing, so as to obtain a character recognition result of the image to be recognized according to the first character and the second character.

[0010] On the other hand, an embodiment of the present application provides a method for character recognition of an image, the method comprising:

[0011] Obtain an image to be recognized;

[0012] Using the image to be recognized as an input of a character detection model, and obtaining a character recognition result of the image to be recognized according to an output result of the character detection model;

[0013] Wherein, the character detection model includes a feature extraction network, an encoder and a decoder;

[0014] The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix;

[0015] The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network;

[0016] The decoder is used to predict the feature value encoding information output by the encoder to obtain a first character, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm to obtain the masked attention map; based on the first character, predict the second character from the masked attention map to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0017] On the other hand, an embodiment of the present application provides a device for character recognition of an image, the device comprising:

[0018] An acquisition unit, used for acquiring an image to be identified, and performing feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes a plurality of eigenvalues;

[0019] A processing unit is used to encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, wherein the eigenvalue encoding information is used to represent the position information and feature information of each eigenvalue in the feature matrix; based on the eigenvalue encoding information combined with an attention algorithm, a first character in a character sequence contained in the image to be recognized is identified, and after predicting the first character, an attention map corresponding to the eigenvalue encoding information obtained based on the attention algorithm is masked to obtain a masked attention map; based on the first character, a second character is predicted from the masked attention map to obtain a character recognition result of the image to be recognized based on the first character and the second character.

[0020] On the other hand, an embodiment of the present application provides a device for character recognition of an image, the device comprising:

[0021] An acquisition unit, used for acquiring an image to be recognized;

[0022] A processing unit, used to use the image to be recognized as an input of a character detection model, and obtain a character recognition result of the image to be recognized according to an output result of the character detection model;

[0023] Wherein, the character detection model includes a feature extraction network, an encoder and a decoder;

[0024] The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix;

[0025] The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network;

[0026] The decoder is used to predict the feature value encoding information output by the encoder to obtain a first character, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm to obtain the masked attention map; based on the first character, predict the second character from the masked attention map to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0027] On the other hand, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0028] a storage device for storing a computer application;

[0029] The processor executes the computer application program stored in the storage device to implement the above-described method for character recognition of an image.

[0030] In one embodiment, the processor is specifically configured to perform the following steps:

[0031] Acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes a plurality of eigenvalues;

[0032] Encoding each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix;

[0033] Based on the feature value encoding information combined with the attention algorithm, a first character in the character sequence contained in the image to be identified is identified, and after predicting the first character, an attention map corresponding to the feature value encoding information obtained based on the attention algorithm is masked to obtain an attention map after masking;

[0034] Based on the first character, a second character is predicted from the attention map after the mask processing, so as to obtain a character recognition result of the image to be recognized according to the first character and the second character.

[0035] In another embodiment, the processor is specifically configured to perform the following steps:

[0036] Obtain an image to be recognized;

[0037] Using the image to be recognized as an input of a character detection model, and obtaining a character recognition result of the image to be recognized according to an output result of the character detection model;

[0038] Wherein, the character detection model includes a feature extraction network, an encoder and a decoder;

[0039] The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix;

[0040] The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network;

[0041] The decoder is used to predict the feature value encoding information output by the encoder to obtain a first character, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm to obtain the masked attention map; based on the first character, predict the second character from the masked attention map to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0042] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer application. When the computer application is executed, the above-described method for character recognition of an image is implemented.

[0043] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device performs the above-mentioned method for character recognition of an image.

[0044] In the embodiment of the present application, each eigenvalue in the feature matrix corresponding to the image to be recognized can be encoded to obtain eigenvalue encoding information, and each character in the character sequence contained in the image to be recognized can be identified based on the eigenvalue encoding information, thereby realizing rapid acquisition of the character recognition result of the image to be recognized. In addition, after recognizing the first character in the character sequence contained in the image to be recognized, the attention map corresponding to the first character can be masked, so that in the process of recognizing the second character in the character sequence, repeated recognition of the first character can be avoided, thereby improving the efficiency of character recognition, being able to more quickly identify each character in the character sequence, and improving the accuracy and speed of character recognition in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1a A schematic diagram of an image to be recognized including a character sequence provided by an exemplary embodiment of the present application is shown;

[0047] Figure 1b A schematic diagram of the architecture of a character recognition system for an image provided by an exemplary embodiment of the present application is shown;

[0048] Figure 2 A schematic diagram of a flow chart of a method for character recognition of an image provided by an exemplary embodiment of the present application is shown;

[0049] Figure 3 A schematic diagram of an image to be recognized provided by an exemplary embodiment of the present application is shown;

[0050] Figure 4 A schematic diagram of determining a character area to be recognized provided by an exemplary embodiment of the present application is shown;

[0051] Figure 5 A schematic diagram of obtaining a characteristic matrix provided by an exemplary embodiment of the present application is shown;

[0052] Figure 6 A schematic diagram of a process of processing an image to be identified provided by an exemplary embodiment of the present application is shown;

[0053] Figure 7 A schematic diagram showing another process of processing an image to be identified provided by an exemplary embodiment of the present application;

[0054] Figure 8 A schematic diagram of a flow chart of a method for character recognition of an image provided by an exemplary embodiment of the present application is shown;

[0055] Fig. 9 A schematic diagram of inputting an image to be recognized through a terminal provided by an exemplary embodiment of the present application is shown;

[0056] Fig.10 A schematic diagram of identifying multiple seals from an image to be identified provided by an exemplary embodiment of the present application is shown;

[0057] Fig.11a A schematic diagram of a flow chart of a method for character recognition of an image provided by an exemplary embodiment of the present application is shown;

[0058] Fig.11b A schematic diagram of a process for training a network model provided by an exemplary embodiment of the present application is shown;

[0059] Fig.12 A schematic diagram of data expansion provided by an exemplary embodiment of the present application is shown;

[0060] Fig.13 A schematic diagram of constructing a first image provided by an exemplary embodiment of the present application is shown;

[0061] Fig.14 A schematic diagram of constructing a second image provided by an exemplary embodiment of the present application is shown;

[0062] Fig.15 A schematic structural diagram of a device for character recognition of an image provided by an exemplary embodiment of the present application is shown;

[0063] Fig.16 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] The embodiment of the present application proposes a character recognition scheme for an image, which involves artificial intelligence machine learning and other technologies, wherein:

[0066] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] Machine Learning (ML) is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. Machine learning can be seen as a task, the goal of which is to allow machines (computers in a broad sense) to acquire human-like intelligence through learning. For example, humans can play Go, and computer programs (AlphaGo or AlphaGoZero) are designed to master Go knowledge and play Go. Among them, a variety of methods can be used to achieve machine learning tasks, such as neural networks, linear regression, decision trees, support vector machines, Bayesian classifiers, reinforcement learning, probabilistic graphical models, clustering and other methods.

[0068] Among them, neural network is a method to achieve machine learning tasks. When talking about neural network in the field of machine learning, it generally refers to "neural network learning". It is a network structure composed of many simple elements. This network structure is similar to the biological nervous system and is used to simulate the interaction between organisms and the natural environment. The more network structures there are, the richer the functions of the neural network tend to be. Neural network is a relatively large concept. For different learning tasks such as speech, text, and images, neural network models that are more suitable for specific learning tasks have been derived, such as recurrent neural network (RNN), convolutional neural network (CNN), fully connected convolutional neural network (FCNN), etc.

[0069] The image to be recognized involved in the character recognition scheme in the embodiment of the present application can be an ordinary image with text content, such as some newspapers and magazines, especially some images with seals, images with character verification codes with curved character sequences, etc. Effectively identifying the seal content or curved character sequence in the image to be recognized is of great significance; taking the image to be recognized as an image of a character verification code with a curved character sequence as an example, in application scenarios such as registration and login through character verification codes, only by accurately identifying the curved character sequence contained in the character verification code can the user's identity be verified, thereby helping the user to quickly complete registration, login and other operations; among them, the exemplary image of the character verification code with a curved character sequence can be seen in Figure 1a The first picture shown. Taking the image to be identified as an image with a seal as an example, the character sequence contained in the seal (i.e., the seal content) is often the identifier of the object with the seal (such as the name of a department, the name of a company, etc.). Therefore, the seal has a certain validity. By stamping a seal (i.e., the seal of any unit or individual) in a document (or image, etc.), it can be indicated that the document is signed by the object indicated by the seal (such as a unit or individual). Then, the validity of the document can be determined based on the seal on the document, and then it can be determined that the rights or obligations associated with the document belong to the object indicated by the seal. Among them, exemplary images with seals can be seen in Figure 1a As shown in the second figure, for seals of different shapes, the character sequences contained in the seals can also be displayed in different distribution shapes; if the shape of the seal is curved (such as a circle or an ellipse), then the character sequence of the seal can be distributed in a curved shape following the shape of the seal, that is, each character contained in the character sequence is displayed according to the position and angle of the curve. For the sake of convenience, the following description will be based on the example that the image to be recognized is an image with a seal, and the seal is in a curved shape.

[0070] From the above description, it can be seen that accurately identifying the character sequence in the image to be identified can provide convenience for a variety of application scenarios (such as automatic seal verification scenarios, new user registration scenarios, etc.). Based on this, the embodiment of the present application also proposes a character recognition system for images, which can be used to implement a character recognition solution for images. Figure 1b As shown, the character recognition system for the image includes a terminal 101 and a server 102. The embodiment of the present application does not limit the number and type of the terminal 101 and the server 102, which is explained here. Among them, the terminal 101 may include but is not limited to: smart phones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), smart TVs, vehicle-mounted devices, head-mounted devices, and other smart devices that can touch the screen. The terminal includes a terminal screen, also known as a display screen, a display screen, etc.; the terminal can run applications (which can be referred to as applications, such as social applications, game applications, video applications, small program applications, web applications, etc.). The server 102 may include but is not limited to: data processing servers, Web servers, application servers, and other devices with complex computing capabilities. Server 102 may be a background server of terminal 101, used to interact with terminal 101 to provide computing and application service support for terminal 101; or, server 102 may also be a background server of any application running in terminal 101, used to interact with the application running in terminal 101 to provide computing and application service support for any application. Server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. Server 102 may also be a node server on a blockchain network. Terminal 101 and server 102 may be directly or indirectly connected in communication via wired or wireless means, and the embodiment of the present application does not limit the connection method between terminal 101 and server 102.

[0071] The character recognition system for images proposed in the embodiment of the present application is implemented by an electronic device (such as terminal 101 or server 102) calling a network model; the network model can be obtained by training a region detection model (such as a yolov5 model) and a character detection model (such as a neural machine translation model) using sample data; the region detection model is used to detect the display position or display area of ​​a character sequence in an image, such as the display position or display area of ​​a seal; the character detection model is used to identify the character sequence in the image to be identified (such as the character sequence contained in the seal). Based on this, the embodiment of the present application involves the training and application of the network model. Since the training and application of the network model have different requirements for the computing power of the electronic device, for example, when training the network model, the computer equipment needs to have a GPU (graphics processing unit) chip to meet the high computing power requirements of the model training.

[0072] In one implementation, the embodiment of the present application supports the server 102 equipped with a GPU chip to perform the training of the network model, and the terminal 101 with a human-computer interaction function is used to execute the application of the network model. At this time, the trained network model is deployed in the terminal 101. Among them, the general process of the interaction between the terminal 101 and the server 102 to execute the character recognition solution for the image may include: training the network model (including the region detection model and the character detection model mentioned in the embodiment of the present application) on the server 102 side, and after the network model converges (that is, the optimized network model is obtained), sending the network model (or the program for running the network model) to the terminal 101; the terminal 101 receives the network model sent by the server 102, and deploys the network model. When the terminal 101 obtains the image to be recognized, the network model can be called to perform the following recognition operations: first, the display position of the character sequence in the image to be recognized can be detected, and the image to be recognized containing the character sequence can be intercepted from the image to be recognized according to the detected display position; secondly, the image to be recognized is subjected to feature analysis to obtain a feature matrix, and each eigenvalue in the feature matrix is ​​encoded to obtain eigenvalue encoding information; finally, each character in the character sequence contained in the image to be recognized is identified based on the eigenvalue encoding information to obtain the character recognition result of the image to be recognized. Based on the above process, the recognition result of the image to be recognized can be obtained, and the recognition result may include: the position recognition result of the display position of the character sequence in the image to be recognized, and the character recognition result. Among them, the expression of the position recognition result may include but is not limited to: outputting the display position in the form of coordinates; or intercepting the part of the image containing the character sequence for output, such as intercepting the part of the image containing the seal for output; or outputting the position information, such as the position information is the lower left corner; and so on. The embodiment of the present application does not limit the expression of the position recognition result.

[0073] In other implementations, after the server 102 has trained the network model, it has deployed the network model itself, and the character recognition scheme for the image provided in the embodiment of the present application can also be executed by the server 102. In this implementation, the general process of executing the character recognition scheme for the image may include: the server 102 performs model training on the network model to obtain a converged network model; when the terminal 101 obtains the image to be recognized, the image to be recognized is sent to the server 102, and the server 102 calls the network model to perform character recognition processing on the image to be recognized, and obtains the recognition result of the image to be recognized, and the recognition result may include the position recognition result and the character recognition result. Among them, the process of the server 102 calling the network model to recognize the image to be recognized can be referred to the above description, and the relevant description of the terminal 102 calling the network model to recognize the image to be recognized will not be repeated here. It should be noted that the recognition results mentioned later all include the position recognition results and the character recognition results, which are specially explained here.

[0074] It is understandable that if the terminal 101 is equipped with a GPU chip that can support higher computing power, then the training and application process of the network model can also be executed by the terminal 101. The process of terminal 101 executing the character recognition solution for the image is of the same type as the above-mentioned implementation method and will not be elaborated here.

[0075] Based on the above-described image character recognition scheme, the embodiment of the present application proposes a more detailed image character recognition method. The image character recognition method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0076] See also Figure 2 , Figure 2 A flowchart of a method for character recognition of an image provided by an exemplary embodiment of the present application is shown; the method for character recognition of an image may be performed by an electronic device, and the electronic device may include: Figure 1b In the terminal 101 or the server 102 in the system shown, the character recognition method for the image includes but is not limited to steps S201-S204.

[0077] S201: Acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes multiple feature values. The image to be identified may refer to an image containing some character sequences, which may be irregularly contained in the image or in a specific shape, such as Figure 1a In the embodiment of the present application, an exemplary image to be recognized can be found in Figure 3 ,like Figure 3As shown, the image to be recognized has a character sequence "Shenzhen Innovation Company" presenting a curved distribution. This character sequence contains multiple characters. For example, this character sequence includes: the character "深 (Shen)", the character "圳 (Zhen)", the character "市 (Shi)", the character "创 (Chuang)", the character "新 (Xin)", the character "公 (Gong)", and the character "司 (Si)", etc. The embodiments of the present application do not limit the types and quantities of characters included in the character sequence. The character types may include, but are not limited to: English characters, Chinese characters, punctuation marks, etc. In some embodiments, Figure 3 What is described is an image with a seal. The image to be recognized can also be some other images obtained by taking screenshots of verification codes displayed in a wavy shape, for example.

[0078] In step S201, a feature extraction network, such as a CNN, can be used to complete feature analysis and obtain a feature matrix. In a CNN-based model, sampling methods are supported for feature analysis of the image to be recognized; specifically, subsampling or downsampling methods can be used to perform feature analysis on the image to be recognized to obtain a feature matrix. Performing subsampling on the image to be recognized can be understood as shrinking the image to be recognized to obtain a reduced image (or thumbnail) of the image to be recognized. In one embodiment, multiple-channel subsampling can be performed on the image to be recognized to obtain a feature matrix of multiple channels.

[0079] Performing the above subsampling process on the image to be recognized can obtain a feature matrix with a size that meets the requirements of subsequent processing. The size required for subsequent processing is based on the output size defined during the training of the feature extraction network. The following combines Figure 5 to give an exemplary introduction to the subsampling process. As Figure 5 shown, assume that the size of the image to be recognized is: 192 pixels * 160 pixels, and the set subsampling ratio is 8 * 8 times and three-channel (such as corner points, horizontal, vertical) subsampling. Then, both the length and width of the image to be recognized are reduced by 8 times. The size of the feature matrix obtained by subsampling the image to be recognized is: (198 / 8) * (160 / 8) = 24 * 20. The above process of introducing feature analysis of the image to be recognized by taking subsampling as an example is described, but it can be understood that the way of performing feature analysis on the image to be recognized can also be other ways. The embodiments of the present application do not limit this and are hereby explained.

[0080] S202: Encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, wherein the eigenvalue encoding information is used to represent the position information and feature information of each eigenvalue in the feature matrix. The encoding of the feature matrix can be implemented by an encoder. Depending on the encoder used, the method of encoding the feature matrix is ​​also different. The embodiment of the present application takes the encoder as a bidirectional LSTM (Long Short Term Mermory network) as an example for explanation. The encoder can be specifically a layer of bidirectional LSTM, as shown in the specific diagram Figure 6 The network 601 in the figure. The LSTM mentioned above is a recurrent neural network that has the ability to remember long-term information and can use the historical record information as a new round of input information to achieve the purpose of recording all information. It is worth noting that in the implementation mode in which the encoder is composed of LSTM, after the encoder is used to encode each eigenvalue in the feature matrix, the eigenvalue encoding information is obtained, and the eigenvalue encoding information is used to represent the position information and feature information of each eigenvalue in the feature matrix.

[0081] Assume that the feature matrix extracted in step S201 is V, and each row of features in the feature matrix is ​​input into the encoder encoder (a layer of bidirectional LSTM) to obtain a new feature matrix V′. The new feature matrix V′ is the feature value encoding information described in the embodiment of the present application, and its specific calculation formula (1) can be shown as follows:

[0082] V′ h,w =RNN(V′ h,w-1 ,V h,w ) (1)

[0083] RNN is a recurrent neural network; the subscript h refers to the number of rows of the corresponding eigenvalue of the input on the feature matrix, and the subscript w refers to the position of the corresponding row of the eigenvalue of the input on the feature matrix. For example, when h=1 and w=1, it refers to the first eigenvalue on the first row of the feature matrix, that is, the eigenvalue of the first row and the first column in the feature matrix. In order to obtain the sequential features of the feature matrix in the column direction, each row feature in the feature matrix is ​​fused and concat with the embedding feature of the current row number before entering the encoder as input. Therefore, in the obtained V′ h,w In the expression, the row and column of the corresponding eigenvalue and the specific position in each row are expressed, such as V′ 1,1 It represents the feature information of the eigenvalue (position information) of the first row and the first column on the feature matrix.

[0084] Combination Figure 6The process of encoding each eigenvalue in the feature matrix and obtaining the eigenvalue encoding information is introduced in more detail; Figure 6 As shown, first, the first row of the feature matrix is ​​input into the encoder, and the feature information of each eigenvalue located in the first row of the feature matrix can be encoded to obtain the first eigenvalue encoding information. The first eigenvalue encoding information can be used to represent the position information and feature information of each eigenvalue located in the first row of the feature matrix. Based on the above description and formula (1), the first eigenvalue encoding information may include (V′ 1,1 , V′ 1,2 , V′ 1,3 , ..., V′ 1,w ); Secondly, the second row of the image features to be identified is input into the encoder, and (V′ 2,1 , V′ 2,2 , V′ 2,3 , ..., V′ 2,w ), then, the third row, the fourth row, ..., and so on of the feature matrix are input to the encoder to obtain the final eigenvalue encoding information, which is expressed as follows:

[0085]

[0086] As shown in formula (2), the eigenvalue encoding information obtained by encoding each eigenvalue in the feature matrix can also be expressed in the form of a matrix.

[0087] S203: Identify the first character in the character sequence contained in the image to be identified based on the feature value coding information combined with the attention algorithm, and after predicting the first character, mask the attention map corresponding to the feature value coding information obtained based on the attention algorithm to obtain the masked attention map.

[0088] Step S203 can be implemented by a decoder, that is, S202 and S203 and the following S204 are implemented by an encoder-decoder network model (such as a character detection model). In the embodiment of the present application, the decoder can be composed of a layer of LSTM network (decoding module) and a layer of attention network (attention module).

[0089] The decoder predicts the characters that have not yet been predicted based on the characters that have been predicted, wherein the mask processing includes: analyzing the feature value encoding information output by the encoder through the attention network to obtain an attention map. In the attention map, the attention area corresponding to the characters that have been predicted is set to 0. In one embodiment, the attention map obtained according to the feature value encoding information output by the encoder can refer to the following formula (3). In the embodiment of the present application, by introducing mask processing mask-attention in the attention network, after predicting a character, the attention area corresponding to the character is masked out in the attention map, that is, the response corresponding to the character is set to zero, thereby reducing the interference of the first character in predicting the second character and avoiding repeated attention.

[0090]

[0091] In the embodiment of the present application, the decoder is trained as a conditional language model based on the predicted character prediction results (y1, y2, ..., yt) and the input feature V′ h,w To predict the next character yt+1, the calculation formula is as follows:

[0092] p(y t+1 |y1,...,y t ,V′)=softmax(W out o t ) (4)

[0093] Among them, the W out is a linear transformation matrix. Softmax() is a function that can map each input value to (0,1). The sum of all output values ​​is 1. It can be considered to be used to calculate probability. In classification tasks, the category with the largest probability after softmax is the predicted output category. t The expression of is as follows (5):

[0094] o t =tanh(Wc[h t ;c t ]) (5)

[0095] Among them, h t is the hidden state of the decoder, which is expressed as follows (6). t is the context feature matrix, which can be calculated based on the feature matrix V′ output by attention and encoder, as shown in the following formula (7). outand Wc are both weight matrices.

[0096] h t =RNN(h t-1 ,[y t-1 ;o t-1 ]) (6)

[0097] e t =α(h t ,{V′ h,w})

[0098] e it =β T tanh(W h h i-1 +W v v′ t )

[0099] α t =softmax(e t ) (7)

[0100] c t =φ({V′ h,w},α t )

[0101] Among them, α t Based on e t The calculated attention weight matrix, α t And the feature matrix V′ output by the encoder h,w As input, after φ transformation, the context feature matrix c is obtained t .e it Is the previous one about e t The α transformation in the formula is a functional expression. β is a hyperparameter, such as a value equal to 0.5. h , W v is the weight matrix. h is the hidden state, and v is the feature matrix output by the encoder above.

[0102] The calculation formula of φ transformation is as follows (8), where φ indicates that the input is a function of these two parameters, that is, the value of position i in the context feature matrix is ​​the feature matrix V′ at time t h,w And the weighted sum of the corresponding attention weight matrix at position i:

[0103] c i =∑ t α it v t (8)

[0104] Among them, c i The calculation function of is the above φ transformation. trepresents the context feature matrix at time t, c i is the i-th element in the context feature matrix at time t.

[0105] In summary, in one embodiment, the present application can be implemented as follows: Figure 7 The character detection model shown is used to realize the recognition of each character on the image to be recognized.

[0106] Based on the description of step S202, it can be known that the eigenvalue encoding information of the feature matrix can be used to represent the position information and feature information of each eigenvalue in the feature matrix. According to the information expressed by the eigenvalue encoding information of the feature matrix, the attention algorithm can be used to identify each character in the character sequence contained in the image to be identified, and then identify the character sequence contained in the image to be identified (such as the content of the seal). In general, after obtaining the eigenvalue encoding information of the feature matrix from the encoder, the eigenvalue encoding information of the feature matrix can be input into the decoder, so that the decoder calls the attention algorithm (or attention mechanism) and the decoding module to predict each character of the character sequence contained in the image to be identified. Among them, character recognition based on the attention algorithm and mask processing can help the decoder to recognize different characters more attentively at different times; for example, the decoder can focus on recognizing the first character on the attention map at the first moment, and after the prediction of the first character is completed, it can focus on recognizing the second character on the attention map at the second moment, and so on.

[0107] In the process of sequentially identifying each character in the image to be identified, since the display modes of each character in the image to be identified are different, such as Figure 3 As shown, the characters in the image to be recognized are not arranged on the same horizontal line, but are distributed in a curved shape. Therefore, when recognizing each character, it is easy to cause repeated recognition of the same character. Based on this, the embodiment of the present application proposes a mask-attention method, which supports masking the attention map related to a character after the prediction of the character is completed. The response logic corresponding to the masked character is set to zero, that is, when the character is recognized, the character does not make any response; this can avoid repeated recognition of the same character in multiple character recognitions, and can improve the efficiency and accuracy of character recognition.

[0108] S204: Based on the first character, predict the second character from the attention map after masking, so as to obtain a character recognition result of the image to be recognized according to the first character and the second character.

[0109] In one embodiment, the above step S203 may specifically include: determining the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information by the attention module in the decoder obtained after training optimization; calculating the value in the attention area in the attention map by the decoding module in the decoder obtained after training optimization to obtain the first character in the character sequence contained in the image to be recognized; setting the value in the attention area in the attention map to a specified value, which may make the attention module no longer pay attention to the area corresponding to the specified value, such as the specified value may be 0, to obtain the attention map after mask processing; on this basis, based on the first character, the implementation method of predicting the second character from the attention map after mask processing may include: determining the attention area corresponding to the second character from the attention map after mask processing by the attention module; calculating the value in the attention area corresponding to the first character and the determined second character by the decoding module to obtain the second character. In one embodiment, since there is a certain positional relationship between the characters in the character sequence, such as in the process of recognizing the characters in the character sequence from left to right, when the first character is located at the left position of the second character, that is, the second character is located at the right position of the first character, the attention module can quickly determine the attention area of ​​the second character based on the attention area of ​​the first character on the attention map.

[0110] Combination Figure 7 , the matrix in the brackets of the attention module can be considered as an attention map. Each value on the attention map is based on the eigenvalue encoding information output by the encoder. The eigenvalue encoding information is used to represent the position information and feature information of each eigenvalue in the feature matrix, and the eigenvalue corresponds to the position and feature of each character. In this way, after obtaining the attention area of ​​the first character, the attention area corresponding to the second character can be quickly determined by the attention area of ​​the first character. Of course, when analyzing the attention area, the attention module can ignore the position information and only consider the eigenvalue. The attention module can be trained and optimized through supervised training.

[0111] After masking the status information corresponding to the first character in the feature value encoding information based on step S203, a masked attention map can be obtained. Then, based on the attention area corresponding to the first character on the attention map, the attention area of ​​the adjacent second character can be quickly determined from the masked attention map to predict the second character. This can be done by analogy, so that each character in the character sequence contained in the image to be recognized can be recognized.

[0112] Of course, if the image to be recognized includes more than the first character and the second character, the embodiment of the present application also supports masking the second character in the feature value encoding information that masks the state information corresponding to the first character, that is, masking the value of the attention area corresponding to the second character on the attention map to obtain an attention map that masks the attention areas corresponding to the first and second characters. Figure 6 As shown in the diagram, the attention areas corresponding to the first character "X1" and the second character "X2" are masked to obtain a masked attention map; and based on the position information of the attention area corresponding to the first character and the position information of the attention area corresponding to the second character, the attention area corresponding to the third character on the attention map can be quickly obtained, and the decoding module can also quickly decode to obtain the third character "X3"; and so on, until all the characters included in the character sequence on the image to be identified are identified; the sequence of the first character, the second character, the third character, ..., which are combined into one, is determined as the character sequence included in the image to be identified, such as Figure 7 As shown, the character sequence of the image to be recognized is "X1X2X3 Company".

[0113] In the embodiment of the present application, each eigenvalue in the feature matrix corresponding to the image to be recognized can be encoded to obtain the eigenvalue encoding information, and each character in the character sequence contained in the image to be recognized can be identified based on the eigenvalue encoding information, thereby realizing rapid acquisition of the character recognition result of the image to be recognized. In addition, after recognizing the first character in the character sequence contained in the image to be recognized, a corresponding masking process of masking attention can be performed, so that in the process of recognizing subsequent characters in the character sequence, repeated recognition of the recognized characters can be avoided, thereby improving the efficiency of character recognition, being able to more quickly recognize each character in the character sequence, and improving the accuracy and speed of character recognition.

[0114] See also Figure 8 , Figure 8 A flow chart of another method for character recognition of an image provided by an exemplary embodiment of the present application is shown; the method for character recognition of an image may be performed by an electronic device, and the electronic device may include: Figure 1b In the terminal 101 or the server 102 in the system shown, the character recognition method for the image includes but is not limited to steps S801-S809.

[0115] S801: Acquire an image, and perform character region recognition processing on the acquired image, and determine the character region to be recognized based on the recognition processing result. Among them, the method of acquiring the image to be recognized is different depending on the execution subject of the method for character recognition of an image proposed in the embodiment of the present application; for example: when the method for character recognition of an image provided in the embodiment of the present application is executed by a server, the server can receive the image to be recognized sent by the terminal; for another example: when the method for character recognition of an image proposed in the embodiment of the present application is executed by a terminal, the terminal can receive the image to be recognized sent by other terminals, or the terminal receives the image to be recognized input by the user.

[0116] A schematic diagram of a user inputting an image to be recognized through a terminal can be seen in Fig. 9 ,like Fig. 9 As shown, an image upload interface 901 is displayed on the terminal screen of the terminal, and the image upload interface 901 includes an input option 9011; when the input option 9011 is triggered, an image candidate interface 902 can be displayed on the terminal screen, and the image candidate interface 902 includes one or more candidate images to be processed, such as candidate image to be processed 9021, candidate image to be processed 9022, candidate image to be processed 9023, ..., and so on; when any candidate image to be processed is selected from one or more candidate images to be processed according to the user's selection operation, such as selecting candidate image to be processed 9021, image 9021 is returned to the image upload interface 901; the image upload interface 901 also includes a submit option 9012. When the user triggers the submit option 9012, the character area recognition processing of the image is started.

[0117] It should be noted that Fig. 9 It is only an exemplary schematic diagram of uploading an image to be processed. The embodiment of the present application also supports other implementation methods to upload images; for example, it supports the image candidate interface 902 to be displayed in the form of a floating window above the image upload interface 901; or, it supports outputting a recognition interface after the user selects an image, and displays a prompt message that it is being recognized in the recognition interface to prompt the user that it is being processed; or, it supports obtaining images by “scanning”; and so on. In summary, the embodiment of the present application does not limit the process of users uploading images to be processed, which is explained here. In addition, the embodiment of the present application also supports the user to determine the character area to be recognized on the terminal screen. This implementation method can be found in Figure 4 ,like Figure 4 As shown, the user can draw the character area to be recognized on the terminal screen. The user's determination of the character area to be recognized can reduce the workload of the terminal to a certain extent and ensure that the character area to be recognized is the area determined by the user.

[0118] The embodiments of the present application support the use of a region detection model to identify a region of characters to be identified in an image to be identified, wherein the region of characters to be identified contains a character sequence; wherein the region detection model may include but is not limited to a one-stage model with a higher detection efficiency, such as the yolov5 model, and the like. Taking the yolov5 model as an example, the yolov5 model integrates a feature pyramid network structure, which can be used to detect character sequences of different sizes in an image (such as different font sizes), so that character sequences of different sizes contained in the image to be identified can be accurately detected; for example, seals containing character sequences of different sizes are detected from the same image to be identified. Among them, an exemplary schematic diagram of identifying character sequences of different sizes from an image can be seen. Fig.10 ,like Fig.10 As shown, the image to be processed includes seal 1, seal 2 and seal 3, and calling the region detection model can accurately detect the character regions to be recognized of seal 1, seal 2 and seal 3, such as the character region to be recognized 1001 includes seal 1, the character region to be recognized 1002 includes seal 2, and the character region to be recognized 1003 includes seal 3.

[0119] S802: Obtain an initial image to be recognized based on the character region to be recognized. After the character region to be recognized containing the character sequence is recognized from the image, a screenshot operation can be performed on the character region to be recognized, thereby obtaining an initial image to be recognized, which contains the character sequence. Fig.10 According to the character area 1001 to be recognized, an initial image 1004 to be recognized can be obtained; according to the character area 1002 to be recognized, an initial image 1005 to be recognized can be obtained; according to the character area 1003 to be recognized, an initial image 1006 to be recognized can be obtained.

[0120] S803: Determine whether the shape of the character sequence in the initial image to be recognized is the target shape. The so-called target shape may refer to a curved shape, such as a circular shape or an elliptical shape. Of course, the target shape may also be other shapes, such as a five-pointed star shape, a prism shape, and the like; the specific shape of the target shape is related to the region detection model described above. The trained region detection model has the function of identifying a character sequence of a certain shape, and the target shape is that shape. For example, if the trained region detection model has the function of identifying a character sequence in the shape of a five-pointed star, the target shape is a five-pointed star; for another example, if the trained region detection model has the function of identifying a character sequence in the shape of an ellipse, the target shape is an elliptical shape; and so on. The embodiment of the present application takes the target shape as a curved shape, including but not limited to: a circular shape and an elliptical shape, as an example for introduction and description.

[0121] S804: If the shape of the character sequence in the initial image to be recognized is not the target shape, the initial image to be recognized is recognized using a method related to the shape of the character sequence to obtain a recognition result of the initial image to be recognized.

[0122] Continue to see Fig.10 , seal 1 and seal 2 are rectangular seals, not seals of the target shape (such as a curved shape), and seal 3 is a seal of the target shape. Then, a rectangular shape-related recognition method can be used to recognize the initial images to be recognized containing seal 1 and seal 2, respectively, to obtain recognition results, which may include: the position recognition result and character recognition result of seal 1 in the initial image to be recognized, and the position recognition result and character recognition result of seal 2 in the initial image to be recognized. Step S805 is performed on the initial image to be recognized containing seal 3 to obtain the recognition result of the image to be recognized containing seal 3. Among them, the embodiment of the present application does not limit which specific rectangular shape-related recognition method is used to recognize the initial images to be recognized containing seal 1 and seal 2, respectively, and is explained here.

[0123] S805: If the appearance shape of the character sequence in the initial image to be recognized is the target shape, the initial image to be recognized is adjusted according to the image adjustment rules to obtain the image to be recognized. The image adjustment rules are related to the sample images input when training the character detection model; in other words, when training the character detection model, the sample images of which size and style are input, then when applying the character detection model, it is also necessary to adjust the initial image to be recognized to an image to be recognized with the same size and style as the sample image according to the image adjustment rules. An exemplary method of adjusting the initial image to be recognized according to the image adjustment rules provided in an embodiment of the present application may include: adjusting the position area where the character sequence in the initial image to be recognized is located to a specified target image position area, and adjusting the size of the initial image to be recognized to any one or a combination of two of the target sizes. Specifically, the image adjustment rules may include: adjusting the position area where the character sequence in the initial image to be recognized is located to the specified target image position area (such as the upper left corner of the initial image to be recognized); or, adjusting the size of the initial image to be recognized to the target size (such as adjusting the length of the initial image to be recognized to 192 pixels, adjusting the width of the initial image to be recognized to 160 pixels, etc.); or, adjusting the position area where the character sequence in the initial image to be recognized is located to the specified target image position area, and adjusting the size of the initial image to be recognized to the target size. The above process adjusts the initial image to be recognized according to the image adjustment rules, and obtains that the style and size of the image to be recognized meet the input requirements of the character detection model, which can be conducive to rapid seal recognition of the image to be recognized, and improve recognition efficiency and recognition accuracy.

[0124] S806: Acquire an image to be recognized, and perform feature analysis on the image to be recognized to obtain a feature matrix, which includes multiple eigenvalues.

[0125] S807: Encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix.

[0126] S808: Identify the first character in the character sequence contained in the image to be identified based on the eigenvalue encoding information combined with the attention algorithm, and after predicting the first character, mask the attention map corresponding to the eigenvalue encoding information obtained based on the attention algorithm to obtain the masked attention map. In one embodiment, the feature matrix is ​​obtained by identifying the image to be identified through a trained and optimized feature extraction network; the eigenvalue encoding information is obtained by encoding each eigenvalue in the feature matrix through a trained and optimized encoder; the first character and the second character are obtained by decoding the eigenvalue encoding information through a trained and optimized decoder, and the decoder includes a decoding module and an attention module based on the attention algorithm.

[0127] S809: Based on the first character, predict the second character from the attention map after masking, so as to obtain a character recognition result of the image to be recognized based on the first character and the second character.

[0128] After executing the relevant steps described in steps S801-S809 on the image to be identified, the recognition result of the image to be identified can be obtained, such as Figure 1b As shown, the recognition result may include: a position recognition result of the display position of the character sequence (or a seal containing the character sequence) in the image to be recognized, and a character recognition result of the character sequence in the image to be recognized.

[0129] It should be noted that the specific implementation process of steps S806-S809 can be found in Figure 2 The relevant description of the specific implementation process shown in steps S201-S204 in the illustrated embodiment is not repeated here.

[0130] In the embodiment of the present application, the display position of the character sequence in the image to be recognized is first detected, and the character area to be recognized containing the character sequence in the image to be recognized can be obtained, and then the initial image to be recognized is generated according to the character area to be recognized, and then the initial image to be recognized is adjusted to obtain the image to be recognized containing the character sequence. In the above process, the display position of the character sequence in the image to be recognized is first detected, and then the character sequence contained in the image to be recognized is recognized, so that the image recognition is concentrated in a partial area, which effectively improves the efficiency of character recognition of the image, and after recognizing the first character in the character sequence contained in the image to be recognized, the corresponding masking processing of masking attention can be performed, so that in the process of recognizing subsequent characters in the character sequence, repeated recognition of recognized characters can be avoided, the efficiency of character recognition can be improved, and each character in the character sequence can be recognized more quickly, thereby improving the accuracy and speed of character recognition.

[0131] See also Fig.11a , Fig.11a A schematic diagram of a flow chart of a method for character recognition of an image provided by an exemplary embodiment of the present application is shown. Fig.11a The present invention mainly introduces the process of identifying a character sequence in an image to be identified based on a character detection model; the character recognition method for an image can be executed by an electronic device, and the character recognition method for an image includes but is not limited to steps S1101-S1102.

[0132] S1101: Obtain an image to be recognized.

[0133] S1102: Using the image to be recognized as an input of a character detection model, and obtaining a character recognition result of the image to be recognized according to an output result of the character detection model.

[0134] Among them, the character detection model includes a feature extraction network, an encoder and a decoder; the feature extraction network is used to perform feature analysis on the image to be recognized to obtain a feature matrix; the encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network; the decoder is used to predict the eigenvalue encoding information output by the encoder to obtain the first character, and after predicting the first character, mask the eigenvalue encoding information to obtain the masked eigenvalue encoding information; based on the first character, the second character is predicted from the masked eigenvalue encoding information to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0135] It should be noted that the specific implementation process of steps S1101-S1102 can be found in Figure 2 In the illustrated embodiment, steps S201-S204, and Figure 8The relevant description of the specific implementation process of steps S806-S809 in the illustrated embodiment is not repeated here.

[0136] The above description Figure 2 , Figure 8 as well as Fig.11a The embodiment mainly introduces the application process of the network model, that is, the process of calling the network model to identify the image to be identified or any image to obtain the recognition result. As described above, the network model may include: a region detection model, a character detection model; for example: calling the region detection model to detect the display position of the character sequence in any image; and for example: calling the character detection model to identify the character sequence in any image to be identified. Among them, the region detection model may include but is not limited to models such as the yolov5 model. The yolov5 model is integrated with a feature pyramid and can be used to identify character sequences of different sizes in the same image, such as identifying seals of different sizes in the same image, and then being able to identify multiple seals in the same image. The character detection model may include but is not limited to models such as the neural machine translation model (NMT). The neural machine translation model can directly identify curved or multi-line text without correction or recognition of text lines first and then line recognition. Based on the advantages of the yolov5 model and the neural machine translation model described above, the embodiment of the present application takes the region detection model as the yolov5 model and the character detection model as the neural machine translation model as an example to introduce the training process of the network model.

[0137] The network model is trained by supervised learning, such as the character detection model. Supervised learning and unsupervised learning are relative concepts. To determine whether there is supervision, we can determine whether the sample data used in training has labels. If the input sample data has labels, the training results of the sample data can be compared with the labels to determine whether the network model has reached the convergence condition. If the difference between the training result and the label meets the difference threshold, it means that the network model can better identify the sample data, and then it is determined that the network model has reached the convergence condition. The above-mentioned training method in which the input sample data contains labels can be called supervised learning; otherwise, it is determined to be unsupervised learning.

[0138] The following is combined with Fig.11b, the process of training the network model is introduced. Among them, the server 102 is often equipped with a GPU chip and has strong computing power, and the terminal 101 can provide a user interaction interface and is a device that can directly interact with the user; based on this, the embodiment of the present application supports model training of the network model on the server 102 side, which can enable the network model to converge faster and better, and thus obtain a network model with better performance. In a specific implementation, the training process of the server using supervised learning to train the network model may include: training of the region detection model, and training of the character detection model. Fig.11b As shown, training the network model may include steps S1111-S1114.

[0139] S1111: Obtain a sample data set. The sample data set includes two parts of data: a real data set and an expanded data set. The real data set includes the collected training images including character sequences. Such training images may refer to: collected data containing character sequences (such as seal character sequences), such as documents or images such as bills, waybills, contracts, character verification codes, etc. The expanded data set includes: expanded images obtained after adjusting the collected training images. The expanded images may refer to: data obtained by data expansion of the real data, wherein data expansion may include but is not limited to: rotating the real data (such as bills), adjusting brightness and contrast, adding noise, and cropping the real data such as images. Among them, data expansion (data augmentation), also known as data enhancement or data expansion, is used to generate more data from limited data and increase the number and diversity of training samples. By using data expansion, the number of sample data can be enriched, thereby avoiding the overfitting problem of the region detection model during the training process, and improving the performance of the trained region detection model, making the region detection model more stable. The following is combined with Fig.12 Explain data augmentation, such as Fig.12 As shown, assuming that the real data is an invoice image 1201 containing a seal, the invoice image 1201 is cropped to obtain a cropped invoice image, such as invoice image 1202; the invoice image 1201 is rotated to obtain a rotated invoice image, such as invoice image 1203; and so on. It can be understood that the above-mentioned multiple data augmentation methods can also be performed on the real data to obtain augmented data; for example, the real data is first rotated, and then the rotated image is cropped, etc. The specific implementation method of data augmentation is not limited in the embodiment of the present application, and is described here.

[0140] In addition, in addition to the sample data, the sample data set also includes labels corresponding to the sample data. The label of any sample data is used to indicate the display position or display area of ​​the character sequence (such as a seal containing the character sequence) in the sample data. Among them, the exemplary process of labeling the sample data and obtaining the label of the sample data may include: assuming that the sample data is a bill image, a coordinate system may be established on the bill image, so that the display position or display area of ​​the seal on the bill image can be labeled in the form of coordinates; of course, the way to label the sample data may change with the style of the sample data. The embodiment of the present application does not limit how to label the sample data specifically, and it is explained here.

[0141] S1112: Use the sample data set to train the region detection model to obtain the trained region detection model. In the specific implementation, a sample data can be selected from the sample data set, and the sample data can be input into the region detection model for identification to obtain the prediction result for the sample data, which can refer to the display position of the predicted character sequence (such as a seal containing a character sequence) in the sample data. The loss value (or difference value) is calculated between the prediction result of the sample data and the label of the sample data to determine the training quality of the current region detection model. When the loss value between the prediction result of the sample data and the label of the sample data is less than the loss threshold, it is determined that the region detection model can more accurately predict the display position of the seal in the sample data, and it is determined that the region detection model has converged to obtain a trained region detection model. On the contrary, when the loss value between the prediction result of the sample data and the label of the sample data is less than or equal to the loss threshold, it is determined that the region detection model cannot accurately predict the display position of the character sequence in the sample data, and it is determined that the region detection model has not converged. At this time, it is necessary to adjust the parameters in the region detection model according to the loss value to obtain an optimized region detection model. Then select another sample data from the sample data set, and input the other sample data into the optimized region detection model, and repeat the above steps until the region detection model meets the training requirements. It should be noted that different region detection models may have different loss thresholds, which will not be described in detail here.

[0142] In summary, based on steps S1111 - S1112 , a trained region detection model can be obtained, and the trained region detection model can relatively accurately identify the display position of a character sequence in any image containing the character sequence.

[0143] S1113: Obtain a training sample set, which is used to train the character detection model. Among them, the training sample set used in training the character detection model includes: a sample image set and an expanded image set. The sample image set includes multiple sample images, and these sample images may include: images downloaded directly from the Internet, or images contained in the sample data set described in step S1111. The expanded images in the expanded image set include: images obtained by data expansion of sample images in the sample image set, and images constructed based on other data; among them, ① the process of data expansion of sample images in the sample image set can be referred to the description of step S1111, and the process of data expansion of real data will not be described in detail here. ② The images constructed based on other data include: any one of the first image set and the second image set, or a combination of both. The first image set and the second image set are image sets constructed by different construction methods; specifically, the first image in the first image set is obtained by performing a curvature processing on the collected horizontal character string; more specifically, the first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by collecting and determining from the edge of the horizontal character string, and the n second control points are determined in the target function curve according to the n first control points; n is an integer greater than zero. The second image in the second image set is obtained by performing a curvature rendering on a plurality of characters extracted from a character set; more specifically, the second image includes: generated after rendering a plurality of characters in a target canvas; wherein each character is extracted from the character set, and the position and display direction of each character in the target canvas on the target canvas are calculated according to the target curvature rule.

[0144] The following is a more detailed description of the two construction methods described above:

[0145] (1) The horizontal string collected is subjected to a curvilinear transformation process to obtain a first image; in other words, the first image is obtained by stretching the horizontal string into a curve using a curvilinear transformation function. Among them, the horizontal string is a string composed of one or more characters such as Chinese characters, English characters, and punctuation marks, and the string is horizontally distributed. For example, the horizontal string is "XXX registration number". The curvilinear transformation function is an interpolation function. For example, the curvilinear transformation function includes a thin plate spline interpolation function (TPS function). The thin plate spline interpolation function is a smooth interpolation curve generated by using a data function for some limited point values, by controlling the estimated variance, and using some feature interfaces and polynomial fitting methods. In specific implementation, first, n first control points are uniformly selected at the edge of the horizontal string, where n is an integer greater than zero. For example, n is 8; second, n second control points corresponding to the n first control points are selected on the target function curve (such as if the display shape of the seal is an ellipse, the target function curve can be an ellipse function curve) by simulating the shape of the curve; finally, the n first control points and the n second control points are input into the curvilinear transformation function (such as the thin plate spline interpolation function), and the curvilinear image after stretching the horizontal string can be obtained, and the curvilinear image is enriched to obtain the first image, that is, the first image is an image of a character sequence containing a curve based on the expansion of the horizontal string. See Fig.13 , assuming that the obtained horizontal string is "XXX registration number", then after performing the steps described above on the horizontal string, a curvilinear image can be obtained. Each character in the curvilinear image is distributed by simulating the shape of the curve. Enriching the curvilinear image can obtain the first image.

[0146] (2) A second image is generated by rendering multiple characters in the target canvas; in other words, the second image is generated by generating character images based on the characters in the character set and then rendering and fitting the curvilinear shape to render multiple character images in the target canvas. Among them, the character set includes multiple characters, such as the character "中", the character "老", "城", etc. One or more characters can be extracted from the character set according to the user's意愿. The target canvas can refer to a blank canvas that supports adding or modifying any elements (such as characters, images, animations, etc.) in the canvas. In specific implementation, first, a target canvas with a target size is generated. The target size is related to the input size of the character detection model. For example, the target size is 300 pixels * 300 pixels; second, the positions and display directions (or display angles) of the preset characters in the target canvas are calculated according to the target function curve, and the multiple characters extracted from the character set are rendered into single-character images using the character rendering method, that is, one character is rendered into one single-character image, and the single-character image only contains the character; finally, the rendered single-character images are pasted according to the predicted positions and display directions of each character in the target canvas, and so on, to obtain the second image. See Fig.14, assuming the characters extracted from the character set include: "中", "国", "税", "务", "局", then by using the character rendering method to render each character, single-character images corresponding to each character can be obtained; and each single-character image is pasted into the target canvas according to the preset position and display direction to obtain the second image.

[0147] In summary, based on the various data augmentation methods described above, a relatively rich training sample set can be obtained. Using the rich training sample set to train the character detection model can train a character detection model with stable performance and high recognition efficiency.

[0148] S1114: Use the training sample set to train the character detection model to obtain the trained character detection model. In a specific implementation, a sample image can be selected from the training sample set, and the sample image is input into the character detection model to obtain a character recognition result, and the parameters of the character detection model are adjusted according to the character recognition result to optimize the character detection model. An exemplary schematic diagram of a character detection model can be seen in Fig.15 , such as Fig.15 As shown, assume that the training sample set contains a sample image, and the curved character sequence contained in the sample image is "X1X2X3 Company"; now the sample image is input into the character detection model, so that the character detection model performs recognition processing on the sample image to obtain a character recognition result. Among them, the specific recognition process may include: First, detect whether the size of the input sample image is the target size. If the size of the sample image is not the target size, the sample image is adjusted according to the image adjustment rule to obtain a sample image that meets the size; Second, input the sample image into a neural network (CNN) for feature analysis to obtain a sample feature matrix corresponding to the sample image; Second, then input the sample feature matrix row by row into the encoder to facilitate the encoder to encode the eigenvalues of the sample feature matrix to obtain eigenvalue encoding information; Finally, based on the eigenvalue encoding information, multiple characters of the character sequence contained in the sample image are recognized to obtain a character recognition result. In addition, the difference value between the character recognition result and the label of the sample image is also calculated. If the difference value is less than the difference threshold, it is determined that the character detection model converges, and the current character detection model is determined as the trained character detection model; if the difference value is greater than or equal to the difference threshold, it is determined that the character detection model does not meet the training conditions, that is, the character detection model does not converge, and another sample image is selected from the training sample set for the above process until the character detection model converges to obtain the trained character detection model. The trained character detection model can more accurately recognize the seal content in the image to be recognized. It should be noted that the difference thresholds of different character detection models may be different, which will not be elaborated here.

[0149] In an embodiment of the present application, on the one hand, data expansion processing can be performed on real data to obtain expanded sample data; on the other hand, it supports the construction of expanded images to be recognized based on horizontal strings or multiple characters; this allows limited data to generate more data, enriches the quantity and diversity of sample data, and thus avoids the overfitting problem of network models (such as character detection models or character detection models) during the training process, improves the performance of the trained network model, and makes the network model more stable. In this way, when the network model is used to recognize the image to be recognized, a recognition result with higher accuracy can be obtained.

[0150] Fig.15 The structure diagram of a character recognition device for an image provided by an embodiment of the present application is shown; the character recognition device for an image is arranged in a terminal or a server. In one embodiment, the character recognition device for an image can be used to perform Figure 2 , Figure 8 In the embodiment shown, the character recognition device for the image may be a client running on a server (such as an application with a seal recognition function); the specific implementation of the units included in the character recognition device for the image may refer to the description of the relevant contents in the above embodiment. Fig.15 , the character recognition device for a pair of images in the embodiment of the present application includes the following units:

[0151] The acquisition unit 1501 is used to acquire the image to be recognized, and perform feature analysis on the image to be recognized to obtain a feature matrix, which includes multiple eigenvalues;

[0152] A processing unit 1502 is used to encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix;

[0153] The processing unit 1502 is further used to identify the first character in the character sequence contained in the image to be identified based on the feature value encoding information combined with the attention algorithm, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm to obtain the masked attention map;

[0154] The processing unit 1502 is further used to predict the second character from the attention map after masking based on the first character, so as to obtain a character recognition result of the image to be recognized based on the first character and the second character.

[0155] In one implementation, the feature matrix is ​​obtained by training an optimized feature extraction network to identify the image to be identified;

[0156] The eigenvalue encoding information is obtained by encoding each eigenvalue in the feature matrix through the trained optimized encoder;

[0157] The first character and the second character are obtained by decoding the feature value encoding information through a trained and optimized decoder, and the decoder includes a decoding module and an attention module based on an attention algorithm.

[0158] In one implementation, the processing unit 1502 is used to identify the first character in the character sequence contained in the image to be identified based on the feature value encoding information combined with the attention algorithm, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm, and when the masked attention map is obtained, it is specifically used to:

[0159] The attention module in the decoder obtained through training optimization determines the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information;

[0160] The decoding module in the decoder obtained through training optimization operates on the value in the attention area of ​​the attention map to obtain the first character in the character sequence contained in the image to be recognized;

[0161] Set the value in the attention area of ​​the attention map to the specified value to obtain the masked attention map;

[0162] When the processing unit 1502 is used to predict the second character from the attention map after the mask processing based on the first character, it is specifically used to:

[0163] Determine the attention area corresponding to the second character from the attention map after masking by the attention module;

[0164] The decoding module operates on the first character and the value in the attention area corresponding to the determined second character to obtain the second character.

[0165] In one implementation, the processing unit 1602 is further configured to:

[0166] Acquire an image to be recognized, perform character region recognition processing on the image to be recognized, and determine the character region to be recognized according to the recognition processing result;

[0167] Obtaining an initial image to be recognized according to the character area to be recognized;

[0168] Adjusting the initial image to be recognized according to the image adjustment rule to obtain the image to be recognized;

[0169] The adjusting of the initial image to be recognized includes: adjusting the position area where the character sequence in the initial image to be recognized is located to the designated target image position area, and adjusting the size of the initial image to be recognized to any one or a combination of two of the target sizes.

[0170] In one implementation, the initial image to be recognized is recognized by a region detection model; the region detection model is trained by supervised learning;

[0171] The sample data set used in training the character detection model includes: a real data set and an expanded data set;

[0172] The real data set contains real data and the extended data set contains extended data, and the extended data is obtained by performing data augmentation processing on the real data.

[0173] In one implementation, each character in the character sequence contained in the image to be recognized is recognized by a character detection model; the character detection model is trained by supervised learning;

[0174] The sample data set used in training the region detection model includes: a real data set and an expanded data set;

[0175] The real data set includes: the collected training images including character sequences, and the extended data set includes: the extended images obtained after adjusting the collected training images, wherein the adjustment processing of the collected training images includes: any one or more of rotation processing, brightness change processing, contrast change processing, image noise increase processing, and image cropping processing.

[0176] In one implementation, each character in the character sequence contained in the image to be recognized is recognized by a character detection model; the character detection model is trained by supervised learning;

[0177] The training sample set used in training the character detection model includes: a sample image set and an expanded image set;

[0178] The expanded image set includes: any one of the first image set and the second image set or a combination of the two;

[0179] The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

[0180] In one implementation, the first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by collecting and determining from the edges of the horizontal character string, and the n second control points are determined in the target function curve based on the n first control points; n is an integer greater than zero.

[0181] In one implementation, the second image includes: generated after rendering multiple characters in a target canvas; each character is extracted from a character set, and the position and display direction of each character in the target canvas are calculated according to a target curve rule.

[0182] In another embodiment, the character recognition device for the image can be used to perform Figure 2 , Figure 8 as well as Fig.11a In the embodiment shown, the character recognition device for the image may be a client running on a server (such as an application with a seal recognition function); the specific implementation of the units included in the character recognition device for the image may refer to the description of the relevant contents in the above embodiment. Fig.15 , the character recognition device for a pair of images in the embodiment of the present application includes the following units:

[0183] An acquisition unit 1501 is used to acquire an image to be recognized;

[0184] The processing unit 1502 is used to use the image to be recognized as an input of the character detection model, and obtain a character recognition result of the image to be recognized according to an output result of the character detection model;

[0185] Among them, the character detection model includes a feature extraction network, an encoder and a decoder;

[0186] The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix;

[0187] The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network;

[0188] The decoder is used to predict the feature value coding information output by the encoder to obtain the first character, and after predicting the first character, perform mask processing on the feature value coding information to obtain the masked feature value coding information; based on the first character, predict the second character from the masked feature value coding information to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0189] In one implementation, the character detection model is trained based on a training sample set; the training sample set includes: a sample image set and an expanded image set;

[0190] The expanded image set includes: any one of the first image set and the second image set or a combination of the two;

[0191] The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

[0192] In one implementation, the first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by sampling from edges of a horizontal character string, and the n second control points are determined in a target function curve based on the n first control points; n is an integer greater than zero;

[0193] And, the second image includes: generated after rendering multiple characters in the target canvas; each character is extracted from the character set, and the position and display direction of each character in the target canvas are calculated according to the target curve rule.

[0194] According to one embodiment of the present application, Fig.15 The various units in the method for character recognition of images shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the device for character recognition of images can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). Figure 2 , Figure 8 as well as Fig.11b A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.15 The character recognition device for an image shown in and the character recognition method for an image in the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0195] In the embodiment of the present application, the acquisition unit 1501 can acquire the image to be recognized, and the processing unit 1502 encodes each eigenvalue in the feature matrix corresponding to the image to be recognized, obtains the eigenvalue encoding information, and recognizes each character in the character sequence contained in the image to be recognized based on the eigenvalue encoding information, thereby realizing the rapid acquisition of the character recognition result of the image to be recognized. In addition, after recognizing the first character in the character sequence contained in the image to be recognized, the processing unit 1502 can mask the information area corresponding to the first character on the eigenvalue encoding information, so that in the process of recognizing the second character in the character sequence, the first character can be avoided from being repeatedly recognized, thereby improving the efficiency of character recognition, being able to more quickly recognize each character in the character sequence, and improving the accuracy and speed of character recognition.

[0196] Fig.16 FIG. 1 shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Fig.16 , the electronic device includes a storage device 1601 and a processor 1602. In the embodiment of the present application, the electronic device also includes a network interface 1603 and a user interface 1604. The electronic device can be a smart phone, a tablet computer, a smart wearable device, etc., which can access the Internet through the network interface 1603, communicate with servers and other electronic devices, and exchange data. The user interface 1604 can include a touch display screen, etc., which can receive user operations and can also display various interfaces to the user to facilitate receiving user operations.

[0197] The storage device 1601 may include a volatile memory, such as a random-access memory (RAM); the storage device 1601 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 1601 may also include a combination of the above-mentioned types of memory.

[0198] The processor 1602 may be a central processing unit (CPU). The processor 1602 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. The PLD may be a field-programmable gate array (FPGA), a generic array logic (GAL), etc.

[0199] The storage device 1601 of the embodiment of the present application stores a computer program, and the processor 1602 calls the computer program in the storage device. When the computer program is executed, the processor 1602 can be used to implement the above-mentioned Figure 2 , Figure 8 as well as Fig.11a The methods described in the corresponding embodiments, etc.

[0200] In one embodiment, the electronic device may correspond to a terminal or a server; the storage device 1601 stores a computer program; the processor 1602 loads and executes the computer program to implement the corresponding steps in the above-mentioned method for character recognition of an image; specifically, the processor 1602 is used to perform the following steps:

[0201] Acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes multiple eigenvalues;

[0202] Encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix;

[0203] Based on the feature value coding information combined with the attention algorithm, the first character in the character sequence contained in the image to be identified is identified, and after the first character is predicted, the attention map corresponding to the feature value coding information obtained based on the attention algorithm is masked to obtain the attention map after the masking process;

[0204] Based on the first character, a second character is predicted from the attention map after masking, so as to obtain a character recognition result of the image to be recognized according to the first character and the second character.

[0205] In one implementation, the feature matrix is ​​obtained by training an optimized feature extraction network to identify the image to be identified;

[0206] The eigenvalue encoding information is obtained by encoding each eigenvalue in the feature matrix through the trained optimized encoder;

[0207] The first character and the second character are obtained by decoding the feature value encoding information through a trained and optimized decoder, and the decoder includes a decoding module and an attention module based on an attention algorithm.

[0208] In one implementation, the processor 1602 is used to identify the first character in the character sequence contained in the image to be identified based on the feature value encoding information combined with the attention algorithm, and after predicting the first character, mask the attention map corresponding to the feature value encoding information obtained based on the attention algorithm, and when the masked attention map is obtained, it is specifically used to perform the following steps:

[0209] The attention module in the decoder obtained through training optimization determines the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information;

[0210] The decoding module in the decoder obtained through training optimization operates on the value in the attention area of ​​the attention map to obtain the first character in the character sequence contained in the image to be recognized;

[0211] Set the value in the attention area of ​​the attention map to the specified value to obtain the masked attention map;

[0212] When the processor 1602 is used to predict the second character from the attention map after the mask processing based on the first character, it is specifically used to perform the following steps:

[0213] Determine the attention area corresponding to the second character from the attention map after masking by the attention module;

[0214] The decoding module operates on the first character and the value in the attention area corresponding to the determined second character to obtain the second character.

[0215] In one implementation, the processor 1602 is further configured to execute the following steps:

[0216] Acquire an image to be recognized, perform character region recognition processing on the image to be recognized, and determine the character region to be recognized according to the recognition processing result;

[0217] Obtaining an initial image to be recognized according to the character area to be recognized;

[0218] Adjusting the initial image to be recognized according to the image adjustment rule to obtain the image to be recognized;

[0219] The adjusting of the initial image to be recognized includes: adjusting the position area where the character sequence in the initial image to be recognized is located to the designated target image position area, and adjusting the size of the initial image to be recognized to any one or a combination of two of the target sizes.

[0220] In one implementation, the initial image to be recognized is recognized by a region detection model; the region detection model is trained by supervised learning;

[0221] The sample data set used in training the character detection model includes: a real data set and an expanded data set;

[0222] The real data set contains real data and the extended data set contains extended data, and the extended data is obtained by performing data augmentation processing on the real data.

[0223] In one implementation, each character in the character sequence contained in the image to be recognized is recognized by a character detection model; the character detection model is trained by supervised learning;

[0224] The sample data set used in training the region detection model includes: a real data set and an expanded data set;

[0225] The real data set includes: the collected training images including character sequences, and the extended data set includes: the extended images obtained after adjusting the collected training images, wherein the adjustment processing of the collected training images includes: any one or more of rotation processing, brightness change processing, contrast change processing, image noise increase processing, and image cropping processing.

[0226] In one implementation, each character in the character sequence contained in the image to be recognized is recognized by a character detection model; the character detection model is trained by supervised learning;

[0227] The training sample set used in training the character detection model includes: a sample image set and an expanded image set;

[0228] The expanded image set includes: any one of the first image set and the second image set or a combination of the two;

[0229] The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

[0230] In one implementation, the first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by collecting and determining from the edges of the horizontal character string, and the n second control points are determined in the target function curve based on the n first control points; n is an integer greater than zero.

[0231] In one implementation, the second image includes: generated after rendering multiple characters in a target canvas; each character is extracted from a character set, and the position and display direction of each character in the target canvas are calculated according to a target curve rule.

[0232] In another embodiment, the electronic device may correspond to a terminal or a server; the storage device 1601 stores a computer program; the processor 1602 loads and executes the computer program to implement the corresponding steps in the above-mentioned method for character recognition of an image; specifically, the processor 1602 is used to perform the following steps:

[0233] Obtain an image to be recognized;

[0234] The image to be recognized is used as the input of the character detection model, and the character recognition result of the image to be recognized is obtained according to the output result of the character detection model;

[0235] Among them, the character detection model includes a feature extraction network, an encoder and a decoder;

[0236] The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix;

[0237] The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network;

[0238] The decoder is used to predict the feature value coding information output by the encoder to obtain the first character, and after predicting the first character, perform mask processing on the feature value coding information to obtain the masked feature value coding information; based on the first character, predict the second character from the masked feature value coding information to obtain the character recognition result of the image to be recognized based on the first character and the second character.

[0239] In one implementation, the character detection model is trained based on a training sample set; the training sample set includes: a sample image set and an expanded image set;

[0240] The expanded image set includes: any one of the first image set and the second image set or a combination of the two;

[0241] The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

[0242] In one implementation, the first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by sampling from edges of a horizontal character string, and the n second control points are determined in a target function curve based on the n first control points; n is an integer greater than zero;

[0243] And, the second image includes: generated after rendering multiple characters in the target canvas; each character is extracted from the character set, and the position and display direction of each character in the target canvas are calculated according to the target curve rule.

[0244] In the embodiment of the present application, each eigenvalue in the feature matrix corresponding to the image to be identified can be encoded to obtain eigenvalue encoding information, and each character in the character sequence contained in the image to be identified can be identified based on the eigenvalue encoding information, thereby realizing rapid acquisition of the character recognition result of the image to be identified. In addition, after identifying the first character in the character sequence contained in the image to be identified, the information area corresponding to the first character in the eigenvalue encoding information can be masked, so that in the process of identifying the second character in the character sequence, repeated recognition of the first character can be avoided, thereby improving the efficiency of character recognition, being able to more quickly identify each character in the character sequence, and improving the accuracy and speed of seal recognition.

[0245] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device. In addition, a computer program (including program code) suitable for being loaded and executed by the processor 1602 is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0246] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the character recognition method for an image provided in the above-mentioned various optional modes.

[0247] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0248] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0249] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for character recognition of an image, characterized in that: include: Acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes a plurality of eigenvalues; Encoding each eigenvalue in the feature matrix to obtain eigenvalue encoding information, where the eigenvalue encoding information is used to represent position information and feature information of each eigenvalue in the feature matrix; The attention module in the decoder obtained through training optimization determines the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information; The decoding module in the decoder obtained after training optimization operates on the value of the attention area in the attention map to obtain the first character in the character sequence contained in the image to be recognized; Setting the value in the attention area of ​​the attention map to a specified value to obtain a masked attention map; Determining, by the attention module, an attention area corresponding to the second character from the attention map after masking; The decoding module operates the values ​​in the attention area corresponding to the first character and the determined second character to obtain the second character, so as to obtain a character recognition result of the image to be recognized based on the first character and the second character.

2. The method according to claim 1, characterized in that The feature matrix is ​​obtained by training an optimized feature extraction network to identify the image to be identified; The eigenvalue encoding information is obtained by encoding each eigenvalue in the feature matrix through a trained and optimized encoder; The first character and the second character are obtained by decoding the feature value encoding information through a trained and optimized decoder, and the decoder includes a decoding module and an attention module based on an attention algorithm.

3. The method according to claim 1, characterized in that Before acquiring the image to be identified, the method further includes: Acquire an image, perform character region recognition processing on the acquired image, and determine the character region to be recognized based on the recognition processing result; Obtaining an initial image to be recognized according to the character area to be recognized; Adjusting the initial image to be recognized according to an image adjustment rule to obtain the image to be recognized; Among them, adjusting the initial image to be recognized includes: adjusting the position area where the character sequence in the initial image to be recognized is located to the specified target image position area, and adjusting the size of the initial image to be recognized to any one or a combination of two of the target sizes.

4. The method according to claim 3, characterized in that The initial image to be identified is identified by a region detection model; the region detection model is trained by supervised learning; The sample data set used in training the region detection model includes: a real data set and an expanded data set; The real data set includes: a collected training image including a character sequence, and the extended data set includes: an extended image obtained after adjusting the collected training image, wherein the adjustment processing of the collected training image includes: any one or more of rotation processing, brightness change processing, contrast change processing, image noise increase processing, and image cropping processing.

5. The method according to claim 1, characterized in that Each character in the character sequence contained in the image to be recognized is recognized by a character detection model; the character detection model is trained by supervised learning; The training sample set used when training the character detection model includes: a sample image set and an expanded image set; The extended image set includes: any one of the first image set and the second image set or a combination of the two; The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

6. The method according to claim 5, characterized in that The first image is obtained by calculating n first control points and n second control points through a curvature processing function, the n first control points are determined by collecting and determining from the edges of the horizontal character string, and the n second control points are determined in the target function curve based on the n first control points; n is an integer greater than zero.

7. The method according to claim 5, characterized in that The second image includes: a plurality of characters rendered in a target canvas; each character is extracted from the character set, and the position and display direction of each character in the target canvas are calculated according to a target curve rule.

8. A method for character recognition of an image, characterized in that: include: Obtain an image to be recognized; Using the image to be recognized as an input of a character detection model, and obtaining a character recognition result of the image to be recognized according to an output result of the character detection model; The character detection model includes a feature extraction network, an encoder and a decoder, and the decoder includes an attention module and a decoding module; The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix; The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network; The decoder is used to determine the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information through the attention module; operate on the values ​​in the attention area in the attention map through the decoding module to obtain the first character in the character sequence contained in the image to be recognized; set the values ​​in the attention area in the attention map to specified values ​​to obtain the attention map after masking; determine the attention area corresponding to the second character from the attention map after masking through the attention module; operate on the values ​​in the attention areas corresponding to the first character and the determined second character through the decoding module to obtain the second character, so as to obtain the character recognition result of the image to be recognized based on the first character and the second character.

9. The method according to claim 8, characterized in that The character detection model is trained based on a training sample set; The training sample set includes: a sample image set and an expanded image set; The extended image set includes: any one of the first image set and the second image set or a combination of the two; The first image in the first image set is obtained by performing a curve processing on the collected horizontal character string, and the second image in the second image set is obtained by performing a curve rendering on a plurality of characters extracted from the character set.

10. The method according to claim 9, characterized in that The first image is obtained by calculating n first control points and n second control points by using a curvature processing function, wherein the n first control points are collected and determined from the edge of the horizontal character string, and the n second control points are determined in the target function curve according to the n first control points; n is an integer greater than zero; And, the second image includes: generated after rendering a plurality of characters in a target canvas; Each character is extracted from the character set, and the position and display direction of each character on the target canvas are calculated according to the target curve rule.

11. A character recognition device for an image, characterized in that: include: An acquisition unit is used to acquire an image to be identified, and perform feature analysis on the image to be identified to obtain a feature matrix, wherein the feature matrix includes a plurality of eigenvalues; A processing unit is used to encode each eigenvalue in the feature matrix to obtain eigenvalue encoding information, wherein the eigenvalue encoding information is used to represent the position information and feature information of each eigenvalue in the feature matrix; an attention module in a decoder obtained after training and optimization determines the attention area corresponding to the first character from the attention map corresponding to the eigenvalue encoding information; a decoding module in a decoder obtained after training and optimization operates on the values ​​in the attention area in the attention map to obtain the first character in the character sequence contained in the image to be recognized; the values ​​in the attention area in the attention map are set to specified values ​​to obtain the masked attention map; the attention module determines the attention area corresponding to the second character from the masked attention map; the decoding module operates on the values ​​in the attention areas corresponding to the first character and the determined second character to obtain the second character, so as to obtain the character recognition result of the image to be recognized based on the first character and the second character.

12. A character recognition device for an image, characterized in that: include: An acquisition unit, used for acquiring an image to be recognized; A processing unit, used to use the image to be recognized as an input of a character detection model, and obtain a character recognition result of the image to be recognized according to an output result of the character detection model; The character detection model includes a feature extraction network, an encoder and a decoder, and the decoder includes an attention module and a decoding module; The feature extraction network is used to perform feature analysis on the image to be identified to obtain a feature matrix; The encoder is used to encode the eigenvalues ​​on the feature matrix output by the feature extraction network; The decoder is used to determine the attention area corresponding to the first character from the attention map corresponding to the feature value encoding information through the attention module; operate on the values ​​in the attention area in the attention map through the decoding module to obtain the first character in the character sequence contained in the image to be recognized; set the values ​​in the attention area in the attention map to specified values ​​to obtain the attention map after masking; determine the attention area corresponding to the second character from the attention map after masking through the attention module; operate on the values ​​in the attention areas corresponding to the first character and the determined second character through the decoding module to obtain the second character, so as to obtain the character recognition result of the image to be recognized based on the first character and the second character.

13. An electronic device, characterized in that: include: a storage device for storing a computer application; The processor executes the computer application stored in the storage device to implement the character recognition method for an image as described in any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer application program, and when the computer application program is executed, the method for character recognition of an image as described in any one of claims 1 to 10 is implemented.