Image-based character recognition method, device, equipment and storage medium

Abstract

The application discloses an image-based character recognition method, device and equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: performing feature extraction on a target image to obtain visual features; performing recognition on the visual features to obtain a plurality of candidate recognition results corresponding to the target image, wherein the candidate recognition results comprise a plurality of recognized characters; determining semantic correlation features of the plurality of characters in the plurality of candidate recognition results; and determining a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The method can reduce individual character recognition errors and improve the accuracy of the character recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a character recognition method and device based on images, equipment and a storage medium. BACKGROUND

[0002] Optical Character Recognition (OCR) refers to a technology that a computer device converts characters in an image into a text format for further editing and processing by a word processing software. At present, for an image containing characters, visual features of the image are usually extracted, and the visual features are recognized to obtain a character recognition result of the image. However, the visual features of some characters are similar, and individual character recognition errors are prone to occur. The accuracy of the character recognition result determined by this method is low. SUMMARY

[0003] The present application provides a character recognition method and device based on images, equipment and a storage medium, which can improve the accuracy of the character recognition result. The technical scheme is as follows:

[0004] In one aspect, a character recognition method based on images is provided, and the method comprises the following steps:

[0005] performing feature extraction on a target image to obtain visual features;

[0006] recognizing the visual features to obtain a plurality of candidate recognition results corresponding to the target image, wherein the candidate recognition results comprise a plurality of recognized characters;

[0007] determining semantic correlation features of the plurality of characters in the plurality of candidate recognition results;

[0008] determining a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results.

[0009] In one aspect, a character recognition device based on images is provided, and the device comprises the following modules:

[0010] a feature extraction module, configured to perform feature extraction on a target image to obtain visual features;

[0011] a recognition module, configured to recognize the visual features to obtain a plurality of candidate recognition results corresponding to the target image, wherein the candidate recognition results comprise a plurality of recognized characters;

[0012] a feature determination module, configured to determine semantic correlation features of the plurality of characters in the plurality of candidate recognition results;

[0013] A result determination module is configured to determine the character recognition result of the target image from the plurality of candidate recognition results based on semantic correlation features of the plurality of characters in the plurality of candidate recognition results.

[0014] In a possible implementation, the result determination module comprises:

[0015] A parameter value determination unit is configured to determine a first recognition parameter value of the plurality of candidate recognition results based on semantic correlation features of the plurality of characters in the plurality of candidate recognition results, where the first recognition parameter value is used to represent a semantic correlation degree of the plurality of characters in the candidate recognition result.

[0016] A result determination unit is configured to determine the character recognition result of the target image from the plurality of candidate recognition results based on the first recognition parameter value of the plurality of candidate recognition results.

[0017] In a possible implementation, the visual features comprise visual sub-features corresponding to the plurality of characters; and the parameter value determination unit is configured to determine the first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results and the visual sub-features corresponding to the plurality of characters.

[0018] In a possible implementation, the parameter value determination unit is configured to, for a first character in the candidate recognition result, determine a first recognition sub-parameter value corresponding to the first character based on a visual sub-feature corresponding to the first character; for a (k+1)th character in the candidate recognition result, determine a first recognition sub-parameter value corresponding to the (k+1)th character based on semantic correlation features of the first k characters and the (k+1)th character and a visual sub-feature corresponding to the (k+1)th character, where k≥1 and k is a positive integer; and determine the first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result.

[0019] In a possible implementation, the result determination unit is configured to obtain a second recognition parameter value of the plurality of candidate recognition results, where the second recognition parameter value is used to represent a matching degree of the candidate recognition result and the visual features; determine a total recognition parameter value of the plurality of candidate recognition results based on the first recognition parameter value and the second recognition parameter value of the plurality of candidate recognition results; and determine the candidate recognition result with the highest total recognition parameter value as the character recognition result.

[0020] In a possible implementation, the second recognition parameter value is determined in a process of recognizing the visual feature to obtain a plurality of candidate recognition results corresponding to the target image; the recognition module is configured to recognize the visual feature to obtain a plurality of candidate recognition results corresponding to the target image and a second recognition parameter value of the plurality of candidate recognition results; and the plurality of candidate recognition results are selected based on the second recognition parameter value of the plurality of candidate recognition results to obtain a candidate recognition result that meets a recognition parameter value condition.

[0021] In a possible implementation, the visual feature of the target image includes visual sub-features corresponding to the plurality of characters; and the recognition module is configured to perform parallel recognition on the visual sub-features corresponding to the plurality of characters to obtain the plurality of candidate recognition results.

[0022] In a possible implementation, the visual feature of the target image includes visual sub-features corresponding to the plurality of characters; and the recognition module is configured to, for a visual sub-feature corresponding to any character, recognize the visual sub-feature to obtain a probability of a plurality of first candidate characters corresponding to the visual sub-feature, select a plurality of second candidate characters that meet a probability requirement condition from the plurality of first candidate characters based on the probability of the plurality of first candidate characters corresponding to the visual sub-feature, and combine the plurality of second candidate characters corresponding to the plurality of visual sub-features to obtain the plurality of candidate recognition results.

[0023] In a possible implementation, the method is performed by a character recognition model, the character recognition model includes a feature extraction sub-model, a first character recognition sub-model, and a second character recognition sub-model, the visual feature is determined by the feature extraction sub-model, the plurality of candidate recognition results are determined by the first character recognition sub-model, and the semantic correlation feature and the character recognition result are determined by the second character recognition sub-model.

[0024] In a possible implementation, the first character recognition sub-model is a parallel recognition sub-model, and the second character recognition sub-model is an autoregressive character recognition sub-model; the character recognition model is configured to recognize an image with character content belonging to a first field; and the apparatus further includes:

[0025] a sample data acquisition module configured to acquire sample data of a second field, the sample data including sample character data and a sample image including the sample character data, the character content of the sample character data belonging to the second field, and the first field and the second field being different fields;

[0026] The training module is configured to identify the sample image by using the character recognition model to obtain predicted character data of the sample image.

[0027] The training module is further configured to train an autoregressive character recognition submodel in the character recognition model based on a difference between the predicted character data and the sample character data to obtain a character recognition model applicable to the second field.

[0028] In a possible implementation, the sample data acquisition module is configured to acquire character data belonging to the second field as sample character data; synthesize a sample image containing the sample character data based on the sample character data; and use the sample character data and the sample image as the sample data.

[0029] In an aspect, a computer device is provided, which includes one or more processors and one or more memories having stored therein at least one program code, which is loaded and executed by the one or more processors to implement operations performed by the image-based character recognition method according to any of the possible implementation manners.

[0030] In an aspect, a computer-readable storage medium is provided, which has stored therein at least one program code, which is loaded and executed by a processor to implement operations performed by the image-based character recognition method according to any of the possible implementation manners.

[0031] In an aspect, a computer program or computer program product is provided, which includes computer program code, which, when executed by a computer, causes the computer to implement operations performed by the image-based character recognition method according to any of the possible implementation manners.

[0032] The image-based character recognition method, apparatus, device, and storage medium provided by the embodiments of the present application consider that individual character recognition errors can lead to incoherent sentences, that is, the characters with recognition errors and other characters are not semantically relevant. Therefore, the embodiments of the present application first determine multiple candidate recognition results through visual features, and then determine a character recognition result from the multiple candidate recognition results based on semantic relevance features of multiple characters in the candidate recognition results, so as to ensure that the semantics of the multiple characters in the character recognition result are relevant, reduce the case of individual character recognition errors, and improve the accuracy of the character recognition result. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0034] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0035] Figure 2 is a flowchart of an image-based character recognition method provided by an embodiment of the present application;

[0036] Figure 3 is a flowchart of an image-based character recognition method provided by an embodiment of the present application;

[0037] Figure 4 is a schematic diagram of a character recognition model provided by an embodiment of the present application;

[0038] Figure 5 is a flowchart of an image-based character recognition method provided by an embodiment of the present application;

[0039] Figure 6 is a structural schematic diagram of an image-based character recognition device provided by an embodiment of the present application;

[0040] Figure 7 is a structural schematic diagram of another image-based character recognition device provided by an embodiment of the present application;

[0041] Figure 8 is a structural schematic diagram of a terminal provided by an embodiment of the present application;

[0042] Figure 9 is a structural schematic diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0044] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the present application, a first order can be referred to as a second order, and a second order can be referred to as a first order.

[0045] As used herein, the terms "at least one", "multiple", "each", "any", include one, two or more, multiple includes two or more, and each refers to each of the corresponding multiple, any refers to any of the multiple, for example, multiple orders include 3 orders, and each refers to each of the 3 orders, any refers to any of the 3 orders, which can be the first, the second or the third.

[0046] It should be noted that the information (including but not limited to user personal information, user equipment information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the orders and cost parameters involved in the present application are obtained under full authorization.

[0047] The image-based character recognition method provided by the embodiments of the present application can be applied to document conversion, license plate recognition, identity card recognition and other scenarios. The embodiments of the present application are only exemplarily described taking the document conversion scenario as an example.

[0048] An image to be recognized is obtained, which can be an image stored locally by a device, an image obtained from other devices, or an image obtained by photographing characters. The embodiments of the present application do not limit the image. The image to be recognized is recognized to obtain a character recognition result of the image. If the image-based character recognition method provided by the embodiments of the present application is used, the accuracy of the obtained character recognition result can be improved.

[0049] The image-based character recognition method provided by the embodiments of the present application is executed by a computer device. In a possible implementation manner, the computer device is a terminal, for example, the terminal is any one of a desktop computer, a tablet computer or a mobile phone. In another possible implementation manner, the computer device is a server. For example, the server can be a server, or a server cluster composed of a plurality of servers, or a cloud computing service center. In another possible implementation manner, the computer device includes a terminal and a server.

[0050] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application, as shown in Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network.

[0051] The terminal 101 installs a target application provided by the server 102, which is optionally an application in the operating system of the terminal 101 or an application provided by a third party. For example, the target application is an image processing application having an image processing function, which can also have other functions such as a commenting function and a sharing function.

[0052] In some embodiments, the terminal 101 sends the target image to the server 102, the server 102 identifies the target image to obtain a character recognition result of the target image, and sends the character recognition result to the terminal 101 for displaying the character recognition result.

[0053] Figure 2 is a flowchart of a character recognition method based on an image provided by an embodiment of the present application. The embodiment of the present application takes a computer device as an example of an execution subject, and includes the following steps:

[0054] 201. The computer device extracts a visual feature from a target image.

[0055] The target image is any image containing characters, which can be an image stored locally by the computer device, an image obtained by the computer device from another device, an image photographed by the computer device, or an image scanned by the computer device from a document. The embodiment of the present application does not limit the target image.

[0056] Different characters have different shapes, so the embodiment of the present application distinguishes different characters by extracting a visual feature. In some embodiments, the visual feature is a feature map of the target image.

[0057] 202. The computer device identifies the visual feature to obtain a plurality of candidate recognition results corresponding to the target image, and the candidate recognition result includes a plurality of recognized characters.

[0058] The visual feature is a visual feature of a plurality of characters, and identifying the visual feature can identify a plurality of characters. It should be noted that the shapes of some characters are relatively similar, so when identifying the visual features of some characters, a plurality of possible characters can be determined, thereby obtaining a plurality of candidate recognition results.

[0059] 203. The computer device determines a semantic correlation feature of the plurality of characters in the plurality of candidate recognition results.

[0060] The plurality of characters included in the target image can be a sentence, a pinyin word, an English word, etc. Whether it is a sentence, a pinyin word, or an English word, there is semantic correlation between the plurality of characters included in the target image, so that the plurality of characters represent a meaning.

[0061] For example, the plurality of characters included in the target image is a sentence, the computer device identifies that the first three characters are "Xiaoming today", and when identifying the fourth character, it is identified that the fourth character can be "day" or "big". According to the semantic correlation of the fourth character and the first three characters, it can be determined that the fourth character is "day".

[0062] As can be seen, the plurality of characters included in the target image has semantic correlation. Therefore, after obtaining a plurality of candidate recognition results, the embodiment of the present application can determine the semantic correlation feature of the plurality of characters in the candidate recognition result to determine whether the plurality of characters in the candidate recognition result have better semantic correlation.

[0063] 204、The computer device determines the character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results.

[0064] If the semantic correlation feature of the plurality of characters in the candidate recognition result indicates that the semantic correlation of the plurality of characters is poor, it means that there is an incorrectly recognized character in the plurality of characters, and the candidate recognition result is not accurate. If the semantic correlation feature of the plurality of characters in the candidate recognition result indicates that the semantic correlation of the plurality of characters is high, it means that the recognition accuracy of the plurality of characters is high, and the candidate recognition result is more accurate.

[0065] The character recognition method based on an image provided by the embodiment of the present application takes into account the case that individual character recognition errors will lead to an incoherent sentence, that is, the incorrectly recognized character and other characters do not have semantic correlation. Therefore, the embodiment of the present application first determines a plurality of candidate recognition results through visual features, and then determines a character recognition result from the plurality of candidate recognition results based on the semantic correlation feature of the plurality of characters in the candidate recognition result, so as to ensure that the semantics of the plurality of characters in the character recognition result are related, reduce the case of individual character recognition errors, and improve the accuracy of the character recognition result.

[0066] Figure 3 is a flowchart of a character recognition method based on an image provided by the embodiment of the present application. The embodiment of the present application takes a computer device as an example of an execution subject and includes the following steps:

[0067] 301、The computer device extracts features from the target image to obtain visual features, the visual features including visual sub-features corresponding to the plurality of characters.

[0068] In some embodiments, the computer device extracts features of the target image by a feature extraction model to obtain the visual features. The feature extraction model is a model used for feature extraction, which can be a CNN (Convolutional Neural Network) model, a RNN (Recurrent Neural Network) model, a combined model of CNN and BLSTM (Bidirectional Long Short-Term Memory Network), or a combined model of CNN and Transformer, where the Transformer is a neural network based on an attention mechanism. The embodiments of the present application do not limit the feature extraction model.

[0069] In some embodiments, the computer device extracts features of the target image by a feature extraction algorithm to obtain the visual features. The embodiments of the present application do not limit the way the computer device extracts features.

[0070] 302. The computer device identifies the visual features to obtain a plurality of candidate recognition results corresponding to the target image, where the candidate recognition results include a plurality of recognized characters.

[0071] In some embodiments, the visual features of the target image include visual sub-features corresponding to a plurality of characters, and the computer device determines the plurality of candidate recognition results by determining candidate characters corresponding to the plurality of visual sub-features. The computer device identifies the visual features to obtain a plurality of candidate recognition results corresponding to the target image, including: for the visual sub-feature corresponding to any character, identifying the visual sub-feature to obtain a probability of a plurality of first candidate characters corresponding to the visual sub-feature, and selecting a plurality of second candidate characters satisfying a probability requirement condition from the plurality of first candidate characters based on the probability of the plurality of first candidate characters corresponding to the visual sub-feature; after obtaining a plurality of second candidate characters corresponding to a plurality of visual sub-features respectively, combining the plurality of second candidate characters corresponding to the plurality of visual sub-features respectively to obtain the plurality of candidate recognition results.

[0072] Wherein, the higher the probability of the visual sub-feature corresponding to the first candidate character, the higher the probability that the visual sub-feature is the visual feature of the first candidate character, and the higher the accuracy of the first candidate character. Optionally, the first candidate character is a character in a character library, and the computer device determines the probability of each character in the character library by the visual sub-feature to obtain the candidate recognition result of the visual sub-feature.

[0073] Optionally, the computer device, for the visual sub-feature corresponding to any character, identifies the visual sub-feature to obtain probabilities of the visual sub-feature corresponding to a plurality of first candidate characters, including: the computer device, for the visual sub-feature corresponding to any character, determines probabilities of the visual sub-feature corresponding to each character based on matching degrees of the visual sub-feature and each character in a character library.

[0074] For example, the character library includes 900 characters, and identifying the visual sub-feature can obtain probabilities of the visual sub-feature corresponding to the 900 characters, such as a probability of the visual sub-feature corresponding to the character "one" being 0.1%, a probability of the visual sub-feature corresponding to the character "four" being 0.1%, a probability of the visual sub-feature corresponding to the character "day" being 80%, and a probability of the visual sub-feature corresponding to the character "big" being 50%.

[0075] Optionally, the first candidate character is determined by the computer device based on the visual sub-feature. For example, the computer device, for the visual sub-feature corresponding to any character, identifies the visual sub-feature to obtain probabilities of the visual sub-feature corresponding to a plurality of first candidate characters, including: the computer device, for the visual sub-feature corresponding to any character, determines a plurality of first candidate characters based on the visual sub-feature and probabilities of the visual sub-feature corresponding to the plurality of first candidate characters.

[0076] In addition, the embodiments of the present application also exemplarily illustrate "selecting a plurality of second candidate characters satisfying a probability requirement condition from the plurality of first candidate characters based on the probabilities of the visual sub-feature corresponding to the plurality of first candidate characters", as follows:

[0077] Optionally, the second candidate character is a first candidate character with the highest probability in the plurality of first candidate characters. The computer device selects a plurality of second candidate characters satisfying a probability requirement condition from the plurality of first candidate characters based on probabilities of the visual sub-feature corresponding to the plurality of first candidate characters, including: the computer device selects a target number of second candidate characters from the plurality of first candidate characters based on the probabilities of the visual sub-feature corresponding to the plurality of first candidate characters, and the probability of the second candidate character is greater than probabilities of remaining first candidate characters.

[0078] The target number can be any integer greater than 1, and the embodiments of the present application do not limit the target number. For example, the target number is 3, and after the computer device determines the probabilities of the visual sub-feature corresponding to the plurality of first candidate characters, the computer device selects the first candidate characters with the top 3 probabilities to obtain the second candidate characters satisfying the probability requirement condition.

[0079] Optionally, the second candidate character is a first candidate character with a probability exceeding a target probability threshold. The computer device selects a second candidate character with a probability exceeding a target probability threshold from the plurality of first candidate characters based on probabilities of the visual sub-feature corresponding to the plurality of first candidate characters.

[0080] wherein the target probability threshold is any numerical value, and embodiments of the present application do not limit the target probability threshold. Optionally, the target probability threshold is an empirical value; optionally, the target probability threshold is a numerical value set by a technician.

[0081] In step 302, the multiple candidate recognition results corresponding to the visual features are determined based on the visual sub-features corresponding to each character to obtain the candidate recognition results. As can be seen, the recognition processes of each character are independent of each other, and therefore, the computer device can sequentially recognize or in parallel recognize the visual sub-features corresponding to the multiple characters when determining the multiple candidate recognition results corresponding to the visual features. In some embodiments, the computer device recognizes the visual features to obtain the multiple candidate recognition results corresponding to the target image, including: in parallel recognizing the visual sub-features corresponding to the multiple characters to obtain the multiple candidate recognition results corresponding to the target image.

[0082] In some embodiments, step 302 is implemented by a first character recognition model. Optionally, the first character recognition model is a parallel character recognition model. In the recognition process of the parallel character recognition model, the recognition processes of each character are independent of each other, and the parallel character recognition model can obtain the character recognition results of the multiple characters through one forward calculation, which is fast in calculation speed and high in recognition efficiency.

[0083] Optionally, the first character recognition model includes a feature extraction layer and a recognition layer, the feature extraction layer is used to further extract features from the visual features in step 401, and the recognition layer is used to recognize the visual features extracted by the feature extraction layer to obtain the candidate recognition results.

[0084] It should be noted that embodiments of the present application do not limit the number of feature extraction layers in the first character recognition model, and do not limit the types of the feature extraction layers and the recognition layer. Optionally, the first character recognition model includes two feature extraction layers and one recognition layer. Optionally, the feature extraction layer is a bidirectional long short-term memory layer, and the recognition layer is a softmax (activation) layer.

[0085] wherein the target function of the recognition layer is a Connectionist temporal classification (CTC) function, which can quickly obtain the candidate recognition results through a locally optimal greedy decoding method, or obtain multiple candidate recognition results through beam search.

[0086] 303、The computer device determines semantic correlation features of the multiple characters in the multiple candidate recognition results.

[0087] In the embodiments of the present application, the semantic correlation feature of the plurality of characters is used to represent whether the semantics of the plurality of characters are correlated, or the semantic correlation feature of the plurality of characters is used to represent the degree of correlation of the semantics of the plurality of characters.

[0088] In some embodiments, the computer device determines the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results, including: the computer device determines the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results through the causal language model.

[0089] In some embodiments, the computer device determines the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results, including: the computer device determines the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results through the self-attention mechanism. Wherein, the self-attention mechanism is to determine the recognition result of the next character through the recognized character, therefore, the computer device can learn the semantic correlation feature of the plurality of characters through the self-attention mechanism.

[0090] 304、The computer device determines the recognition parameter value of the plurality of candidate recognition results based on the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results.

[0091] In some embodiments, the computer device determines the first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results, the first recognition parameter value is used to represent the semantic correlation degree of the plurality of characters in the candidate recognition result; and determines the character recognition result of the target image from the plurality of candidate recognition results based on the first recognition parameter value of the plurality of candidate recognition results.

[0092] Wherein, the higher the semantic correlation degree of the plurality of characters is, the higher the first recognition parameter value is. Optionally, the computer device determines the character recognition result of the target image from the plurality of candidate recognition results based on the first recognition parameter value of the plurality of candidate recognition results, including: the computer device determines the candidate recognition result corresponding to the highest first recognition parameter value as the character recognition result of the target image based on the first recognition parameter value of the plurality of candidate recognition results.

[0093] Wherein, the process of determining the recognition parameter value of the plurality of candidate recognition results based on the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results can be regarded as the process of determining whether the plurality of candidate recognition results are accurate based on the semantic correlation feature of the plurality of characters in the plurality of candidate recognition results.

[0094] The verification of whether the plurality of candidate recognition results is accurate can be based on semantic correlation features of the plurality of characters and can also be verified from a visual perspective, that is, based on visual sub-features corresponding to the plurality of characters. Optionally, the visual features include visual sub-features corresponding to the plurality of characters. The computer device determines a first recognition parameter value of the plurality of candidate recognition results based on semantic correlation features of the plurality of characters in the plurality of candidate recognition results, including: determining the first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results and visual sub-features corresponding to the plurality of characters.

[0095] Optionally, the computer device determines the first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results and visual sub-features corresponding to the plurality of characters, including: for a first character in the candidate recognition result, determining a first recognition sub-parameter value corresponding to the first character based on visual features corresponding to the first character; for a k+1th character in the candidate recognition result, determining a first recognition sub-parameter value corresponding to the k+1th character based on semantic correlation features of the first k characters and the k+1th character and a visual sub-feature corresponding to the k+1th character, where k≥1 and k is a positive integer; and determining a first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result.

[0096] In some embodiments, the first recognition parameter value is determined by the computer device through a second character recognition model. Optionally, the second character recognition model is an autoregressive character recognition model. The second character recognition model can determine a character recognition result of a target image based on visual features of the target image. For example, the visual features of the target image include visual sub-features corresponding to the plurality of characters; and the computer device performs the following steps through the second character recognition model to identify the target image to obtain the character recognition result of the target image: identifying a first visual sub-feature to recognize a first character; identifying a second visual sub-feature and the recognized first character to recognize a second character; and iteratively identifying a k+1th visual sub-feature and the recognized k characters to recognize a k+1th character.

[0097] The process of identifying the k+1th character based on the k+1th visual sub-feature and the recognized k characters by the computer device can be: determining, by the computer device, probabilities of each character in a character library being the k+1th character based on the k+1th visual sub-feature and the recognized k characters, and determining a character corresponding to the highest probability as the k+1th character.

[0098] In the embodiment of the present application, the second character recognition model scores multiple candidate recognition results, and therefore, the processing procedure of the second character recognition model can be: for the first character in the candidate recognition result, based on the visual features corresponding to the first character, determining the probabilities of each character in the character library being the first character, and determining the probability corresponding to the first character in the candidate recognition result as the first recognition sub-parameter value corresponding to the first character; for the k+1th character in the candidate recognition result, based on the visual features corresponding to the first k characters and the k+1th character in the candidate recognition result, determining the probabilities of each character in the character library being the k+1th character, and determining the probability corresponding to the k+1th character in the candidate recognition result as the first recognition sub-parameter value corresponding to the k+1th character; and based on the first recognition sub-parameter values corresponding to the multiple characters in the candidate recognition result, determining the first recognition parameter value corresponding to the candidate recognition result.

[0099] It should be noted that the second character recognition model includes an encoding layer and a decoding layer, wherein the encoding layer is used to further extract the visual features to obtain further visual features, and the decoding layer is used to determine the first recognition parameter value of the candidate recognition result. In the embodiment of the present application, the number of the encoding layer and the decoding layer in the second character recognition model is not limited, for example, the second character recognition model includes two encoding layers and one decoding layer. The encoding layer can be a transformer-encoder layer, and the decoding layer can be a transformer-decoder layer.

[0100] It should be noted that in the embodiment of the present application, the multiple characters in the candidate recognition result are known, and therefore, the second character recognition model does not need to determine the probability of the k+1th character after determining the first k characters, and the second character recognition model can determine the recognition sub-parameter values corresponding to the multiple characters in the candidate recognition result in parallel, thereby accelerating the recognition speed.

[0101] Optionally, the computer device determines the first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, including: the computer device obtains a product value of the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, to obtain the first recognition parameter value corresponding to the candidate recognition result. Optionally, the computer device determines the first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, including: the computer device obtains a sum value of the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, to obtain the first recognition parameter value corresponding to the candidate recognition result. Optionally, the computer device determines the first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, including: the computer device obtains an average value of the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result, to obtain the first recognition parameter value corresponding to the candidate recognition result.

[0102] For example, the first recognition parameter value corresponding to the candidate recognition result is:

[0103]

[0104] wherein p represents a probability, represents the first recognition parameter value of the candidate recognition result, w represents the candidate recognition result, o represents the visual feature, represents the tthcharacter in the candidate recognition result, K represents the number of characters in the candidate recognition result, and represents a continuous multiplication function.

[0105] In some embodiments, the computer device determines the character recognition result of the target image from the plurality of candidate recognition results based on the recognition result parameter values of the plurality of candidate recognition results, including: obtaining the second recognition parameter value of the plurality of candidate recognition results, the second recognition parameter value being used to represent the matching degree of the candidate recognition result and the visual feature; determining the total recognition parameter value of the plurality of candidate recognition results based on the first recognition parameter values and the second recognition parameter values of the plurality of candidate recognition results.

[0106] For example, the sum of the first recognition parameter value and the second recognition parameter value of the candidate recognition result is obtained as the total recognition parameter value of the candidate recognition result. For another example, the first recognition parameter value and the second recognition parameter value of the candidate recognition result are weighted and averaged to obtain the total recognition parameter value of the candidate recognition result.

[0107] Optionally, the second recognition parameter value is determined in a process of recognizing the visual feature to obtain the plurality of candidate recognition results corresponding to the target image. The computer device recognizes the visual feature to obtain the plurality of candidate recognition results corresponding to the target image, including: recognizing the visual feature to obtain the plurality of candidate recognition results corresponding to the target image and the second recognition parameter values of the plurality of candidate recognition results, and selecting the candidate recognition result satisfying the condition of the recognition parameter value from the plurality of candidate recognition results based on the second recognition parameter values of the plurality of candidate recognition results.

[0108] As described in the above step 302, the plurality of candidate recognition results are determined based on the probabilities corresponding to the characters in the candidate recognition result, that is, the second recognition parameter values of the plurality of candidate recognition results are determined based on the probabilities corresponding to the characters in the candidate recognition result. In some embodiments, the computer device determines the second recognition parameter values of the plurality of candidate recognition results, including: the computer device obtains the product value, sum value or average value of the probabilities corresponding to the characters in the candidate recognition result as the second recognition parameter value of the candidate recognition result.

[0109] That is, the embodiments of the present application determine the recognition parameter values determined by the first character recognition model and the second character recognition model to determine the character recognition result.

[0110] 305. The computer device determines the candidate recognition result with the highest recognition parameter value as the character recognition result of the target image.

[0111] In some embodiments, the computer device determines the candidate recognition result with the highest recognition parameter value as the character recognition result of the target image, including: the computer device sorts the plurality of candidate recognition results based on the recognition parameter values of the plurality of candidate recognition results. If the plurality of candidate recognition results are sorted in the order from high to low of the recognition parameter values, the first candidate recognition result is selected as the character recognition result of the target image. If the plurality of candidate recognition results are sorted in the order from low to high of the recognition parameter values, the last candidate recognition result is selected as the character recognition result of the target image.

[0112] In some embodiments, the image-based character recognition method is performed by a character recognition model, as shown in Figure 4 The visual feature is determined by the feature extraction sub-model, the plurality of candidate recognition results are determined by the first character recognition sub-model, and the semantic correlation feature and the character recognition result are determined by the second character recognition sub-model. Optionally, as shown in Figure 5As shown, after the feature extraction sub-model determines the visual feature, the visual feature is input to the first character recognition sub-model and the second character recognition sub-model respectively. The first character recognition sub-model determines a plurality of candidate recognition results based on the visual feature, and inputs the plurality of candidate recognition results to the second character recognition sub-model. The second character recognition sub-model determines the character recognition result of the target image based on the input visual feature and the plurality of candidate recognition results.

[0113] In some embodiments, the first character recognition sub-model is a parallel recognition sub-model, and the second character recognition sub-model is an autoregressive character recognition sub-model. Optionally, the character recognition model is used to recognize images with character content belonging to a first domain. If it is desired to use the character recognition model to recognize images of a second domain, it is necessary to train the character recognition model based on sample data of the second domain.

[0114] The training method is as follows: the computer device obtains sample data of a second domain, the sample data including sample character data and a sample image including the sample character data, the character content of the sample character data belonging to the second domain, the first domain being different from the second domain; the character recognition model is used to recognize the sample image to obtain predicted character data of the sample image; the autoregressive character recognition sub-model in the character recognition model is trained based on the difference between the predicted character data and the sample character data to obtain a character recognition model applicable to the second domain.

[0115] Since the feature extraction sub-model and the first character recognition sub-model are only related to visual features, and the second character recognition sub-model is not only related to visual features but also the same as semantic features, when the character recognition model is applied to the second domain, it is not necessary to learn visual features again, but only to learn semantic features of the second domain. Therefore, only the autoregressive character recognition sub-model needs to be trained.

[0116] In addition, since it is not necessary to learn visual features again when the character recognition model is applied to the second domain, when obtaining sample data of the second domain, only character data with character content belonging to the second domain needs to be obtained, and a sample image is synthesized based on the character data, without the need to search for images of the second domain to obtain sample images.

[0117] Optionally, the computer device obtains sample data of a second domain, including: obtaining character data belonging to the second domain as sample character data; synthesizing a sample image containing the sample character data corresponding to the sample character data based on the sample character data; and taking the sample character data and the sample image as sample data.

[0118] Wherein, based on the sample character data, when synthesizing the sample image corresponding to the sample character data, the synthesis can be performed according to a fixed picture format, without paying attention to the image style of the sample image.

[0119] It should be noted that the present application has carried out experiments in the data sets of two fields of goods and drugs, and the experimental results are shown in Tables 1 and 2.

[0120]

[0121] Table 1 Comparison of error rates

[0122] According to Table 1, the character recognition model provided by the embodiment of the present application has a greater improvement in recognition accuracy compared with the parallel character recognition model, and the accuracy is slightly higher than that of the autoregressive character recognition model. In addition, the embodiment of the present application also provides a method for adjusting the character recognition model based on sample data of a certain field. According to Table 1, the accuracy of the adjusted model has been further improved.

[0123]

[0124] Table 2

[0125] According to Table 2, the character recognition model provided by the embodiment of the present application has a much faster recognition speed than the autoregressive character recognition model.

[0126] As can be seen, the character recognition model provided by the embodiment of the present application has a much higher accuracy than the parallel character recognition model in the related art, and has a much faster recognition speed than the autoregressive character recognition model in the related art.

[0127] The image-based character recognition method provided by the embodiment of the present application takes into account that individual character recognition errors can lead to incoherent sentences, that is, the characters with recognition errors and other characters have no semantic correlation. Therefore, the embodiment of the present application first determines a plurality of candidate recognition results through visual features, and then determines a character recognition result from the plurality of candidate recognition results based on semantic correlation features of a plurality of characters in the candidate recognition results, so as to ensure that the semantics of the plurality of characters in the character recognition result are relevant, reduce the case of individual character recognition errors, and improve the accuracy of the character recognition result.

[0128] In addition, the character recognition model is provided in the embodiment of the present application to complete the image-based character recognition method. The character recognition model comprises a feature extraction sub-model, a parallel character recognition sub-model and a self-recurrent character recognition sub-model. In this way, multiple candidate recognition results can be quickly obtained through the parallel character recognition sub-model, and a more accurate recognition result can be directly selected from the multiple candidate recognition results as the character recognition result based on the self-recurrent character recognition sub-model. On the basis of ensuring the accuracy of the character recognition result, the recognition speed is improved.

[0129] In addition, since the feature extraction sub-model and the first character recognition sub-model are only related to visual features, and the second character recognition sub-model is not only related to visual features but also related to semantic features, when the character recognition model is put into the second field, it is not necessary to learn the visual features again, but only needs to learn the semantic features of the second field. Therefore, only the self-recurrent character recognition sub-model needs to be trained, the training efficiency of the model is improved, and the training difficulty of the model is reduced.

[0130] In addition, since the visual features do not need to be learned again when the character recognition model is put into the second field, when the sample data of the second field is acquired, only the character data of the character content belonging to the second field needs to be acquired, and the sample image can be synthesized based on the character data. Therefore, it is not necessary to search for the image of the second field to obtain the sample image, and the difficulty of acquiring the sample data is reduced.

[0131] Figure 6 It is a structure schematic diagram of an image-based character recognition device provided by the embodiment of the present application, referring to Figure 6 The device comprises:

[0132] The feature extraction module 601 is configured to perform feature extraction on the target image to obtain visual features.

[0133] The recognition module 602 is configured to recognize the visual features to obtain multiple candidate recognition results corresponding to the target image, wherein the candidate recognition results comprise multiple recognized characters.

[0134] The feature determination module 603 is configured to determine semantic correlation features of the multiple characters in the multiple candidate recognition results.

[0135] The result determination module 604 is configured to determine a character recognition result of the target image from the multiple candidate recognition results based on the semantic correlation features of the multiple characters in the multiple candidate recognition results.

[0136] As shown in Figure 7 In a possible implementation manner, the result determination module 604 comprises:

[0137] The parameter value determination unit 6041 is configured to determine a first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results, where the first recognition parameter value is used to represent a semantic correlation degree of the plurality of characters in the candidate recognition result.

[0138] The result determination unit 6042 is configured to determine the character recognition result of the target image from the plurality of candidate recognition results based on the first recognition parameter value of the plurality of candidate recognition results.

[0139] In a possible implementation, the visual features include visual sub-features corresponding to the plurality of characters; and the parameter value determination unit 6041 is configured to determine the first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results and the visual sub-features corresponding to the plurality of characters.

[0140] In a possible implementation, the parameter value determination unit 6041 is configured to determine, for a first character in the candidate recognition result, a first recognition sub-parameter value corresponding to the first character based on a visual sub-feature corresponding to the first character; determine, for a (k+1)th character in the candidate recognition result, a first recognition sub-parameter value corresponding to the (k+1)th character based on semantic correlation features of the first k characters and the (k+1)th character and a visual sub-feature corresponding to the (k+1)th character, where k≥1 and k is a positive integer; and determine the first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result.

[0141] In a possible implementation, the result determination unit 6042 is configured to obtain a second recognition parameter value of the plurality of candidate recognition results, where the second recognition parameter value is used to represent a matching degree of the candidate recognition result and the visual features; determine a total recognition parameter value of the plurality of candidate recognition results based on the first recognition parameter value and the second recognition parameter value of the plurality of candidate recognition results; and determine a candidate recognition result with the highest total recognition parameter value as the character recognition result.

[0142] In a possible implementation, the second recognition parameter value is determined in a process of recognizing the visual features to obtain a plurality of candidate recognition results corresponding to the target image; and the recognition module 602 is configured to recognize the visual features to obtain the plurality of candidate recognition results corresponding to the target image and the second recognition parameter value of the plurality of candidate recognition results; and select a candidate recognition result that meets a recognition parameter value condition from the plurality of candidate recognition results based on the second recognition parameter value of the plurality of candidate recognition results.

[0143] In a possible implementation, the visual features of the target image include visual sub-features corresponding to the plurality of characters; the identification module 602 is configured to perform parallel identification on the visual sub-features corresponding to the plurality of characters, to obtain the plurality of candidate recognition results corresponding to the target image.

[0144] In a possible implementation, the visual features of the target image include visual sub-features corresponding to the plurality of characters; the identification module 602 is configured to, for a visual sub-feature corresponding to any character, perform identification on the visual sub-feature to obtain probabilities of a plurality of first candidate characters corresponding to the visual sub-feature, select a plurality of second candidate characters satisfying a probability requirement condition from the plurality of first candidate characters based on the probabilities of the plurality of first candidate characters corresponding to the visual sub-feature; and after obtaining the plurality of second candidate characters corresponding to the plurality of visual sub-features respectively, combine the plurality of second candidate characters corresponding to the plurality of visual sub-features respectively to obtain the plurality of candidate recognition results.

[0145] In a possible implementation, the method is performed by a character recognition model, the character recognition model includes a feature extraction sub-model, a first character recognition sub-model, and a second character recognition sub-model, the visual features are determined by the feature extraction sub-model, the plurality of candidate recognition results are determined by the first character recognition sub-model, and the semantic correlation features and the character recognition result are determined by the second character recognition sub-model.

[0146] In a possible implementation, the first character recognition sub-model is a parallel recognition sub-model, and the second character recognition sub-model is an autoregressive character recognition sub-model; the character recognition model is configured to recognize images with character content belonging to a first domain; and the apparatus further includes:

[0147] a sample data acquisition module 605 configured to acquire sample data of a second domain, the sample data including sample character data and a sample image including the sample character data, the character content of the sample character data belonging to the second domain, and the first domain and the second domain being different domains;

[0148] a training module 606 configured to identify the sample image by using the character recognition model to obtain predicted character data of the sample image.

[0149] The training module 606 is further configured to train an autoregressive character recognition sub-model in the character recognition model based on a difference between the predicted character data and the sample character data, to obtain a character recognition model applicable to the second domain.

[0150] In a possible implementation, the sample data obtaining module 605 is configured to obtain character data belonging to the second field as sample character data, synthesize a sample image containing the sample character data based on the sample character data, and take the sample character data and the sample image as the sample data.

[0151] It should be noted that the image-based character recognition apparatus provided in the above embodiments is only taken as an example for the division of the above functional modules in recognizing characters. In actual applications, the above functions can be distributed to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above-described functions. In addition, the image-based character recognition apparatus and the image-based character recognition method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0152] In an example embodiment, a computer device is provided, which includes one or more processors and one or more memories having stored therein at least one program code, which is loaded and executed by the one or more processors to implement the image-based character recognition method in the above embodiments.

[0153] Optionally, the computer device is provided as a terminal. Figure 8 A structural block diagram of a terminal 800 provided by an example embodiment of the present application is shown. The terminal 800 can be a smart phone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer or a desktop computer. The terminal 800 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0154] The terminal 800 includes a processor 801 and a memory 802.

[0155] The processor 801 can include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 801 can be implemented in the form of at least one of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 801 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 801 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 801 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0156] The memory 802 can include one or more computer-readable storage media that can be non-transitory. The memory 802 can also include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code for being executed by the processor 801 to implement the image-based character recognition method provided by the method embodiments of the present application.

[0157] In some embodiments, the terminal 800 can also optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802, and the peripheral device interface 803 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 803 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 804, a display screen 805, a camera 806, an audio circuit 807, a positioning component 808, and a power supply 809.

[0158] The peripheral interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802 and the peripheral interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802 and the peripheral interface 803 can be implemented on a separate chip or circuit board, and the present embodiments are not limited to this.

[0159] The radio frequency circuit 804 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 804 can also include NFC (Near Field Communication) related circuit, and the present application is not limited to this.

[0160] The display screen 805 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 is further configured to capture touch signals on or above the surface of the display screen 805. The touch signals can be input to the processor 801 as control signals for processing. In this case, the display screen 805 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 805 can be one, arranged on the front panel of the terminal 800; in other embodiments, the display screen 805 can be at least two, arranged on different surfaces of the terminal 800 or in a folding design; in still other embodiments, the display screen 805 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 800. Even, the display screen 805 can also be arranged in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0161] The camera assembly 806 is configured to capture images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. The front-facing camera is arranged on the front panel of the terminal, and the rear-facing camera is arranged on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 806 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0162] The audio circuit 807 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 801 for processing, or input to the radio frequency circuit 804 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, and arranged at different parts of the terminal 800. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert an electrical signal from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker can be a conventional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert an electrical signal into a sound wave audible to humans, but also can convert an electrical signal into an inaudible sound wave to humans for ranging purposes, etc. In some embodiments, the audio circuit 807 can also include a headphone jack.

[0163] The positioning component 808 is used to position the current geographic location of the terminal 800 to realize navigation or LBS (Location Based Service). The positioning component 808 can be a positioning component based on the GPS (Global Positioning System) of the United States, the Beidou system of China, the GLONASS system of Russia, or the Galileo system of the European Union.

[0164] The power supply 809 is used to supply power to each component in the terminal 800. The power supply 809 can be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power supply 809 includes rechargeable batteries, the rechargeable batteries can support wired charging or wireless charging. The rechargeable batteries can also be used to support fast charging technology.

[0165] In some embodiments, the terminal 800 also includes one or more sensors 810. The one or more sensors 810 include, but are not limited to, an acceleration sensor 811, a gyroscope sensor 812, a pressure sensor 813, a fingerprint sensor 814, an optical sensor 815, and a proximity sensor 816.

[0166] The acceleration sensor 811 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal 800. For example, the acceleration sensor 811 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 811. The acceleration sensor 811 can also be used for game or user motion data collection.

[0167] The gyroscope sensor 812 can detect the body direction and rotation angle of the terminal 800, and can collect 3D motions of the user with respect to the terminal 800 in cooperation with the acceleration sensor 811. The processor 801 can implement the following functions according to the data collected by the gyroscope sensor 812: motion sensing (e.g., changing a UI according to a tilt operation of the user), image stabilization during photographing, game control, and inertial navigation.

[0168] The pressure sensor 813 can be disposed on the side frame of the terminal 800 and / or the lower layer of the display 805. When the pressure sensor 813 is disposed on the side frame of the terminal 800, the grip signal of the user with respect to the terminal 800 can be detected, and the left / right hand recognition or shortcut operation can be performed by the processor 801 according to the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is disposed on the lower layer of the display 805, the operable control on the UI can be controlled by the processor 801 according to the pressure operation of the user with respect to the display 805. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0169] The fingerprint sensor 814 is used to collect the fingerprint of the user, and the identity of the user can be recognized by the processor 801 according to the fingerprint collected by the fingerprint sensor 814, or by the fingerprint sensor 814 according to the collected fingerprint. When the identity of the user is recognized as a trusted identity, the processor 801 authorizes the user to perform a related sensitive operation, which includes unlocking the screen, viewing encrypted information, downloading software, payment, and changing settings, etc. The fingerprint sensor 814 can be disposed on the front, back, or side of the terminal 800. When the physical button or the manufacturer's logo is disposed on the terminal 800, the fingerprint sensor 814 can be integrated with the physical button or the manufacturer's logo.

[0170] The optical sensor 815 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display 805 according to the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display 805 is increased, and when the ambient light intensity is low, the display brightness of the display 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the photographing parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 815.

[0171] The proximity sensor 816, also referred to as a distance sensor, is arranged on the front panel of the terminal 800. The proximity sensor 816 is used to collect the distance between the user and the front of the terminal 800. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the terminal 800 gradually decreases, the display screen 805 is switched from the bright screen state to the screen-off state under the control of the processor 801; when the proximity sensor 816 detects that the distance between the user and the front of the terminal 800 gradually increases, the display screen 805 is switched from the screen-off state to the bright screen state under the control of the processor 801.

[0172] Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the terminal 800, and the terminal 800 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components. Figure 8

[0173] Optionally, the computer device is provided as a server. Figure 9 FIG. 9 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 900 can be quite different in configuration or performance, and can include one or more processors (Central Processing Units, CPUs) 901 and one or more memories 902, wherein the memory 902 stores at least one program code, which is loaded and executed by the processor 901 to implement the method provided by the above-mentioned various method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, and other components for implementing the functions of the device, which are not described here.

[0174] The server 900 is used to execute the steps performed by the server in the above-mentioned method embodiments.

[0175] In an exemplary embodiment, a computer readable storage medium, such as a memory including program code, is also provided, and the above-mentioned program code can be executed by the processor in the computer device to complete the image-based character recognition method in the above-mentioned embodiments. For example, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0176] In an exemplary embodiment, a computer program or computer program product is also provided, and the computer program or computer program product includes computer program code, which, when executed by a computer, causes the computer to implement the image-based character recognition method in the above-mentioned embodiments.

[0177] ​Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0178] The above only describes optional embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image-based character recognition method, characterized by, The method comprises: performing feature extraction on a target image to obtain a visual feature; performing recognition on the visual feature to obtain a plurality of candidate recognition results corresponding to the target image, the candidate recognition results comprising a plurality of recognized characters; determining semantic correlation features of the plurality of characters in the plurality of candidate recognition results; determining a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results; The method comprises: determining a first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results, the first recognition parameter value being used to represent a semantic correlation degree of the plurality of characters in the candidate recognition result; determining a character recognition result of the target image from the plurality of candidate recognition results based on the first recognition parameter value of the plurality of candidate recognition results; the visual feature comprises visual sub-features corresponding to the plurality of characters; the method comprises: determining a first recognition parameter value of the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results and the visual sub-features corresponding to the plurality of characters.

2. The method of claim 1, wherein, The method comprises: for a first character in the candidate recognition result, determining a first recognition sub-parameter value corresponding to the first character based on a visual sub-feature corresponding to the first character; for a k+1th character in the candidate recognition result, determining a first recognition sub-parameter value corresponding to the k+1th character based on semantic correlation features of the first k characters and the k+1th character and a visual sub-feature corresponding to the k+1th character, k≥1 and k is a positive integer; determining a first recognition parameter value corresponding to the candidate recognition result based on the first recognition sub-parameter values corresponding to the plurality of characters in the candidate recognition result.

3. The method of claim 1, wherein, The method comprises: obtaining a second recognition parameter value of the plurality of candidate recognition results, the second recognition parameter value being used to represent a matching degree of the candidate recognition result and the visual feature; determining a total recognition parameter value of the plurality of candidate recognition results based on the first recognition parameter value and the second recognition parameter value of the plurality of candidate recognition results; determining the candidate recognition result with the highest total recognition parameter value as the character recognition result.

4. The method of claim 3, wherein, The second recognition parameter value is determined in the process of performing recognition on the visual feature to obtain the plurality of candidate recognition results corresponding to the target image. The process of identifying the visual features to obtain multiple candidate recognition results corresponding to the target image includes: The visual features are identified to obtain multiple candidate recognition results corresponding to the target image and second recognition parameter values ​​of the multiple candidate recognition results; Based on the second recognition parameter value of the plurality of candidate recognition results, candidate recognition results that meet the recognition parameter value conditions are selected from the plurality of candidate recognition results.

5. The method of claim 1, wherein, The visual features of the target image include visual sub-features corresponding to the multiple characters; the process of recognizing the visual features to obtain multiple candidate recognition results corresponding to the target image includes: Parallel recognition is performed on the visual sub-features corresponding to the multiple characters to obtain multiple candidate recognition results corresponding to the target image.

6. The method of claim 1, wherein, The visual features of the target image include visual sub-features corresponding to the multiple characters; the process of recognizing the visual features to obtain multiple candidate recognition results corresponding to the target image includes: For any visual sub-feature corresponding to a character, the visual sub-feature is identified to obtain the probability of the visual sub-feature corresponding to multiple first candidate characters. Based on the probability of the visual sub-feature corresponding to multiple first candidate characters, multiple second candidate characters that meet the probability requirements are selected from the multiple first candidate characters. After obtaining multiple second candidate characters corresponding to multiple visual sub-features, the multiple second candidate characters corresponding to the multiple visual sub-features are combined to obtain the multiple candidate recognition results.

7. The method of claim 1, wherein, The method is executed by a character recognition model, which includes a feature extraction sub-model, a first character recognition sub-model, and a second character recognition sub-model. The visual features are determined by the feature extraction sub-model, the multiple candidate recognition results are determined by the first character recognition sub-model, and the semantic relevance features and the character recognition results are determined by the second character recognition sub-model.

8. The method of claim 7, wherein, The first character recognition sub-model is a parallel recognition sub-model, and the second character recognition sub-model is an autoregressive character recognition sub-model; The character recognition model is used to recognize images whose character content belongs to a first domain; the method further includes: Obtain sample data in a second domain, the sample data including sample character data and sample images including the sample character data, the character content of the sample character data belonging to the second domain, and the first domain and the second domain being different domains; The character recognition model is used to identify the sample image to obtain the predicted character data of the sample image. Based on the difference between the predicted character data and the sample character data, the autoregressive character recognition sub-model in the character recognition model is trained to obtain a character recognition model applicable to the second domain.

9. The method of claim 8, wherein, The acquisition of sample data in the second domain includes: Obtain character data belonging to the second domain as sample character data; Based on the sample character data, a sample image containing the sample character data is synthesized; The sample character data and the sample image are used as the sample data.

10. An image-based character recognition apparatus, characterized by comprising: The device includes: The feature extraction module is configured to perform feature extraction on the target image to obtain visual features. The recognition module is configured to recognize the visual features to obtain a plurality of candidate recognition results corresponding to the target image, the candidate recognition results including a plurality of recognized characters. The feature determination module is configured to determine semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results. The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results.

11. A computer device, comprising: The result determination module is configured to determine a character recognition result of the target image from the plurality of candidate recognition results based on the semantic correlation features of the plurality of characters in the plurality of candidate recognition results.

12. A computer-readable storage medium, characterized in that, The computer device includes one or more processors and one or more memories, the one or more memories storing at least one program code, the at least one program code being loaded and executed by the one or more processors to implement the operations performed by the image-based character recognition method according to any one of claims 1 to 9. The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the image-based character recognition method according to any one of claims 1 to 9.

Images