Model training method and text recognition method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a model training method and a text recognition method, device, equipment and storage medium. The text recognition method includes: obtaining an image containing text to be recognized; obtaining a first image feature and a second image feature of the image; predicting the probability of occurrence of a plurality of preset radicals in the image according to the first image feature; and outputting the text contained in the image according to the second image feature and the occurrence probability. Since similar characters are an important factor affecting the accuracy of text recognition results, and the differences between similar characters are often reflected in the differences in radicals or components, in this solution, by predicting the radicals contained in the image, with the assistance of the prediction results, a good text recognition result can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a model training method, a text recognition method, apparatus, device, and storage medium. Background Art

[0002] In some application scenarios, there is a need to recognize the text contained in an image. For example, by performing text recognition on the video data collected by a number of cameras deployed on the road, the name of the store on the street can be recognized, providing a basis for updating relevant data in the electronic map.

[0003] However, in the actual process of text recognition, the text recognition result is often incorrect due to interference such as similar-shaped characters and image noise. For example, the following two Chinese characters: "mei" and "mo", the glyphs of these two Chinese characters are very similar, and they are often misrecognized when performing text recognition based on the existing text recognition model. Summary of the Invention

[0004] Embodiments of the present invention provide a model training method, a text recognition method, apparatus, device, and storage medium, which can improve the accuracy of text recognition.

[0005] In a first aspect, an embodiment of the present invention provides a text recognition method, the method including:

[0006] Obtaining an image containing text to be recognized;

[0007] Obtaining a first image feature and a second image feature of the image;

[0008] Predicting the occurrence probability of a plurality of preset radicals in the image according to the first image feature;

[0009] Outputting the text contained in the image according to the second image feature and the occurrence probability.

[0010] In a second aspect, an embodiment of the present invention provides a text recognition apparatus, the apparatus including:

[0011] An obtaining module, configured to obtain an image containing text to be recognized;

[0012] An extraction module, configured to obtain a first image feature and a second image feature of the image;

[0013] A prediction module, configured to predict the occurrence probability of a plurality of preset radicals in the image according to the first image feature;

[0014] An output module, configured to output the text contained in the image according to the second image feature and the occurrence probability.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the text recognition method described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the text recognition method described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present invention provides a model training method, comprising:

[0018] Obtain a sample image and a radical set, wherein the sample image contains text;

[0019] Determine the annotation information corresponding to the sample image, the annotation information includes the text and radicals contained in the sample image, the radicals exist in the radical set;

[0020] Under the supervision of the annotation information, the text recognition model is trained.

[0021] In a sixth aspect, an embodiment of the present invention provides a model training device, the device comprising:

[0022] An acquisition module, used for acquiring a sample image and a radical set, wherein the sample image contains text;

[0023] A labeling module, used for determining labeling information corresponding to the sample image, wherein the labeling information includes characters and radicals contained in the sample image, and the radicals exist in the radical set;

[0024] The training module is used to train the text recognition model under the supervision of the annotation information.

[0025] In the seventh aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the model training method described in the fifth aspect.

[0026] In the eighth aspect, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model training method described in the fifth aspect.

[0027] In a ninth aspect, an embodiment of the present invention provides a text recognition method, comprising:

[0028] receiving a request for invoking a target service, wherein the request includes an image, and the target service is used to recognize text included in the image;

[0029] The resources corresponding to the target service are used to obtain the first image feature and the second image feature of the image, and the occurrence probabilities of a plurality of preset radicals in the image are predicted based on the first image feature. The text contained in the image is output based on the second image feature and the occurrence probabilities.

[0030] The text recognition scheme provided by the embodiment of the present invention can be used to recognize text contained in an image, and in particular, to accurately recognize Chinese characters contained therein. In practical applications, when an image is acquired and the text contained therein needs to be recognized, a pre-trained text recognition model can be used to complete the text recognition task. The working process of the text recognition module is simply as follows: first, feature extraction is performed on the image to obtain a first image feature and a second image feature of the image. Among them, the first image feature can be used to predict the radicals contained in the image, so that it can be known which radicals are contained in the image. The prediction result of the radicals contained in the image plays an auxiliary role in accurately identifying the text contained in the image, so that, based on the second image feature and the prediction result, accurate recognition of the text contained in the image can be completed.

[0031] Since similar characters are an important factor affecting the accuracy of text recognition results, and the differences between similar characters are often reflected in the differences in radicals or components, in this solution, a text recognition model that can distinguish between radicals and components is provided, so that good text recognition results can be obtained based on the text recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flow chart of a model training method provided by an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the composition of a text recognition model provided by an embodiment of the present invention;

[0035] Figure 3A schematic diagram of another text recognition model provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of another text recognition model provided by an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of another text recognition model provided by an embodiment of the present invention;

[0038] Figure 6 A flowchart of a text recognition method provided by an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of a text recognition application scenario provided by an embodiment of the present invention;

[0040] Figure 8 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention;

[0041] Fig. 9 For Figure 8 A schematic diagram of the structure of an electronic device corresponding to the model training device provided in the illustrated embodiment;

[0042] Fig.10 A structural schematic diagram of a text recognition device provided by an embodiment of the present invention;

[0043] Fig.11 For Fig.10 The structure diagram of the electronic device corresponding to the text recognition device provided in the illustrated embodiment is shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0046] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0048] The text recognition method provided in the embodiment of the present invention can be executed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, an intelligent robot, etc., or a server in the cloud. The server can be a physical server including an independent host, or a virtual server, or a cloud server.

[0049] The text recognition method provided by the embodiment of the present invention can be used to recognize text contained in an image. The image can be, for example, any frame image obtained by segmenting a video captured by a camera set up beside a road.

[0050] In order to realize the recognition processing of the text contained in the image, the embodiment of the present invention provides a text recognition model. In summary, the training process of the text recognition model includes:

[0051] Obtain a sample image and a radical set, wherein the sample image contains text;

[0052] Determine the annotation information corresponding to the sample image, the annotation information including the text and radicals contained in the sample image, the radicals being present in the radical set;

[0053] The text recognition model is trained under the supervision of the annotation information.

[0054] In practical applications, a training sample set containing several sample images can be collected, each of which contains text. Based on this, the radical set can be obtained in the following way:

[0055] Determine multiple words included in the training sample set;

[0056] Determine a plurality of radicals corresponding to the plurality of characters;

[0057] Determine the number of occurrences of each of the plurality of radicals;

[0058] A radical set consisting of radicals whose occurrence times meet the set conditions is determined.

[0059] In practical applications, the text contained in each sample image can be manually counted, so as to summarize all the text contained in the training sample set (here, repeated text can be deduplicated or not), and then each character is disassembled according to the radicals given in the dictionary, and the number of occurrences of each radical is counted. Finally, according to the number of occurrences corresponding to each radical, the radical whose number of occurrences meets the set conditions is determined.

[0060] Among them, optionally, the setting condition is, for example: sorting the radicals from high to low according to the number of occurrences, and finally selecting the radicals ranked in the top N positions, where N is a preset value, such as 200.

[0061] Optionally, the setting condition may also be: the number of occurrences is greater than a preset value, and the preset value is, for example, 30 times.

[0062] In the embodiment of the present invention, the annotation information corresponding to the sample image includes the radicals of the characters contained therein, which means that the character recognition model is finally trained to have the ability to distinguish the radicals and radicals of the characters. Since similar characters are an important factor affecting the accuracy of the character recognition results, and the differences between similar characters are often reflected in the differences in radicals or radicals, therefore, in the training process of the character recognition model, by introducing a loss function for predicting radicals, the character recognition model can distinguish different radicals in similar characters, thereby expanding the encoding features of similar characters, so that good character recognition results can be obtained based on the character recognition model.

[0063] The following is a detailed description of the training process and composition structure of the text recognition model.

[0064] Figure 1 A flow chart of a model training method provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0065] 101. Obtain a sample image containing text and a radical set, and determine annotation information corresponding to the sample image, where the annotation information includes the text and radical contained in the sample image, and the radical exists in the radical set.

[0066] 102. Obtain a first image feature and a second image feature of a sample image.

[0067] 103. Classify and predict the first image feature through a radical classification network to determine the occurrence probability of each radical in the radical set in the sample image.

[0068] 104. Determine a first loss function corresponding to the radical classification network according to the radicals and occurrence probabilities annotated in the sample image.

[0069] 105. Input the second image feature and the occurrence probability into a decoding network, so as to output the text contained in the sample image through the decoding network.

[0070] 106. Determine a second loss function of the text recognition model according to the text output by the decoding network and the text annotated in the sample image, and train the text recognition model according to the first loss function and the second loss function.

[0071] In summary, the text recognition model provided by the embodiment of the present invention can be a new text recognition model obtained by improving the traditional text recognition model. The core point of the improvement is: adding a radical classification network for predicting the probability of occurrence of each radical in the radical set in the current sample image.

[0072] From the above steps, it can be seen that the text recognition model provided in the embodiment of the present invention includes a radical classification network and a decoding network. In addition, the text recognition model also includes one or more encoding networks, so as to complete the acquisition of the above-mentioned first image feature and the second image feature through these encoding networks.

[0073] In the following, several optional ways of acquiring the first image feature and the second image feature are exemplified in combination with the composition of several optional text recognition models.

[0074] Figure 2 A schematic diagram of the composition of a text recognition model provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the text recognition model includes:

[0075] The radical classification network includes a first coding network. In addition, the radical classification network may also include Figure 2 The feature transformation layer (Conv) and the fully connected layer (FC) illustrated in FIG. 1 , wherein the feature transformation layer can be implemented as a convolutional layer.

[0076] In addition to the above-mentioned radical classification network, the character recognition model also includes Figure 2 The second encoding network and decoding network shown in FIG.

[0077] Wherein, optionally, the first encoding network and the second encoding network can be implemented as a convolutional neural network (CNN). The decoding network can be implemented as a long short-term memory network (LSTM), or a bidirectional long short-term memory network (Bi-directional LongShort-Term Memory, Bi-LSTM). Of course, this is not limited to this. In practical applications, the decoding network can be a decoding network using an attention mechanism.

[0078] based on Figure 2 The structure of the text recognition model shown in FIG. 1 can be implemented as follows:

[0079] Encoding the sample image through a first encoding network to obtain a first image feature;

[0080] The sample image is encoded through a second encoding network to obtain a second image feature.

[0081] like Figure 2 As shown in , at this time, the sample image containing the words "Sisters Distillery" serves as both the input of the first encoding network and the input of the second encoding network.

[0082] For the radical classification network, after extracting features from the sample image through the first encoding network to obtain the first image features, the first image features are connected to the FC layer after transformation by the Conv layer to output the occurrence probability of each radical in the radical set in the sample image. Based on the radicals pre-annotated on the sample image and the occurrence probability output by the FC layer, the first loss function corresponding to the radical classification network can be obtained.

[0083] Among them, the first loss function can be expressed as loss attr . Its mathematical form can be as follows:

[0084]

[0085] Among them, freq attr Indicates the number of occurrences of the currently predicted i-th radical (the number of occurrences of the radical when forming the radical set), is determined based on the annotated radicals of the current sample image. For the currently predicted ith radical, if it is determined to exist in the current sample image based on the annotated information, then The value is 1, otherwise the value is 0. iIt is the occurrence probability of the i-th radical in the actual predicted output of the radical classification network in the current sample image. Nattr represents the total number of radicals in the radical set, and the i-th radical refers to the i-th radical in the radical set.

[0086] It can be understood that if the current sample image does not contain any radical in the radical set, the value of the above first loss function is 0.

[0087] As Figure 2 shown, after the sample image is encoded by the second encoding network, a second image feature can be obtained. The above-mentioned occurrence probability output by the radical classification network and this second image feature can both be input into the decoding network, so that the decoding network decodes the second image feature based on the guidance of this occurrence probability to predict and output the text contained in the sample image: sister distillery EOS. Here, EOS is a placeholder for the empty position.

[0088] For example, assume that the sample image contains the character "妹". This character contains the following radicals: 女, 未. When the above-mentioned occurrence probability output by the radical classification network indicates that these two radicals are contained in the sample image and does not contain the radical "末", the decoding network will output the character "妹" based on this guidance information and will not erroneously output the character "妺".

[0089] Finally, based on the text contained in the pre-annotated sample image and the text actually output by the decoding network, the second loss function corresponding to this text recognition model can be obtained, denoted as loss batch , and the text recognition model is trained according to the first loss function and the second loss function. To be precise, this second loss function corresponds to the encoding-decoding structure composed of the second encoding network and the decoding network in Figure 2 .

[0090] Finally, the loss function of the entire text recognition model can be expressed as: loss sum = loss batch + λ attr loss attr , where λ attr is the weight of the first loss function, and this weight can be a preset value, such as 5.

[0091] It is worth noting that in practical applications, in order to reduce the amount of calculation and eliminate the interference of redundant information, the sample image can be first subjected to text detection processing to detect the text area in the sample image, that is, the minimum rectangular area containing the text contained in the sample image. After the text area is cut out, it is used as the input of the text recognition model. In addition, after the text area is cut out, the text area can be rescaled (resized) to a fixed size: wxh. Wherein w represents width and h represents height. In general, it is necessary to retain the aspect ratio of the original sample image for scaling. Afterwards, the text area is input into the first encoding network and the second encoding network.

[0092] Taking the second encoding network as an example (the same as the first encoding network), the second encoding network performs feature encoding on the text area to obtain high-level image features, which are called second image features. At this time, after the convolution and pooling layer operations of the second encoding network, the feature size corresponding to the second image feature can be expressed as w'x1, indicating that the image height has been reduced from h to a dimension of 1, and the image width has also been reduced from w to w'.

[0093] Figure 3 A schematic diagram of another text recognition model provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the text recognition model includes Figure 2 In addition to the radical classification network, the second encoding network and the decoding network shown in the figure, it also includes a third encoding network connected to the output end of the second encoding network.

[0094] Optionally, the third encoding network can be implemented as a long short-term memory network (Long Short-Term Memory, referred to as LSTM), or a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, referred to as Bi-LSTM). Of course, this is not limited to this.

[0095] The third encoding network includes multiple hidden layers, such as Figure 3 Multiple Bi-LSTMs illustrated in FIG.

[0096] based on Figure 3 The structure of the text recognition model shown in FIG. 1 can be implemented as follows:

[0097] Encoding the sample image through a first encoding network to obtain a first image feature;

[0098] The sample image is encoded by the second encoding network, and the image features output by the second encoding network are input into the third encoding network to determine that the second image features of the sample image are a feature sequence composed of image features corresponding to multiple moments output by the third encoding network.

[0099] Based on the above assumptions, assume that the second coding network outputs image features of size w'x1. This feature has no temporal sequence, and each feature corresponds to a region in the input text region. Because text is usually a sequence, the features extracted by the second coding network (CNN) are used to extract higher-level sequence features through the third coding network (Bi-LSTM), and each feature of the text region is temporally associated to obtain higher-level features containing temporal information. The feature sequence output by the third coding network is the second image feature, denoted as c = {c1, c2, ..., c |w’|}.

[0100] For the processing process after obtaining the first image feature and the second image feature in the above manner, please refer to the above description and will not be repeated here.

[0101] Figure 4 A schematic diagram of another text recognition model provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the text recognition model includes:

[0102] Radical classification network, second encoding network and decoding network.

[0103] Among them, Figure 4 As shown in , in this embodiment, the radical classification network may no longer include the first encoding network mentioned above, but only include Figure 2 The feature transformation layer (Conv) and the fully connected layer (FC) shown in FIG. At this time, it is equivalent to that the radical classification network shares the second encoding network.

[0104] based on Figure 4 The structure of the text recognition model shown in FIG. 1 can be implemented as follows:

[0105] Encoding the sample image through a second encoding network;

[0106] The image features output by the second encoding network are determined as the first image features and the second image features.

[0107] That is to say, the image features output by the second encoding network are used as the input of the decoding network on the one hand, and as the input of the Conv layer on the other hand.

[0108] For the processing process after obtaining the first image feature and the second image feature in the above manner, please refer to the above description and will not be repeated here.

[0109] Figure 5 A schematic diagram of another text recognition model provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the text recognition model contains Figure 4 In addition to the radical classification network, the second encoding network and the decoding network shown in the figure, a third encoding network is also included.

[0110] Optionally, the third encoding network can be implemented as a long short-term memory network (Long Short-Term Memory, referred to as LSTM), or a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, referred to as Bi-LSTM). Of course, this is not limited to this.

[0111] The third encoding network includes multiple hidden layers, such as Figure 5 Multiple Bi-LSTMs illustrated in FIG.

[0112] based on Figure 5 The structure of the text recognition model shown in FIG. 1 can be implemented as follows:

[0113] Inputting the image feature output by the second encoding network into the third encoding network, determining that the second image feature is a feature sequence composed of image features corresponding to a plurality of moments output by the third encoding network;

[0114] An image feature corresponding to a target moment output by the third encoding network is determined as a first image feature, wherein the target moment includes a last moment among the multiple moments.

[0115] Based on the example above, assume that the output of the third coding network is c = {c1, c2, ..., c |w’|}, then the feature sequence is used as the second image feature. Optionally, the output c at the last moment can be |w’| As the first image feature input to the Conv layer. |w’| As the global feature identifier of the sample image.

[0116] For the processing process after obtaining the first image feature and the second image feature in the above manner, please refer to the above description and will not be repeated here.

[0117] The above introduces the composition of several text recognition models and the model training process. Under the definition of the above two loss functions, the text recognition model is trained, and finally the text recognition model can distinguish the radicals of characters. For similar characters, when a pair of similar characters are encoded based on the trained text recognition model, the encoded image features will have obvious distinguishability, for example, the distance between the features will be large.

[0118] The text recognition process based on the above text recognition model is explained below in conjunction with the following embodiments.

[0119] Figure 6 A flowchart of a text recognition method provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the method comprises the following steps:

[0120] 601. Obtain an image containing text to be recognized.

[0121] 602. Acquire a first image feature and a second image feature of an image.

[0122] 603. Predict occurrence probabilities of a plurality of preset radicals in the image according to the first image feature.

[0123] 604. Output the text contained in the image according to the second image feature and the occurrence probability.

[0124] In practical applications, combined with the relevant descriptions above, the image may be an original image directly captured by a camera, or may be a text region obtained by text region detection in the original image.

[0125] Based on the composition of the several text recognition models introduced above, step 602 can be implemented in the following ways.

[0126] The first one is when the text recognition model is Figure 2 When the structure shown in FIG. 1 is shown in FIG. 2 , that is, the character recognition model includes: a radical classification network including a first encoding network, a second encoding network and a decoding network, then step 602 can be implemented as follows:

[0127] encoding the image through a first encoding network in a radical classification network to obtain a first image feature;

[0128] The image is encoded by a second encoding network to obtain a second image feature.

[0129] The second type is when the text recognition model is Figure 3When the structure shown in FIG. 1 is shown in FIG. 2 , that is, the character recognition model includes: a radical classification network including a first encoding network, a second encoding network, a third encoding network and a decoding network, then step 602 can be implemented as follows:

[0130] encoding the image through a first encoding network in a radical classification network to obtain a first image feature;

[0131] The image is encoded by a second encoding network, and the image features output by the second encoding network are input into a third encoding network; and it is determined that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network.

[0132] The third type is when the text recognition model is Figure 4 When the structure shown in FIG. 1 is shown in FIG. 2 , that is, the character recognition model includes: a radical classification network that does not include the first encoding network, a second encoding network, and a decoding network. In this case, step 602 can be implemented as follows:

[0133] The image is encoded by a second encoding network, and image features output by the second encoding network are determined as first image features and second image features.

[0134] Fourth, when the text recognition model is Figure 5 When the structure shown in FIG. 1 is shown in FIG. 2 , that is, the character recognition model includes: a radical classification network that does not include the first encoding network, a second encoding network, a third encoding network, and a decoding network. In this case, step 602 can be implemented as follows:

[0135] Encoding the image through a second encoding network, and inputting the image features output by the second encoding network into a third encoding network;

[0136] Determine that the second image feature is a feature sequence composed of image features corresponding to a plurality of moments output by the third encoding network;

[0137] An image feature corresponding to a target moment output by the third encoding network is determined as a first image feature, where the target moment includes a last moment among the multiple moments.

[0138] After obtaining the first image feature and the second image feature of the image by any of the above methods, the first image feature can be classified and predicted by the radical classification network to determine the occurrence probability of multiple radicals (i.e., each radical in the radical set) in the image. The second image feature and the occurrence probability are input into the decoding network to output the text contained in the image through the decoding network.

[0139] The following examples illustrate some application scenarios to which the above text recognition solution can be applied.

[0140] There are often many cameras installed on both sides of the roads in the city. The images collected by these cameras may not only include vehicles, pedestrians, and various road signs on the road, but may also capture the names of shops and residential communities on both sides of the road.

[0141] Based on this, in one application scenario, the names of shops and residential areas can be obtained by performing text recognition processing on the video data captured by each camera. Then, the locations of the shops and residential areas can be obtained through positioning technology. In this way, for example, for applications that provide electronic map navigation services, points of interest (POIs) such as shops and residential areas on the electronic map can be updated based on the above text recognition results.

[0142] In the above-mentioned application scenario, taking the video data collected by any camera as an example, the video data can be segmented into image frames to sample and obtain multiple frames of images, and text recognition processing can be performed on each frame of the image.

[0143] like Figure 7 As shown in , assuming that a frame image is Figure 7 The image X shown in FIG. 1 includes the store "Sisters Shochu House". Figure 7 The text recognition model assumed in Figure 5 In the model structure shown in FIG, after the image X is input into the text recognition model, the second image feature c={c1, c2,…, c |w’|} and the first image feature c |w’| , the first image feature c |w’| Input to the radical classification network to predict the probability of occurrence of each radical in the radical set in the image X. The probability of occurrence is related to the second image feature c = {c1, c2, ..., c |w’|} are concatenated and input into the decoding network to decode the text contained in the output image X: Sisters Distillery.

[0144] Of course, in actual applications, the application scenarios in which the text recognition solution provided in this article can be used are not limited to the above-mentioned example scenarios.

[0145] As mentioned above, the text recognition method provided by the present invention can be executed in the cloud, and a number of computing nodes can be deployed in the cloud, each of which has processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide a certain service, and of course, a computing node can also provide one or more services.

[0146] According to the solution provided by the present invention, the cloud can provide a service for identifying text contained in an image, which is called a target service. When a user needs to use the target service, the target service is called to trigger a request to the cloud to call the target service, and the request carries an image containing text that needs to be identified. The cloud determines a computing node that responds to the request and uses the resources in the computing node to perform the following steps:

[0147] A first image feature and a second image feature of the image are obtained, the occurrence probabilities of a plurality of preset radicals in the image are predicted according to the first image feature, and the text contained in the image is output according to the second image feature and the occurrence probabilities.

[0148] The specific implementation methods of the above steps can refer to the relevant descriptions in other aforementioned embodiments and will not be repeated here.

[0149] It is understandable that the text recognition model mentioned above may be pre-deployed in the computing node that provides the above-mentioned text recognition service, so as to complete the text recognition process by using the text recognition model.

[0150] The following will describe in detail the text recognition device and model training device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.

[0151] Figure 8 A schematic diagram of the structure of a model training device provided by an embodiment of the present invention is shown in FIG. Figure 8 As shown, the device includes: an acquisition module 11, a labeling module 12, and a training module 13.

[0152] The acquisition module 11 is used to acquire a sample image and a radical set, wherein the sample image contains text.

[0153] The annotation module 12 is used to determine annotation information corresponding to the sample image, wherein the annotation information includes characters and radicals contained in the sample image, and the radicals exist in the radical set.

[0154] The training module 13 is used to train the text recognition model under the supervision of the annotation information.

[0155] Optionally, the text recognition model includes a radical classification network, and the training module 13 can be specifically used to: obtain a first image feature of the sample image; perform classification prediction on the first image feature through the radical classification network to determine the occurrence probability of each radical in the radical set in the sample image; and determine a first loss function corresponding to the radical classification network based on the radicals annotated in the sample image and the occurrence probability.

[0156] Optionally, the text recognition model includes a decoding network, and the training module 13 can be specifically used to: obtain a second image feature of the sample image; input the second image feature and the occurrence probability into the decoding network to output the text contained in the sample image through the decoding network; determine a second loss function of the text recognition model based on the text output by the decoding network and the text annotated in the sample image; and train the text recognition model based on the first loss function and the second loss function.

[0157] In an optional embodiment, the radical classification network includes a first encoding network, and the text recognition model includes a second encoding network; in this case, the training module 13 is used to: encode the sample image through the first encoding network to obtain the first image feature; encode the sample image through the second encoding network to obtain the second image feature.

[0158] In another optional embodiment, the radical classification network includes a first encoding network, the text recognition model includes a second encoding network and a third encoding network, and the third encoding network includes multiple hidden layers; in this case, the training module 13 is used to: encode the sample image through the first encoding network to obtain the first image feature; input the image feature output by the second encoding network into the third encoding network, and determine that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network.

[0159] In another optional embodiment, the text recognition model includes a second encoding network; in this case, the training module 13 is used to: encode the sample image through the second encoding network; and determine the image features output by the second encoding network as the first image features and the second image features.

[0160] In another optional embodiment, the text recognition model includes a second encoding network and a third encoding network, and the third encoding network includes multiple hidden layers; in this case, the training module 13 is used to: input the image features output by the second encoding network into the third encoding network; determine that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network; determine the image feature corresponding to a target moment output by the third encoding network as the first image feature, and the target moment includes the last moment among the multiple moments.

[0161] Optionally, the acquisition module 11 can be specifically used to: determine multiple characters included in a training sample set, wherein the sample image is any sample in the training sample set; determine multiple radicals corresponding to the multiple characters; determine the number of occurrences corresponding to each of the multiple radicals; and determine the radical set composed of radicals whose number of occurrences meets set conditions.

[0162] Figure 8 The device shown can execute the model training method provided in the aforementioned embodiments. The detailed execution process and technical effects are described in the aforementioned embodiments and will not be repeated here.

[0163] In one possible design, the above Figure 8 The structure of the model training device shown can be implemented as an electronic device, such as Fig. 9 As shown, the electronic device may include: a first processor 21 and a first memory 22. The first memory 22 stores executable code, and when the executable code is executed by the first processor 21, the first processor 21 can at least implement the model training method provided in the above-mentioned embodiment.

[0164] Optionally, the electronic device may further include a first communication interface 23 for communicating with other devices.

[0165] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the model training method provided in the aforementioned embodiment.

[0166] Fig.10 A structural diagram of a text recognition device provided by an embodiment of the present invention is shown in FIG. Fig.10 As shown, the device includes: an acquisition module 31, an extraction module 32, a prediction module 33, and an output module 34.

[0167] The acquisition module 31 is used to acquire an image containing text to be recognized.

[0168] The extraction module 32 is used to obtain the first image feature and the second image feature of the image.

[0169] The prediction module 33 is used to predict the occurrence probability of a plurality of preset radicals in the image according to the first image feature.

[0170] The output module 34 is used to output the text contained in the image according to the second image feature and the occurrence probability.

[0171] Optionally, the prediction module 33 is specifically used to: perform classification prediction on the first image feature through a radical classification network in a text recognition model to determine the occurrence probability of the multiple radicals in the image.

[0172] Optionally, the output module 34 is specifically used to: input the second image feature and the occurrence probability into a decoding network in a text recognition model, so as to output the text contained in the image through the decoding network.

[0173] In an optional embodiment, the radical classification network includes a first encoding network, and the text recognition model includes a second encoding network; in this case, the extraction module 32 is used to: encode the image through the first encoding network to obtain the first image feature; encode the image through the second encoding network to obtain the second image feature.

[0174] In an optional embodiment, the radical classification network includes a first encoding network, the character recognition model includes a second encoding network and a third encoding network, and the third encoding network includes a plurality of hidden layers; in this case, the extraction module 32 is used to:

[0175] encoding the image through the first encoding network to obtain the first image feature;

[0176] The image feature output by the second encoding network is input to the third encoding network, and it is determined that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network.

[0177] In an optional embodiment, the text recognition model includes a second encoding network; in this case, the extraction module 32 is used to: encode the image through the second encoding network; and determine the image features output by the second encoding network as the first image features and the second image features.

[0178] In an optional embodiment, the text recognition model includes a second encoding network and a third encoding network, and the third encoding network includes multiple hidden layers; in this case, the extraction module 32 is used to: input the image features output by the second encoding network into the third encoding network, and the third encoding network includes multiple hidden layers; determine that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network; determine the image feature corresponding to a target moment output by the third encoding network as the first image feature, and the target moment includes the last moment among the multiple moments.

[0179] Optionally, the first encoding network includes: a convolutional neural network.

[0180] Optionally, the second encoding network includes: a convolutional neural network; the third encoding network includes: a long short-term memory network, or a bidirectional long short-term memory network.

[0181] Fig.10 The device shown can execute the text recognition method provided in the aforementioned embodiments. The detailed execution process and technical effects can be found in the description of the aforementioned embodiments, which will not be repeated here.

[0182] In one possible design, the above Fig.10 The structure of the text recognition device shown can be implemented as an electronic device, such as Fig.11 As shown, the electronic device may include: a second processor 41 and a second memory 42. The second memory 42 stores executable codes, and when the executable codes are executed by the second processor 41, the second processor 41 can at least implement the text recognition method provided in the above embodiments.

[0183] Optionally, the electronic device may further include a second communication interface 43 for communicating with other devices.

[0184] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the text recognition method provided in the aforementioned embodiment.

[0185] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for character recognition, characterized in that: include: Obtain an image containing text to be recognized; Acquire a first image feature and a second image feature of the image; Predicting the occurrence probabilities of a plurality of preset radicals in the image according to the first image feature; outputting the text contained in the image according to the second image feature and the occurrence probability; The method of obtaining the first image feature and the second image feature of the image includes: encoding the image through a second encoding network in a text recognition model, inputting the image feature output by the second encoding network into a third encoding network in the text recognition model, determining the image feature corresponding to a target moment output by the third encoding network as the first image feature, determining that the second image feature is a feature sequence composed of image features corresponding to multiple moments respectively output by the third encoding network, the third encoding network includes multiple hidden layers, and the target moment includes the last moment of the multiple moments.

2. The method according to claim 1, characterized in that The predicting, based on the first image feature, the occurrence probabilities of the preset plurality of radicals in the image includes: The first image feature is classified and predicted by a radical classification network in a character recognition model to determine the occurrence probability of the multiple radicals in the image.

3. The method according to claim 1, characterized in that Outputting the text contained in the image according to the second image feature and the occurrence probability includes: The second image feature and the occurrence probability are input into a decoding network in a text recognition model, so as to output the text contained in the image through the decoding network.

4. The method according to claim 1, characterized in that: The second encoding network includes: a convolutional neural network; the third encoding network includes: a long short-term memory network, or a bidirectional long short-term memory network.

5. A text recognition device, characterized in that: include: An acquisition module, used for acquiring an image containing text to be recognized; An extraction module, used for acquiring a first image feature and a second image feature of the image; A prediction module, used for predicting the occurrence probability of a plurality of preset radicals in the image according to the first image feature; an output module, configured to output the text contained in the image according to the second image feature and the occurrence probability; Among them, the extraction module is used to encode the image through the second encoding network in the text recognition model, input the image features output by the second encoding network into the third encoding network in the text recognition model, determine the image features corresponding to the target moment output by the third encoding network as the first image features, determine that the second image features are feature sequences composed of image features corresponding to multiple moments output by the third encoding network, the third encoding network includes multiple hidden layers, and the target moment includes the last moment of the multiple moments.

6. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the text recognition method as described in any one of claims 1 to 4.

7. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor is caused to execute the text recognition method according to any one of claims 1 to 4.

8. A model training method, characterized in that: The character recognition model includes a second encoding network and a third encoding network, wherein the third encoding network includes a plurality of hidden layers, and the method includes: Obtain a sample image and a radical set, wherein the sample image contains text; Determine the annotation information corresponding to the sample image, the annotation information includes the text and radicals contained in the sample image, the radicals exist in the radical set; Under the supervision of the annotation information, the text recognition model is trained, wherein the training of the text recognition model includes: Acquire a first image feature and a second image feature of the sample image; Performing classification prediction on the first image feature to determine the occurrence probability of each radical in the radical set in the sample image, and determining a first loss function of the text recognition model according to the radicals annotated in the sample image and the occurrence probability; Outputting the text contained in the sample image according to the second image feature and the occurrence probability, and determining a second loss function of the text recognition model according to the output text and the text annotated in the sample image; Training the text recognition model according to the first loss function and the second loss function; Among them, obtaining the first image feature and the second image feature of the sample image includes: encoding the image through the second encoding network, inputting the image feature output by the second encoding network into the third encoding network, determining the image feature corresponding to the target moment output by the third encoding network as the first image feature, and determining that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network, wherein the target moment includes the last moment of the multiple moments.

9. The method according to claim 8, characterized in that The character recognition model includes a radical classification network; The classifying and predicting the first image feature to determine the occurrence probability of each radical in the radical set in the sample image, and determining the first loss function of the text recognition model according to the radicals annotated in the sample image and the occurrence probability, comprises: Performing classification prediction on the first image feature by the radical classification network to determine the occurrence probability of each radical in the radical set in the sample image; A first loss function corresponding to the radical classification network is determined according to the radicals annotated in the sample image and the occurrence probabilities.

10. The method according to claim 9, characterized in that The text recognition model includes a decoding network; Outputting the text contained in the sample image according to the second image feature and the occurrence probability includes: The second image feature and the occurrence probability are input into the decoding network, so as to output the text contained in the sample image through the decoding network.

11. The method according to claim 8, characterized in that The step of obtaining the radical set includes: Determine a plurality of characters included in a training sample set, wherein the sample image is any sample in the training sample set; Determine a plurality of radicals corresponding to the plurality of characters; Determine the number of occurrences of each of the plurality of radicals; The radical set consisting of radicals whose occurrence times meet the set conditions is determined.

12. A model training device, characterized in that: The character recognition model includes a second encoding network and a third encoding network, wherein the third encoding network includes a plurality of hidden layers, and the device includes: An acquisition module, used for acquiring a sample image and a radical set, wherein the sample image contains text; A labeling module, used for determining labeling information corresponding to the sample image, wherein the labeling information includes characters and radicals contained in the sample image, and the radicals exist in the radical set; A training module, used for training a text recognition model under the supervision of the annotation information; The training module is specifically used to obtain the first image feature and the second image feature of the sample image; classify and predict the first image feature to determine the occurrence probability of each radical in the radical set in the sample image, and determine the first loss function corresponding to the text recognition model according to the radicals annotated in the sample image and the occurrence probability; output the text contained in the sample image according to the second image feature and the occurrence probability, and determine the second loss function of the text recognition model according to the output text and the text annotated in the sample image; train the text recognition model according to the first loss function and the second loss function; Among them, in the process of obtaining the first image feature and the second image feature of the sample image, the training module is specifically used to: encode the image through the second encoding network, input the image feature output by the second encoding network into the third encoding network, determine the image feature corresponding to the target moment output by the third encoding network as the first image feature, and determine that the second image feature is a feature sequence composed of image features corresponding to multiple moments output by the third encoding network, and the target moment includes the last moment of the multiple moments.

13. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model training method as described in any one of claims 8 to 11.

14. A non-transitory machine-readable storage medium, characterized in that: The non-temporary machine-readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor executes the model training method as described in any one of claims 8 to 11.

15. A method for character recognition, characterized in that: include: receiving a request for invoking a target service, wherein the request includes an image, and the target service is used to recognize text included in the image; Using the resources corresponding to the target service to obtain a first image feature and a second image feature of the image, predicting the occurrence probabilities of a plurality of preset radicals in the image according to the first image feature, and outputting the text contained in the image according to the second image feature and the occurrence probabilities; The method of obtaining the first image feature and the second image feature of the image includes: encoding the image through a second encoding network in a text recognition model, inputting the image feature output by the second encoding network into a third encoding network in the text recognition model, determining the image feature corresponding to a target moment output by the third encoding network as the first image feature, determining that the second image feature is a feature sequence composed of image features corresponding to multiple moments respectively output by the third encoding network, the third encoding network includes multiple hidden layers, and the target moment includes the last moment of the multiple moments.

Images