Method, apparatus, device and storage medium for recognizing characters in an image

Abstract

According to embodiments of the present disclosure, methods, apparatuses, devices and storage media for recognizing characters in an image are provided. The method includes generating, based on training images, a sequence of predicted features and a sequence of reference features according to a first model and a second model configured to recognize characters in an image, respectively, the second model being based on a different architecture than the first model. The method further includes mapping a predicted feature to a reference feature in the sequence of reference features based on a position of the predicted feature in the sequence of predicted features and a position of the reference feature in the sequence of reference features, the reference feature potentially representing a same character as the predicted feature. The method further includes determining parameters of the first model based on a result of the mapping. In this way, a light-weight but less accurate architecture can learn from a more accurate architecture. In this way, an efficient and accurate character recognition model can be obtained.

Description

TECHNICAL FIELD

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and in particular, to a method, an apparatus, a device and a computer readable storage medium for recognizing characters in an image. BACKGROUND

[0002] Optical Character Recognition (OCR) refers to a process of analyzing an image to obtain textual information in the image. OCR is usually divided into two steps of text detection and text recognition. Text detection refers to detecting the position and range of text in an image and its layout. Text recognition is to recognize the content of text based on text detection. In text recognition, characters in an image need to be recognized to recover the content of text. OCR has a wide range of applications. It is desirable to achieve efficient and accurate character recognition. SUMMARY

[0003] In a first aspect of the present disclosure, a method for recognizing characters in an image is provided. The method comprises: generating, based on training images, a sequence of predicted features according to a first model configured to recognize characters in an image; generating, based on the training images, a sequence of reference features according to a second model configured to recognize characters in an image, the second model being based on a different architecture than the first model; mapping, based on positions of the predicted features in the sequence of predicted features and positions of the reference features in the sequence of reference features, the predicted features to reference features in the sequence of reference features that potentially represent the same characters; and determining parameters of the first model based on a result of the mapping.

[0004] In a second aspect of the present disclosure, an apparatus for recognizing characters in an image is provided. The apparatus comprises: a predicted feature generation module configured to generate, based on training images, a sequence of predicted features according to a first model configured to recognize characters in an image; a reference feature generation module configured to generate, based on the training images, a sequence of reference features according to a second model configured to recognize characters in an image, the second model being based on a different architecture than the first model; a feature mapping module configured to map, based on positions of the predicted features in the sequence of predicted features and positions of the reference features in the sequence of reference features, the predicted features to reference features in the sequence of reference features that potentially represent the same characters; and a parameter determination module configured to determine parameters of the first model based on a result of the mapping.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device comprises at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium has stored thereon a computer program, the computer program being executable by a processor to implement the method of the first aspect.

[0007] It should be understood that all statements herein made regarding the exemplary embodiments of the present disclosure are intended to encompass both structural and functional equivalents thereof. Additionally, it should be understood that although the exemplary embodiments of the present disclosure have been specifically illustrated and described, various modifications and changes can be made without departing from the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, aspects and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings:

[0009] FIG. 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] FIG. 2 A schematic diagram showing a model training architecture according to some embodiments of the present disclosure is shown;

[0011] FIG. 3A A schematic diagram showing a first model according to some embodiments of the present disclosure is shown;

[0012] FIG. 3B A schematic diagram showing a second model according to some embodiments of the present disclosure is shown;

[0013] FIG. 4 A schematic diagram showing one example of a mapping relationship between a predicted feature sequence and a reference feature sequence according to some embodiments of the present disclosure is shown;

[0014] FIG. 5A A schematic diagram showing another example of a mapping relationship between a predicted feature sequence and a reference feature sequence according to some embodiments of the present disclosure is shown;

[0015] FIG. 5B A schematic diagram showing yet another example of a mapping relationship between a predicted feature sequence and a reference feature sequence according to some embodiments of the present disclosure is shown;

[0016] FIG. 5C A schematic diagram showing yet another example of a mapping relationship between a predicted feature sequence and a reference feature sequence according to some embodiments of the present disclosure is shown;

[0017] FIG. 6 A flowchart showing a process for recognizing characters in an image according to some embodiments of the present disclosure is shown;

[0018] FIG. 7A block diagram showing an apparatus for recognizing characters in an image according to some embodiments of the present disclosure is shown; and

[0019] FIG. 8 A block diagram showing an apparatus capable of implementing a number of embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted in a limited sense as set forth in the embodiments set forth herein, rather, these embodiments are provided so as to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of embodiments of the present disclosure, the term "comprising" and its conjugations should be understood to encompass the meanings of "consisting of" and "consisting essentially of", i.e., "comprising but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions can also be included below.

[0022] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that the corresponding output can be generated for a given input after the training is completed. The generation of the model can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes input and provides a corresponding output by using multiple layers of processing units. In this document, "model" can also be referred to as "machine learning model", "machine learning network" or "network", which are used interchangeably herein. A model can further include different types of processing units or networks. As used herein, the term "determining parameters of a model" or similar expressions refers to determining the values of the parameters of the model (also referred to as parameter values), including specific values, sets of values or ranges of values, etc. As used herein, "unit" can be composed of any suitable structure of machine learning model or network.

[0023] As used herein, the term "character" refers to a class of units or symbols, including but not limited to letters, words, numbers, operator symbols, punctuation marks and other symbols. For example, a character can be a Chinese character, an English letter, an Arabic numeral, a punctuation mark, a graphical symbol or a control symbol, etc.

[0024] Generally, machine learning can include three stages, i.e., a training stage, a testing stage, and an application stage (also referred to as an inference stage). In the training stage, a given model can be trained using a large amount of training data, iteratively updating the parameter values of the model until the model is able to obtain consistent inferences from the training data that satisfy an expected target. Through training, the model can be considered to have learned the association (also referred to as the mapping) between the input and the output from the training data. The parameter values of the trained model are determined. In the testing stage, test inputs are applied to the trained model to determine whether the model is able to provide correct outputs, thereby determining the performance of the model. In the application stage, the model can be used to process actual inputs based on the parameter values obtained through training to determine the corresponding outputs.

[0025] As briefly mentioned above, in OCR, it is desirable to achieve efficient and accurate character recognition. Currently, there are two main types of character recognition models, i.e., a connection time classification (CTC) model and an attention model. The CTC model includes, for example, a convolutional recurrent neural network (CRNN). The inference time of the CTC model is short, but the accuracy is not high enough. The attention model includes, for example, a Transformer. The attention model mostly utilizes autoregressive attention decoding, which usually has higher accuracy than the CTC model, but the autoregressive attention decoding involves cyclic calculation, resulting in a longer inference time than the CTC model.

[0026] On the other hand, distillation learning has been applied to the training of character recognition models. For example, to improve the accuracy of a lightweight CTC model, one possible way is to use a larger CTC model as a teacher to distill the lightweight CTC model. However, due to the accuracy of the CTC model itself, the accuracy improvement achieved by distilling between two CTC models is limited.

[0027] Embodiments of the present disclosure propose a scheme for recognizing characters in an image. According to various embodiments of the present disclosure, in training, based on a training image, a predicted feature sequence and a reference feature sequence are respectively generated according to a student model and a teacher model. The student model and the teacher model are based on different architectures. Based on the positions of the predicted features in the predicted feature sequence and the positions of the reference features in the reference feature sequence, a mapping relationship between the predicted features and the reference features is determined. The predicted features and the reference features that are mapped together potentially represent the same character in the training image. The parameters of the learning model are determined according to the mapping result.

[0028] In embodiments of the present disclosure, distillation learning can be performed between character recognition models based on different architectures. In this way, an originally lightweight but not high-accuracy architecture can learn from a high-accuracy architecture. In this way, an efficient and accurate character recognition model can be obtained. In particular, in the case where the learning model is based on a CTC architecture, a lightweight and high-accuracy CTC model can be obtained.

[0029] Example Environment

[0030] FIG. 1 A schematic diagram showing an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the environment 100, a first model 110 and a second model 120 are deployed in a computing device 150. The first model 110 and the second model 120 are configured to recognize characters in images. The second model 120 can be already trained. In the process of training the first model 110 using distillation learning, the parameters of the second model 120 can remain unchanged. In this document, the first model 110 is also referred to as a “student model”, and the second model 120 is also referred to as a “teacher model”.

[0031] The first model 110 and the second model 120 are based on different architectures. The first model 110 and the second model 120 can be based on any suitable architecture, including any existing and future developed architecture. In particular, the second model 120 can be based on a high-accuracy architecture, while the first model 110 can be based on a lightweight but not high-accuracy architecture. In some embodiments, the first model 110 can be based on a CTC architecture, while the second model 120 can be based on an attention architecture. In such embodiments, a lightweight and high-accuracy CTC model can be obtained.

[0032] In training, the computing device 150 trains the first model 110 using a plurality of training images 101-1, 101-2, … (which are also collectively or individually referred to as training images 101) and corresponding labels 102-1, 102-2, … (which are also collectively or individually referred to as labels 102), thereby determining the parameters of the first model 110. The training images 101 can be segmented from larger images by a text detection process. The labels 102 indicate the true characters in the corresponding training images 101. It should be understood that although FIG. 1 Only two training images 101-1 and 101-2 and two labels 102-1 and 102-2 are shown in the figure, but this is merely illustrative. The computing device 150 can utilize N training images and corresponding N labels, where N is an integer greater than or equal to 1.

[0033] In the environment 100, the computing device 150 can be any type of computing device having computing capability, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a game device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system / server such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0034] It should be appreciated that the structure and functionality of the environment 100 are described for illustrative purposes only and are not intended to imply any limitation on the scope of the present disclosure.

[0035] Overall Architecture for Model Training

[0036] FIG. 2 A schematic diagram of a model training architecture is shown in accordance with some embodiments of the present disclosure. FIG. 2 An example of a training image 101 is shown. It should be appreciated that FIG. 2 The training image 101 and the text therein shown in FIG. 1 are illustrative only and are not intended to limit the scope of the present disclosure. In embodiments of the present disclosure, the training image 101 can include any number and any type of characters.

[0037] In FIG. 2 In the left branch of the architecture 100, the first model 110 generates a predicted feature sequence 210 based on the training image 101. The predicted features in the predicted feature sequence 210 are high-level image features, and the position of a predicted feature in the predicted feature sequence 210 depends on the position of the element (e.g., a character in the image, a blank, a background, etc.) that the predicted feature represents in the training image 101. Each predicted feature can be represented by a feature vector.

[0038] It is assumed that the size of the training image 101 is (H, W), where H represents the height of the training image 101 and W represents the width of the training image 101. The size of the predicted feature sequence 210 is (N, D), where N is the number of predicted features in the predicted feature sequence 210. N = H / Rh*W / Rw, where Rh, Rw are the down-sampling multiples of the first model 110 in the width and height dimensions, respectively. For example, Rh can be H and Rw can be 4. In the example of the architecture 100, N is 16. D is the number of feature channels (such as 512), i.e., D is the dimension of the feature vector. FIG. 2 ​

[0039] It should be appreciated that the various numbers and values shown above are merely exemplary, and embodiments of the present disclosure can be used for any suitable sampling factor, and the predicted features can have any suitable number and dimension. In the case of one-shot input of B training images (e.g., in batch training), the first model 110 can generate B sequences of predicted features of size (N, D), i.e., a feature map of overall size (B, N, D). FIG. 2 It should be appreciated that the various numbers and values shown above are merely exemplary, and embodiments of the present disclosure can be used for any suitable sampling factor, and the predicted features can have any suitable number and dimension. In the case of one-shot input of B training images (e.g., in batch training), the first model 110 can generate B sequences of predicted features of size (N, D), i.e., a feature map of overall size (B, N, D).

[0040] FIG. 3A An example of the first model 110 according to some embodiments of the present disclosure is shown. In the example of FIG. 3A The first model 110 includes a convolutional neural network (CNN) 310 and an encoder 320. The training images 101 are preliminarily processed by the CNN 310 to generate convolutional feature maps with semantic features. The convolutional feature maps are further processed by the encoder 320 to obtain the predicted feature sequence 210 of size (N, D). In some embodiments, the first model 110 is based on a CTC architecture. It should be appreciated that the first model 110 can be based on any suitable architecture. FIG. 3A The structure of the first model 110 in

[0041] With reference back to FIG. 2 The predicted feature sequence 210 is input to a post-processing unit 231. The post-processing unit 231 generates recognition results 240 of the characters in the training images 101 based on the predicted feature sequence 210. The symbol “-” in the recognition results 240 represents an interval between different characters, and is also referred to as an “interval symbol”. The post-processing unit 231 can have any suitable network, and the scope of the present disclosure is not limited in this respect. By way of example, the post-processing unit 231 can include a fully connected layer and an argmax function. It can be appreciated that the post-processing unit 231 can be trained together with the first model 101. The post-processing unit 231 and the first model 101 together constitute a character recognition model used in the inference stage.

[0042] In the right branch of FIG. 2 The second model 120 generates a reference feature sequence 220 based on the training images 101. The reference features in the reference feature sequence 220 are high-level image features, and the position of a reference feature in the reference feature sequence 210 depends on the position of the character represented by the reference feature in the training image 101. Each reference feature can be represented by a feature vector. In embodiments of the present disclosure, unless explicitly stated otherwise, a reference feature in a reference feature sequence refers to those features that represent a character in an image.

[0043] The size of the reference feature sequence 220 is (T, D), where T is the target length. In some embodiments, T can equal the length of the real characters in the training image 101 plus 1, for example in the case of the second model 120 being based on an attention architecture, as it is common to include an end-of-sequence token <eos>In FIG. 2 , T is shown as 5 by way of example only and without intending to limit the scope of the present disclosure. D is the number of feature channels (such as 512), i.e. the dimension of the feature vector. In the case of one-shot input of B training images (e.g. in batch training), the second model 120 can generate B reference feature sequences of size (T, D), i.e. a feature map of overall size (B, T, D).

[0044] FIG. 3B An example of the second model 120 according to some embodiments of the present disclosure is shown. In FIG. 3B , the second model 120 comprises a CNN 330, an encoder 340 and a decoder 350. The training image 101 is processed by the CNN 330, the encoder 340 and the decoder 350 in turn to obtain the reference feature sequence 220 of size (T, D).

[0045] In some embodiments, the second model 120 is based on an attention architecture. For example, the encoder 340 can be a Transformer encoder, while the decoder 350 can be a Transformer decoder. In the case that the decoder 350 is based on autoregressive decoding, the decoder 350 further comprises a loop structure, which is not shown.

[0046] It should be understood that FIG. 3B the structure of the second model 120 in is merely exemplary and without intending to limit the scope of the present disclosure. In embodiments of the present disclosure, the second model 120 can be implemented with any suitable architecture different from the first model 110.

[0047] FIG. 2 With reference back to , the reference feature sequence 220 is input to a post-processing unit 232. The post-processing unit 232 generates the recognition result 260 of the character in the training image 101 based on the reference feature sequence 220. The post-processing unit 232 can have any suitable network, the scope of the present disclosure is not limited in this respect. By way of example, the post-processing unit 232 can comprise a fully connected layer and an argmax function. The post-processing unit 232 has been trained together with the second model 120. In the training of the first model 110, the parameters of the post-processing unit 232 remain unchanged.

[0048] Feature Mapping

[0049] A model for recognizing characters in an image typically comprises a backbone, a neck and a head. In conventional distillation learning, the heads of the student model and the teacher model are of the same type, e.g. based on CTC. In the training, the parameters of the teacher model are fixed, supervising the teacher model and the student model in the features of at least one of the backbone, the head.

[0050] If the teacher model and the student model are based on different architectures, for example, the teacher model is based on an attention architecture and the student model is based on a CTC architecture, there are some problems. Since the shapes of the features output by different architectures are usually different, the feature supervision between the teacher model and the student model can only be done in the backbone part. However, the feature spaces of the two models in the backbone part are not completely consistent. Therefore, the effect achieved by only doing feature supervision in the backbone part is very limited.

[0051] In embodiments of the present disclosure, the supervision of the head features is done between the first model 110 and the second model 120 based on different architectures. As mentioned above, the position of a prediction feature in the prediction feature sequence 210 depends on the position of the element represented by the prediction feature in the training image 101, and the position of a reference feature in the reference feature sequence 220 depends on the position of the character represented by the reference feature in the training image 101. In view of this, based on the position of the prediction feature in the prediction feature sequence 210 and the position of the reference feature in the reference feature sequence 220, the prediction feature and the reference feature that potentially represent the same character can be mapped together or corresponded.

[0052] Depending on the specific implementation of the prediction feature sequence and the reference feature sequence, different strategies can be employed to map the prediction features to the reference features that potentially represent the same character. In some embodiments, the prediction feature sequence can include character features representing the characters in the training image 101 and interval features representing the intervals between the characters. In such embodiments, the character features can be mapped to the reference features in the reference feature sequence based on the relative positions of the character features and the interval features in the prediction feature sequence, and the positions of the reference features in the reference feature sequence. In the case where the first model 110 is based on a CTC architecture, such a prediction feature sequence can be generated.

[0053] FIG. 2 Such an example is shown. In FIG. 2 the white-filled rectangles indicate the interval features, which correspond to the interval symbols "-" in the recognition result 240, respectively. FIG. 4 The mapping relationship between the prediction feature sequence 210 and the reference feature sequence 220 in FIG. 2 is shown. As FIG. 4 shown, the reference feature sequence 220 includes reference features 421-1, 421-2, 421-3, 421-4, 421-5, which are collectively referred to as or individually referred to as reference features 421. Each reference feature 421 represents a character in the image 101. For example, the reference feature 421-1 represents the character "R", the reference feature 421-5 represents the character "m", and so on.

[0054] The sequence of prediction features 210 includes character features 411-1, 411-2, 411-3, 411-4, 411-5, 411-6 representing characters in the training image 101, which are also collectively or individually referred to as character features 411. In addition to the character features 411, the sequence of prediction features 210 also includes interval features representing character intervals.

[0055] As the first character feature in the sequence of prediction features 210, the character feature 411-1 is mapped to the first reference feature 421-1 in the sequence of reference features 220. The character feature 411-2 follows the character feature 411-1 and there is an interval feature between the character feature 411-1 and the character feature 411-2. Thus, the character feature 411-2 can be considered to represent a different character in the training image 101 than the character feature 411-1. Accordingly, the character feature 411-2 is mapped to the reference feature 421-2, which is located after the reference feature 421-1.

[0056] The character feature 411-3 follows the character feature 411-2 and there is an interval feature between the character feature 411-2 and the character feature 411-3. Thus, the character feature 411-3 can be considered to represent a different character in the training image 101 than the character feature 411-2. Accordingly, the character feature 411-3 is mapped to the reference feature 421-3, which is located after the reference feature 421-2.

[0057] The character feature 411-4 follows the character feature 411-3 and there is no interval feature between the character feature 411-3 and the character feature 411-4. Thus, the character feature 411-4 can be considered to represent the same character in the training image 101 as the character feature 411-3. Accordingly, the character feature 411-4 is also mapped to the reference feature 421-3. That is, prediction features representing the same character are mapped to the same reference feature.

[0058] The character feature 411-5 follows the character feature 411-4 and there is an interval feature between the character feature 411-4 and the character feature 411-5. Thus, the character feature 411-5 can be considered to represent a different character in the training image 101 than the character feature 411-4. Accordingly, the character feature 411-5 is mapped to the reference feature 421-4, which is located after the reference feature 421-3.

[0059] The character feature 411-6 follows the character feature 411-5 and there are three interval features between the character feature 411-5 and the character feature 411-6. Thus, the character feature 411-6 can be considered to represent a different character in the training image 101 than the character feature 411-5. Accordingly, the character feature 411-6 is mapped to the reference feature 421-5, which is located after the reference feature 421-4.

[0060] In FIG. 4 In the example of the prediction feature sequence 210, each of the prediction features (i.e., character features) in the prediction feature sequence 210 representing a character in the training image 101 is mapped to a corresponding reference feature, and each of the reference features in the reference feature sequence 220 is mapped with a prediction feature. Such a mapping result is also referred to as a "complete mapping".

[0061] Since the first model 110 is in training, it is inevitable that there are cases of missed recognition and multiple recognition, and thus there can be cases of "incomplete mapping". FIG. 5A Another example of mapping relationship between a prediction feature sequence and a reference feature sequence is shown. The reference feature sequence 520 includes reference features 521-1, 521-2, 521-3, 521-4, 521-5. The prediction feature sequence 510 includes character features 511-1, 511-2, 511-3, 511-4, 511-5.

[0062] According to the position-based strategy described above with reference to FIG. 4 The character feature 511-1 is mapped to the reference feature 521-1, the character feature 511-2 is mapped to the reference feature 521-2, the character features 511-3 and 511-4 are mapped to the reference feature 521-3, and the character feature 511-5 is mapped to the reference feature 521-4. There is a reference feature 521-5 in the reference feature sequence 520 that is not mapped with any prediction feature. Such a mapping result is an incomplete mapping.

[0063] FIG. 5B Yet another example of mapping relationship between a prediction feature sequence and a reference feature sequence is shown. The reference feature sequence 540 includes reference features 541-1, 541-2, 541-3, 541-4. The prediction feature sequence 530 includes character features 531-1, 531-2, 531-3, 531-4, 531-5, 531-6.

[0064] According to the position-based strategy described above with reference to FIG. 4 The character feature 531-1 is mapped to the reference feature 541-1, the character feature 531-2 is mapped to the reference feature 541-2, the character features 531-3 and 531-4 are mapped to the reference feature 541-3, and the character feature 531-5 is mapped to the reference feature 541-4. There is a character feature 531-6 in the prediction feature sequence 540 that is not mapped to any reference feature. Such a mapping result is another incomplete mapping.

[0065] Reference is made to FIG. 4 , FIG. 5A and FIG. 5B Examples of full mapping and incomplete mapping are described. It should be understood that these examples are merely for better illustrating the feature mapping and are not intended to limit the scope of the present disclosure. Appropriate feature mapping strategies can be employed according to the characteristics of the predicted feature sequence and the reference feature sequence. For example, in some embodiments, the reference feature sequence can include reference features representing non-characters. Accordingly, such reference features can be extracted from the reference feature sequence before mapping the predicted features to the reference features by position.

[0066] For another example, in some embodiments, one predicted feature can be mapped to multiple reference features. FIG. 5C One such example is shown. The reference feature sequence 560 includes reference features 561-1, 561-2, 561-3, 561-4, 561-5, 561-6. Each two reference features represent the same character. The predicted features in the predicted feature sequence 550 are mapped to the reference features in the reference feature sequence 560 by position in turn. The reference feature 551-1 is mapped to the predicted features 561-1 and 561-2, the reference feature 551-2 is mapped to the predicted features 561-3 and 561-4, and the reference feature 551-3 is mapped to the predicted features 561-5 and 561-6.

[0067] In summary, based on the position of the predicted features in the predicted feature sequence 210 and the position of the reference features in the reference feature sequence 220, the predicted features can be mapped to the reference features that potentially represent the same character. Specific mapping strategies can be employed according to the characteristics of the predicted feature sequence and the reference feature sequence.

[0068] Loss for Model Training

[0069] The mapping of the predicted features to the reference features is described above. Based on the result of the mapping of the predicted features to the reference features, the parameters of the first model 110 are determined. According to the result of the mapping, the loss function used to iteratively update the parameters of the first model 110 can be determined until the iteration converges or reaches a predetermined number of times, thereby determining the parameters of the first model 110.

[0070] Continuing to refer to FIG. 2 If the mapping between the predicted feature sequence 210 and the reference feature sequence 220 is full mapping, such as the case shown in FIG. 4 a first loss function representing a distillation loss is used. The first loss function is determined based on the difference between each predicted feature and the reference feature it is mapped to. For example, for the case shown in FIG. 4 For example, based on the differences between the predicted features 411-1 and the reference features 421-1, the differences between the predicted features 411-2 and the reference features 421-2, the differences between the predicted features 411-3 and the reference features 421-3, the differences between the predicted features 411-4 and the reference features 421-3, the differences between the predicted features 411-5 and the reference features 421-4, and the differences between the predicted features 411-6 and the reference features 421-5.

[0071] The parameters of the first model 110 can be determined at least by minimizing the first loss function. In addition to the first loss function, a second loss function as described below can also be used.

[0072] In this way, the predicted features generated by the first model 110 are consistent with the reference features. This helps to improve the accuracy of the first model 110. Using the second model 120 with high accuracy, such as a model based on an attention architecture, is equivalent to providing a strong teacher for the learning of the first model 110. In this way, the lightweight first model 110 can achieve high recognition accuracy.

[0073] If the mapping between the predicted feature sequence 210 and the reference feature sequence 220 is an incomplete mapping, such as the case shown in FIG. 5A or FIG. 5B the first loss function is abandoned, but other loss functions are still retained. For example, the second loss function can be determined based on the differences between the recognition result 240 and the real characters in the label 102. The parameters of the first model 110 are determined by minimizing the second loss function. In this embodiment, abandoning the use of the first loss function can avoid inaccurate feature mapping misleading the first model 110.

[0074] Overall, in the case of complete mapping between the predicted feature sequence and the reference feature sequence, at least the first loss function as a distillation loss is used. In the case of incomplete mapping between the predicted feature sequence and the reference feature sequence, the first loss function is abandoned, and at least the second loss function is used. In some embodiments, in addition to the first loss function and the second loss function, the features of the backbone part of the first model 110 and the second model 120 can also be supervised. That is, a third loss function can also be determined based on the differences between the features output by the backbone part of the two models.

[0075] Example Process

[0076] FIG. 6 A flowchart of a process 600 of recognizing characters in an image according to some embodiments of the present disclosure is shown. The process 600 can be implemented at the computing device 150. Reference is made below to FIG. 1 and FIG. 2 The process 600 is described.

[0077] At block 610, the computing device 150 generates, based on the training images 101, a sequence of predicted features 210 from a first model 110 configured to recognize characters in images. At block 620, the computing device 150 generates, based on the training images 101, a sequence of reference features 220 from a second model 120 configured to recognize characters in images. The second model 120 is based on a different architecture than the first model 110. In some embodiments, the first model 110 is based on a CTC architecture and the second model 120 is based on an attention architecture.

[0078] At block 630, the computing device 150 maps the predicted features to reference features in the sequence of reference features 220 that potentially represent the same characters based on the positions of the predicted features in the sequence of predicted features 210 and the positions of the reference features in the sequence of reference features 220. That is, the predicted features and the reference features that are mapped together potentially represent the same character. The mapping strategy can be employed that is consistent with the characteristics of the sequence of reference features and the sequence of predicted features.

[0079] In some embodiments, the sequence of predicted features 210 includes character features representing characters in the training images 101 and interval features representing intervals between the characters, such as FIG. 4 In such embodiments, to map the predicted features to the reference features, the character features can be mapped to reference features in the sequence of reference features 220 based on the relative positions of the character features and the interval features in the sequence of predicted features 210 and the positions of the reference features in the sequence of reference features 220.

[0080] In some embodiments, a first character feature in the sequence of predicted features can be mapped to a first reference feature in the sequence of reference features. If a second character feature after the first character feature does not have an interval feature between the first character feature and the second character feature, the second character feature can be mapped to the first reference feature. For example, the character feature 411-3 is mapped to the reference feature 421-3. The character feature 411-4 is after the character feature 411-3 and does not have an interval feature between the character feature 411-3 and the character feature 411-4. The character feature 411-4 is also mapped to the reference feature 421-3.

[0081] In some embodiments, a third character feature in the predicted feature sequence can be mapped to a second reference feature in the reference feature sequence. If there is at least one interval feature between the fourth character feature after the third character feature and the third character feature, the fourth character feature can be mapped to a third reference feature after the second reference feature. For example, character feature 411-5 is mapped to character feature 421-4. Character feature 411-6 is located after character feature 411-5, and there are three interval features between character feature 411-5 and character feature 411-6. Character feature 411-6 is mapped to reference feature 421-5 which is located after reference feature 421-4.

[0082] At block 640, the computing device 150 determines the parameters of the first model 110 based on the results of the mapping. According to the results of the mapping, the loss function used can be determined to iteratively update the parameters of the first model 110 until the iteration converges or reaches a predetermined number of times, thereby determining the parameters of the first model 110.

[0083] In some embodiments, if all the predicted features in the predicted feature sequence 210 representing the characters in the training image 101 are mapped to the corresponding reference features, and all the reference features in the reference feature sequence 220 are mapped with the predicted features in the predicted feature sequence 210, a first loss function is determined based on the difference between the corresponding predicted features and the mapped reference features. The parameters of the first model 110 are determined at least by minimizing the first loss function. For the case of complete mapping as shown in FIG. 4 The first loss function is used at least for the case of complete mapping as shown in

[0084] In some embodiments, if at least one of the following is determined, the recognition result of the characters in the training image 101 is obtained based on the predicted feature sequence 210: there is a predicted feature in the predicted feature sequence 210 representing the characters in the training image 101 that is not mapped to a reference feature, there is a reference feature in the reference feature sequence 220 that is not mapped with a predicted feature. For example, the recognition result 240 can be generated by the post-processing unit 231. A second loss function can be determined based on the difference between the recognition result and the true characters in the training image. For example, the second loss function can be determined based on the difference between the recognition result 240 and the true characters in the label 102. The parameters of the first model are determined by minimizing the second loss function. For the case of incomplete mapping as shown in FIG. 5A and FIG. 5B The first loss function is abandoned for the case of incomplete mapping as shown in

[0085] Example Devices and Apparatus

[0086] FIG. 7 A schematic structural block diagram of an apparatus 700 for recognizing characters in an image according to certain embodiments of the present disclosure is shown. The apparatus 700 can be implemented as or included in the computing device 150. Various modules / components in the apparatus 700 can be implemented by hardware, software, firmware, or any combination thereof.

[0087] As shown, the apparatus 700 includes a predicted feature generation module 710 configured to generate, based on training images, a predicted feature sequence according to a first model configured to recognize characters in an image. The apparatus 700 also includes a reference feature generation module 720 configured to generate, based on the training images, a reference feature sequence according to a second model configured to recognize characters in an image, the second model being based on a different architecture than the first model. The apparatus 700 further includes a feature mapping module 730 configured to map, based on positions of the predicted features in the predicted feature sequence and positions of the reference features in the reference feature sequence, the predicted features to reference features in the reference feature sequence that potentially represent the same characters. The apparatus 700 further includes a parameter determination module 740 configured to determine, based on results of the mapping, parameters of the first model.

[0088] In some embodiments, the predicted feature sequence includes character features representing characters in the training images and interval features representing character intervals, and the feature mapping module 730 includes a character feature processing module configured to map, based on relative positions of the character features and the interval features in the predicted feature sequence and positions of the reference features in the reference feature sequence, the character features to the reference features in the reference feature sequence.

[0089] In some embodiments, the character feature processing module is further configured to map a first character feature in the predicted feature sequence to a first reference feature in the reference feature sequence, and map a second character feature to the first reference feature if there is no interval feature between the second character feature and the first character feature.

[0090] In some embodiments, the character feature processing module is further configured to map a third character feature in the predicted feature sequence to a second reference feature in the reference feature sequence, and map a fourth character feature to a third reference feature after the second reference feature if there is at least one interval feature between the fourth character feature and the third character feature.

[0091] In some embodiments, the parameter determining module 740 comprises: a first loss determining module configured to determine a first loss function based on a difference between the corresponding predicted feature and the mapped reference feature if the predicted features in the predicted feature sequence representing the character in the training image are all mapped to the corresponding reference feature, and the reference features in the reference feature sequence are all mapped with the predicted features in the predicted feature sequence; and a first minimizing module configured to determine the parameters of the first model by at least minimizing the first loss function.

[0092] In some embodiments, the parameter determining module 740 comprises: an identification result obtaining module configured to obtain an identification result of the character in the training image based on the predicted feature sequence if at least one of the following is determined: there is a predicted feature representing the character in the training image in the predicted feature sequence that is not mapped to a reference feature, and there is a reference feature in the reference feature sequence that is not mapped with a predicted feature; and a second loss determining module configured to determine a second loss function based on a difference between the identification result and the real character in the training image; and a second minimizing module configured to determine the parameters of the first model by minimizing the second loss function.

[0093] In some embodiments, the first model is based on a connection time sequence classification (CTC) architecture, and the second model is based on an attention architecture.

[0094] FIG. 8 A block diagram illustrating a computing device 800 in which one or more embodiments of the disclosure can be implemented is shown. It should be understood that FIG. 8 The computing device 800 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments described herein. FIG. 8 The computing device 800 shown can be used to implement the computing device 150. FIG. 1 of the embodiments described herein.

[0095] As FIG. 8 shown, the computing device 800 is in the form of a general-purpose computing device. Components of the computing device 800 can include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 can be a real or virtual processor and is capable of executing a variety of processing according to programs stored in the memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 800.

[0096] The computing device 800 typically includes a plurality of computer storage media. Such media can be volatile, nonvolatile, removable, and / or non-removable media implemented in any method or technology for storage of information such as program modules, data, and / or data files. Storage 830 can be implemented as a database management system, object-oriented database, or any other suitable data management system. Storage 830 can be a removable or non-removable media, and can include machine-readable media, such as flash drives, magnetic disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible by the computing device 800.

[0097] The computing device 800 can further include additional removable / non-removable, volatile / nonvolatile storage media. Although not shown, a floppy disk drive, a magnetic tape drive, and / or optical disk drive can be provided and connected to bus 810 by a drive interface (not shown). In such instances, each can be connected to bus 810 by one or more data media interfaces. Storage 830 can include a computer program product 825 having one or more program modules configured to carry out the various methods or actions of the embodiments of the present disclosure. FIG. 8

[0098] The communication unit 840 enables communications with other computing devices over a communication media. Additionally, the functionality of the components of the computing device 800 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating over a communication connection. Thus, the computing device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0099] The input device(s) 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. The output device(s) 860 can be one or more output devices, such as a display, speakers, printer, etc. The computing device 800 can also include other peripheral components, such as a modem, a scanner, or a network interface card, etc. The input and output devices are connected to bus 810 via input / output (I / O) interfaces (not shown). The peripheral components and bus interfaces can be implemented as interface circuits, which enable the transfer of information between the components and the peripheral components and / or the bus interfaces.

[0100] ​According to an example implementation of the present disclosure, a computer readable storage medium is provided having computer executable instructions stored thereon, where the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided that is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, where the computer executable instructions are executed by a processor to implement the method described above.

[0101] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0102] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium. The instructions stored on the computer readable storage medium can be used to program a computer, a programmable data processing apparatus, and / or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0103] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0104] The computer program product of the present disclosure can have a signal including said computer program. This signal can be electronic, electromagnetic, optical, or any other suitable type of signal. Such a signal can be provided through a communication connection, such as electrical wiring, optical fiber, wireless interface, etc. Examples of computer program products include computer program implemented on a personal computer, server, or other networked device. A non-transitory computer readable medium, such as a floppy disk, CD-ROM, DVD-ROM, Blu-ray Disc, hard disk drive, or any other suitable non-transitory computer readable medium can store the computer program product.

[0105] Various implementations of the disclosure have been described in detail above. The foregoing description is exemplary and explanatory only, and is not intended to be exhaustive or to limit various implementations of the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings without departing from the scope and spirit of the disclosure. It is intended that the scope of the disclosure be limited only by the claims and the equivalents thereof. The use of the terms "including," "containing," "comprising," "having," "in involving," "portions," "elements," "components," "steps," "phases," "processes," "operations," "steps," "stages," "procedures," "methods," "mechanisms," "devices," "systems," "apparatuses," "units," "means," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "apparatuses," "units," "devices," "systems," "< / eos>

Claims

1. A method for recognizing characters in an image, comprising: Based on the training images, a predicted feature sequence is generated according to a first model configured to recognize characters in the images; Based on the training images, a reference feature sequence is generated according to a second model configured to recognize characters in the images, wherein the second model is based on a different architecture than the first model; Based on the position of the predicted feature in the predicted feature sequence and the position of the reference feature in the reference feature sequence, the predicted feature is mapped to a reference feature in the reference feature sequence that potentially represents the same character; and Based on the mapping result, the parameters of the first model are determined; The predicted feature sequence includes character features representing characters in the training image and interval features representing character intervals, and the reference features that map the character features to the reference feature sequence include: If there is no gap feature between the first character feature in the predicted feature sequence and the second character feature following the first character feature, both the first character feature and the second character feature are mapped to the first reference feature in the reference feature sequence. as well as If there is at least one gap feature between the third character feature in the predicted feature sequence and the fourth character feature following the third character feature, the third character feature is mapped to the second reference feature in the reference feature sequence, and the fourth character feature is mapped to the third reference feature following the second reference feature.

2. The method according to claim 1, wherein determining the parameters of the first model includes: If at least one of the following is determined, the recognition result of the characters in the training image is obtained based on the predicted feature sequence: The predicted feature sequence contains predicted features representing characters in the training image that are not mapped to the reference features. The reference feature sequence contains reference features that have not been mapped to predicted features; as well as Based on the difference between the recognition result and the real characters in the training image, a second loss function is determined; as well as The parameters of the first model are determined by minimizing the second loss function.

3. The method according to claim 1, wherein determining the parameters of the first model includes: If all predicted features representing characters in the training image in the predicted feature sequence are mapped to corresponding reference features, and all reference features in the reference feature sequence are mapped to predicted features in the predicted feature sequence, a first loss function is determined based on the difference between the corresponding predicted features and the mapped reference features; and The parameters of the first model are determined by minimizing the first loss function.

4. The method according to claim 1, wherein the first model is based on the Connection Temporal Classification (CTC) architecture, and the second model is based on the attention architecture.

5. An apparatus for recognizing characters in an image, comprising: The predictive feature generation module is configured to generate a predictive feature sequence based on the training image and a first model configured to recognize characters in the image; A reference feature generation module is configured to generate a reference feature sequence based on the training image and according to a second model configured to recognize characters in the image, wherein the second model is based on a different architecture than the first model. The feature mapping module is configured to map the predicted features to reference features in the reference feature sequence that potentially represent the same character, based on the position of the predicted features in the predicted feature sequence and the position of the reference features in the reference feature sequence. as well as The parameter determination module is configured to determine the parameters of the first model based on the result of the mapping. The predicted feature sequence includes character features representing characters in the training image and interval features representing character intervals, and the feature mapping module includes: The character feature processing module is configured to map both the first character feature and the second character feature to a first reference feature in the reference feature sequence if there is no gap feature between the first character feature in the predicted feature sequence and the second character feature following the first character feature. And if there is at least one interval feature between the third character feature in the predicted feature sequence and the fourth character feature following the third character feature, the third character feature is mapped to the second reference feature in the reference feature sequence, and the fourth character feature is mapped to the third reference feature following the second reference feature.

6. The apparatus according to claim 5, wherein the parameter determining module comprises: The recognition result acquisition module is configured to acquire the recognition result of the characters in the training image based on the predicted feature sequence if at least one of the following is determined: The predicted feature sequence contains predicted features representing characters in the training image that are not mapped to the reference features. The reference feature sequence contains reference features that have not been mapped to predicted features; as well as The second loss determination module is configured to determine a second loss function based on the difference between the recognition result and the real characters in the training image; as well as The second minimization module is configured to determine the parameters of the first model by minimizing the second loss function.

7. The apparatus according to claim 5, wherein the parameter determining module comprises: The first loss determination module is configured to determine a first loss function based on the difference between the corresponding predicted features and the mapped reference features if all predicted features representing characters in the training image in the predicted feature sequence are mapped to corresponding reference features, and all reference features in the reference feature sequence are mapped to predicted features in the predicted feature sequence; and The first minimization module is configured to determine the parameters of the first model by at least minimizing the first loss function.

8. The apparatus of claim 5, wherein the first model is based on a connection-time classification (CTC) architecture, and the second model is based on an attention architecture.

9. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 4 when executed by the at least one processing unit.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 4.

Images