Model training and text recognition method and device, and storage medium

Abstract

Embodiments of the present application provide a model training and text recognition method and device, and a storage medium. The model training method comprises: obtaining image features and text features of a target image; fusing the image features and the text features to obtain image-text features of the target image; and training a text correction model based on the image-text features and a corrected text label of the text features, the text correction model being used to correct the text features. In the scheme of the embodiments of the present application, the image features and the text features are fused in the image-text features, so that more context semantic factors represented by the image features are included in the fused features. Therefore, the correction ability of the text correction model is improved based on the fused features to correct the text features, and the text recognition effect is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training and text recognition method, apparatus and storage medium. Background Technology

[0002] Scene text recognition (STR) is a long-standing and challenging research area in computer vision. With the rise of deep learning, STR methods based on convolutional neural networks (CNNs) have also made significant progress and have been widely applied, generating enormous research and application value.

[0003] When using the current STR algorithm for text recognition, the text in the scene often exhibits characteristics such as blurriness, variable fonts, uneven lighting, partial occlusion, perspective changes, low image resolution, and poor image quality.

[0004] Therefore, existing text recognition models have poor text recognition performance. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a model training and text recognition method, apparatus, and storage medium to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the present invention, a model training method is provided, comprising: acquiring image features and text features of a target image; fusing the image features and text features to obtain image-text features of the target image; and training a text correction model based on the image-text features and corrected text labels of the text features, wherein the text correction model is used to correct the text features.

[0007] In another implementation of the present invention, the step of obtaining image features and text features of the target image includes: extracting features from the target image to obtain image features; and performing text recognition on the image features to obtain text features.

[0008] In another implementation of the present invention, fusing the image features and text features to obtain the image-text features of the target image includes: concatenating the dimensional representations of the image features and the dimensional representations of the text features to obtain the image-text features of the target image.

[0009] In another implementation of the present invention, the text correction model includes a context fusion layer and a correction layer, wherein the output of the context fusion layer is connected to the input of the correction layer. Training the text correction model based on the image-text features and the corrected text labels of the text features includes: training the text correction model using the image-text features as input to the context fusion layer and the corrected text labels as output to the correction layer. The context fusion layer is used to perform context fusion on the text features and the image features, and the correction layer is used to correct the context-fused image-text features.

[0010] In another implementation of the present invention, the correction layer includes a deletion operation module and an insertion operation module. The output of the deletion operation module is connected to the input of the insertion operation module. The input of the deletion operation module serves as the input of the correction layer, and the output of the insertion operation module serves as the output of the correction layer. The deletion operation module is used to delete characters from the context-fused image-text features, and the insertion operation module is used to insert characters into the image-text features containing the deleted characters.

[0011] In another implementation of the present invention, the insertion operation module includes a placeholder addition module and a character prediction module. The input of the deletion operation module is used as the input of the insertion operation module, the output of the deletion operation module is connected to the input of the placeholder addition module, and the output of the placeholder addition module is used as the output of the insertion operation module. The placeholder addition module is used to add placeholders to the graphic features of the deleted character, and the character prediction module is used to predict the characters of the placeholders in the graphic features with added placeholders.

[0012] According to a second aspect of the present invention, a text recognition method is provided, comprising: extracting features from an image to be recognized to obtain image features of the image to be recognized; fusing the image features and text features of the image to be recognized to obtain image-text features of the image to be recognized; and inputting the image-text features into a text correction model to obtain corrected text of the text features, wherein the text correction model is trained using the method according to the first aspect.

[0013] In another implementation of the present invention, the step of inputting the image and text features into a text correction model to obtain the corrected text of the text features includes: inputting the image and text features into a text correction model to obtain the initial corrected text of the text features; and updating the image and text features by replacing the text features with the initial corrected text.

[0014] In another implementation of the present invention, the method further includes: performing text recognition based on the image features of the image to be recognized to obtain the text features of the image to be recognized.

[0015] In another implementation of the present invention, the method further includes: obtaining a recognition service request, the recognition service request including the image to be recognized; and responding to the recognition service request by returning the corrected text of the text features.

[0016] According to a third aspect of the present invention, a text recognition method is provided, comprising: acquiring a trigger instruction input in a human-computer interaction interface for an image to be recognized; responding to the trigger instruction and generating a recognition service request based on the image to be recognized; sending the recognition service request to perform text recognition based on the text recognition method described in the second aspect; and acquiring corrected text of the text features.

[0017] According to a fourth aspect of the present invention, a model training apparatus is provided, comprising: an acquisition module for acquiring image features and text features of a target image; a fusion module for fusing the image features and text features to obtain image-text features of the target image; and a training module for training a text correction model based on the image-text features and corrected text labels of the text features, wherein the text correction model is used to correct the text features.

[0018] According to a fifth aspect of the present invention, a text recognition device is provided, comprising: an extraction module for extracting features from an image to be recognized to obtain image features of the image to be recognized; a fusion module for fusing the image features and text features of the image to be recognized to obtain image-text features of the image to be recognized; and a correction module for inputting the image-text features into a text correction model to obtain corrected text of the text features, wherein the text correction model is trained using the method described in the first aspect.

[0019] According to a sixth aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first or second aspect.

[0020] According to a seventh aspect of the present invention, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first or second aspect.

[0021] In the embodiment of the present invention, the image and text features are fused together, so that the fused features include more contextual semantic factors represented by the image features. Therefore, based on the fused features, the text features are corrected, which improves the correction capability of the text correction model and thus improves the text recognition effect. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0023] Figure 1 This is a schematic diagram illustrating an example text recognition method.

[0024] Figure 2A This is a flowchart of the steps of a model training method according to another embodiment of the present invention.

[0025] Figure 2B for Figure 2A The flowchart of the text recognition method corresponding to the embodiment.

[0026] Figure 2C This is a flowchart illustrating the steps of a text recognition method according to another embodiment of the present invention.

[0027] Figure 3A and 3B According to Figure 2B and Figure 2A A schematic block diagram illustrating a text processing procedure as a specific example of an embodiment.

[0028] Figure 4 This is a structural block diagram of an apparatus according to another embodiment of the present invention.

[0029] Figure 5 This is a structural block diagram of an apparatus according to another embodiment of the present invention.

[0030] Figure 6 This is a schematic diagram of the structure of an electronic device according to another embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art should fall within the protection scope of the present invention.

[0032] The specific implementation of the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0033] Figure 1This is a schematic diagram illustrating an example text recognition method. Figure 1 The text recognition method is executed by a pre-trained scene text recognition model 120. The image 110 containing text information is input into the scene text recognition model 120, and the text recognition result 130 is output accordingly.

[0034] Specifically, the scene text recognition model 120 can be obtained by training a CNN network. The convolutional layers in the CNN network can perform text recognition on the image 110 to obtain the separated characters "V", "A", "C", "A", "T", "I", "O" and "N". Then, the characters are concatenated into a string to obtain the text recognition result 130.

[0035] However, if the image 110 has low resolution, or if the image 110 has missing parts that make the text information incomplete, the text recognition result 130 may be inaccurate. For example, the character "C" may be incorrectly recognized as "G", resulting in the incorrect recognition result: the string "VAGATION".

[0036] Figure 2A This is a flowchart illustrating the steps of a model training method according to another embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in cloud servers such as dedicated clouds, private clouds, or hybrid clouds. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference operations.

[0037] The model training method in this embodiment includes:

[0038] S210: Obtain the image features and text features of the target image.

[0039] It should be understood that the text in the embodiments of the present invention only includes text in the form of characters (including words and Chinese characters), sentences, paragraphs, and chapters. The text in the training samples may not have undergone word embedding processing (characters as units), and word embedding processing may be performed before participating in training; the text in the training samples may be text that has undergone word embedding processing, and the text after word embedding processing can directly participate in model training.

[0040] It should also be understood that text features can be text features obtained from text recognition based on the target image. For example, they can be based on... Figure 1 Model 120 performs text recognition on the target image to obtain text features; alternatively, other text recognition methods can be used to recognize the target image and obtain text features. Alternatively, the text features correspond to the target image; for example, the text features are features determined based on the target image. Alternatively, the text features can be features that do not match the target image; for example, the text features are unrelated to the target image.

[0041] S220: Fuse image features and text features to obtain the image and text features of the target image.

[0042] It should be understood that the fusion process can be achieved by adding the dimensional representations of image features and text features. That is, the image features and text features are aligned, and then each element in the dimensional representation of the image features is added to each element in the dimensional representation of the text features to obtain the dimensional representation of the image-text features. Alternatively, a concatenation process can be used, for example, concatenating each element in the dimensional representation of the image features with each element in the dimensional representation of the text features. The resulting dimensional representation of the image-text features is the sum of the dimensional representations of the image features and the dimensional representations of the text features.

[0043] S230: Based on image and text features, corrected text labels are generated, and a text correction model is trained. The text correction model is used to correct text features.

[0044] It should be understood that the above training can be supervised training, and the text correction model can be a classification model or a machine model based on imitation learning.

[0045] It should also be understood that the text correction model may include at least one correction layer, each performing at least one correction operation. The at least one correction operation includes a text deletion operation and / or a text insertion operation. For example, a text deletion operation may delete a portion of text containing at least one character, and a text insertion operation may insert a portion of text containing at least one character. Specifically, the text insertion operation may be further implemented as a serial operation including a text placeholder operation and a text prediction operation.

[0046] In one example, during supervised training, the input and output of each correction layer can be connected to the image features and the correction text labels, respectively.

[0047] In another example, the individual correction operations can also perform serial correction processing, and the individual correction layers can be connected serially. For example, one can delete a portion of text and then insert another portion; or, for another example, one can insert a portion of text and then delete another portion.

[0048] In the embodiment of the present invention, the image and text features are fused together, so that the fused features include more contextual semantic factors represented by the image features. Therefore, based on the fused features, the text features are corrected, which improves the correction capability of the text correction model and thus improves the text recognition effect.

[0049] In other examples, image features and text features of the target image are obtained, including: performing feature extraction on the target image to obtain image features, and performing text recognition on the image features to obtain text features.

[0050] Therefore, text features are determined based on the target image and are related to image features, which improves the training effect of the model and makes the predicted ability of the trained model stronger.

[0051] In other examples, image features and text features are fused to obtain the image-text features of the target image, including: concatenating the dimensional representations of the image features and the dimensional representations of the text features to obtain the image-text features of the target image.

[0052] Therefore, the concatenation operation improves the efficiency of data processing that integrates text and image features, and enhances the flexibility of data processing compared to the method of performing addition processing based on the dimensional representations of both after alignment.

[0053] Figure 2B for Figure 2A A flowchart illustrating the steps of the text recognition method corresponding to this embodiment. The text recognition method in this embodiment includes:

[0054] S270: Extract features from the image to be recognized to obtain the image features of the image to be recognized.

[0055] S280: Fuse image features and text features of the image to be identified to obtain the image-text features of the image to be identified.

[0056] S290: Based on the input of image and text features into the text correction model, the corrected text with text features is obtained. The text correction model is trained using the model training method.

[0057] In other examples, the text correction model is input based on image and text features to obtain corrected text, including: inputting image and text features into the text correction model to obtain initial corrected text. Then, the image and text features are updated by replacing the text features with the initial corrected text. Thus, through multiple iterations, further text correction is performed, improving the text recognition performance.

[0058] In other examples, text recognition methods also include: performing text recognition based on the image features of the image to be recognized, thereby obtaining the text features of the image to be recognized.

[0059] In other examples, the text correction model includes a context fusion layer and a correction layer, with the output of the context fusion layer connected to the input of the correction layer. Training the text correction model based on corrected text labels derived from image and text features includes: training the text correction model using image and text features as input to the context fusion layer and corrected text labels as output to the correction layer. The context fusion layer performs contextual fusion of text and image features, and the correction layer corrects the context-fused image and text features.

[0060] Therefore, the context fusion layer can perform contextual fusion of image features and text features, enabling the model to learn more contextual semantic features in the target image during training, thereby improving the training effect of the model.

[0061] Furthermore, the correction layer includes a deletion operation module and an insertion operation module. The output of the deletion operation module is connected to the input of the insertion operation module. The input of the deletion operation module serves as the input of the correction layer, and the output of the insertion operation module serves as the output of the correction layer. The deletion operation module is used to delete characters from the context-fused image-text features, and the insertion operation module is used to insert characters into the image-text features containing the deleted characters.

[0062] Therefore, the deletion and insertion operation modules can perform more reliable correction processing. Furthermore, the deletion and insertion operations cover various possible text correction methods, which improves the reliability of the model's text recognition. At the same time, it makes the trained model more interpretable, allowing for more targeted adjustment of the model's parameters based on the model's structure, thus increasing the potential for further improvement.

[0063] Furthermore, the insertion operation module includes a placeholder addition module and a character prediction module. The input of the deletion operation module is used as the input of the insertion operation module. The output of the deletion operation module is connected to the input of the placeholder addition module, and the output of the placeholder addition module is used as the output of the insertion operation module.

[0064] The placeholder addition module is used to add placeholders to the image and text features of deleted characters, and the character prediction module is used to predict the characters of the placeholders in the image and text features with added placeholders.

[0065] Furthermore, the text recognition method also includes: obtaining a recognition service request, the recognition service request including the image to be recognized; responding to the recognition service request and returning the corrected text of the text features.

[0066] It should be understood that when the text recognition method in this embodiment is applied to the server, a service request can be obtained from the client. A terminal device with the client installed can generate a service request based on the image to be recognized, and the server parses the image to be recognized based on the service request. Then, after text recognition is completed, corrected text with text features can be returned to the client. The client can then directly or indirectly display the corrected text with text features in the human-computer interaction interface.

[0067] Alternatively, the client on the terminal device can extract features from the image to be recognized, obtain the image features of the image, and then fuse the image features with the text features of the image to be recognized to obtain the image-text features of the image to be recognized. Then, the image-text features of the image to be recognized are included in the service request, and the server parses the image-text features of the image to be recognized based on the service request. Finally, after the text recognition is completed, the corrected text of the text features can be returned to the client.

[0068] It should also be understood that when the text recognition method of this embodiment is applied to a terminal device, the terminal device may have a client installed, which stores a text correction model. In other words, the client can be used to implement local text recognition or offline text recognition. In this case, the recognition service request can be equivalent to a text recognition instruction. For example, in a human-computer interaction interface, a text recognition instruction is generated based on a trigger instruction of the image to be recognized. Then, in the human-computer interaction interface, the corrected text with text features is displayed directly or indirectly.

[0069] Figure 2C This is a flowchart illustrating the steps of a text recognition method according to another embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, such as a mobile terminal (e.g., a mobile phone, a tablet, etc.) or a PC. The electronic device may include a human-computer interaction interface.

[0070] During the model training phase, computing devices configured with a CPU (processor unit) + GPU (acceleration unit) architecture (e.g., data centers) can be used to train the encoder-decoder model based on training samples. These computing devices, such as data centers, can be deployed in cloud servers such as dedicated clouds, private clouds, or hybrid clouds. Similarly, during the inference phase, computing devices configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference computations.

[0071] The text recognition method of this embodiment includes:

[0072] S2100; Obtain the trigger command input for the image to be recognized from the human-computer interaction interface.

[0073] S2200: In response to the trigger command, generate a recognition service request based on the image to be recognized.

[0074] S2300: Send a recognition service request to perform text recognition based on a text recognition method.

[0075] S2400: Corrected text that acquires text features.

[0076] It should be understood that trigger commands can be input on the image to be recognized on the client side. For example, long-pressing or clicking on the image to be recognized on the client's human-computer interaction interface can generate a recognition service request. More generally, terminal devices with the client installed can also generate service requests based on the image to be recognized based on other operations.

[0077] It should also be understood that an identification service request can be sent to the server. The server can then execute... Figure 2B The method parses the image to be recognized based on the service request. Then, after text recognition is complete, corrected text with text features can be returned to the client. The client can then directly or indirectly display the corrected text with text features in the human-computer interaction interface.

[0078] Specifically, refer to Figure 3A The text recognition processing is implemented by a character recognition network. The character recognition network includes at least a CNN network 310, a dimension alignment module 320, and a transformer encoder 330.

[0079] Furthermore, the input image is fed into the CNN network 310 for image feature extraction to obtain an intermediate feature map. The intermediate feature map is processed by the dimension alignment module 320 to obtain sequence features t. Then, the sequence features t are processed by the transformer encoder 330 to obtain sequence features T.

[0080] In other words, the image to be recognized as input can be image X (32x128x3), where 32 and 128 represent the number of pixels in the two dimensions of the image, respectively, and 3 represents the number of RGB channels. An intermediate layer feature of 8x32x512 is obtained through CNN encoding. This intermediate layer feature is divided into two processing branches. One branch uses a positional attention mechanism to obtain an Nx512-dimensional feature F aligned with the text length N. Then, the cross-entropy loss function is used to drive the network for text recognition training, yielding the text features, i.e., sequence features t, as the prediction result of the visual model.

[0081] In addition, the intermediate feature map serves as another processing branch. Based on the dimension alignment module 340, the feature height is reduced to obtain a 5x32x512 dimensional feature V for subsequent feature fusion.

[0082] Similarly, the dimensional representation described above also applies to data samples during the training phase, which will not be elaborated upon in this embodiment.

[0083] Then, the string sequence feature t is encoded using the word embedding module, and then a transformer is used for feature learning to obtain the Nx512-dimensional text feature T. Here, the string can be a visual prediction result or the output of subsequent "delete" and "insert" operations.

[0084] More specifically, refer to Figure 3B As a specific example of the model training phase, the deletion operation module can be implemented as deleting classifier 370, the placeholder addition module can be implemented as adding classifier 380, and the character prediction module can be implemented as predicting classifier 390.

[0085] Furthermore, sequence feature V+T is an example of image-text features, generated by concatenating sequence feature V and sequence feature T. Sequence feature V and sequence feature T are generated from the target images used as training samples.

[0086] The text correction model includes a Transformer encoder 350, a dimension alignment module 360, a deletion classifier 370, a placeholder addition classifier 380, and a character prediction classifier 390. Specifically, the Transformer encoder 350 is an example of a context fusion module, which performs context fusion on the sequence feature V+T to obtain the fused sequence feature V+T. The dimension alignment module 360 ​​can perform dimension alignment on the fused sequence feature V+T to obtain a sequence feature t' with the same length as the sequence feature T.

[0087] The sequence feature t' is obtained as a text correction label by at least one of the deletion classifier 370, the placeholder addition classifier 380, and the character prediction classifier 390.

[0088] In one example, the output of the deletion classifier 370 can also be connected to the sequence feature T' because the output of the deletion operation module can be connected to the text correction label, enabling efficient and reliable training of the parameters in the deletion classifier 370 when only characters need to be deleted to perform correction.

[0089] In another example, the output of the dimension alignment module 360 ​​can also be connected to the input of the placeholder addition classifier 380. This is because the output of the dimension alignment module 360 ​​can be connected to the input of the correction layer, enabling efficient and reliable parameters to be trained in the placeholder addition classifier 380 and the character prediction classifier 390 when only characters need to be inserted to perform correction.

[0090] In another example, the deletion classifier 370, the placeholder addition classifier 380, and the character prediction classifier 390 are connected in sequence because, in cases where characters need to be deleted and inserted to perform corrections, efficient and reliable parameters can be trained in the deletion classifier 370, the placeholder addition classifier 380, and the character prediction classifier 390.

[0091] It should be understood that the characters mentioned above are only examples of partial text. In addition, the connection positions of placeholder addition classifier 380 and character prediction classifier 390 and deletion classifier 370 can be interchanged. In other words, a part of the text can be inserted first, and then another part of the text can be deleted.

[0092] In addition, refer to again Figure 3B As a specific example of performing text correction during the inference phase,

[0093] First, the image to be recognized is processed to obtain sequence features V+T. Specifically, text recognition can be performed on the image based on a CNN network to obtain individual characters, and then these characters are concatenated into a string as text features. Additionally, image features can be extracted from intermediate feature maps in text recognition using a CNN network.

[0094] Then, the Transformer encoder 350, as an example of the context fusion module, performs context fusion on the sequence feature V+T to obtain the fused sequence feature V+T. The dimension alignment module 360 ​​can perform dimension alignment on the fused sequence feature V+T to obtain a sequence feature t' with the same length as the sequence feature T.

[0095] The sequence feature t' is obtained by removing at least one of the classifier 370, the placeholder addition classifier 380, and the character prediction classifier 390. The sequence feature T' can be used as the correction text or as the initial correction text.

[0096] With sequence feature T' as the initial corrected text, the corrected text can be obtained through iterative processing. Specifically, sequence feature T is updated based on sequence feature T' to obtain sequence feature V+T'. The updated sequence feature V+T' can then be input again to the input side of the context fusion module (i.e., the input side of the text correction model). It should be understood that the number of times the sequence feature is updated is the number of iterations.

[0097] In one example, the iterative processing described above can be performed based on a predetermined number of iterations. In other words, when the predetermined number of iterations is reached, the output of the text correction model will be used as the corrected text.

[0098] Alternatively, it can be determined whether the output of the text correction model after the current iteration is the same as the output of the text correction model after the previous iteration, or whether the difference between the two is less than a predetermined condition. If they are the same or the difference between them is less than the predetermined condition, then the output of the text correction model after the current iteration is the corrected text; in other words, the correction process stops and no further iterations are performed. For example, with... Figure 1 Taking the data as an example, the output of the text correction model after the current iteration is the sequence "VACATION", and the output of the text correction model after the previous iteration is also the sequence "VACATION". Since they are the same, "VACATION" is used as the correction text (which can also be called the correction text sequence). Alternatively, the output of the text correction model after the current iteration is the sequence "VACATION", the output of the text correction model after the previous iteration is the sequence "VAGATION", and the output of the text correction model before the previous iteration is "VAON". Since "VAGATION" and "VAON" are significantly different (4 characters), the difference reaches 1 / 2 > the predetermined condition 1 / 4. Therefore, the current iteration continues. Then, compared with the previous iteration, "VACATION" and "VAGATION" differ by only one character, that is, the difference is 1 / 8 < the predetermined condition 1 / 4. Therefore, directly using "VAGATION" output from the current iteration as the correction text can also be called the correction text sequence.

[0099] Alternatively, it can be determined whether the proportion of identical outputs after multiple iterations of the text correction model is higher than a predetermined condition. If the proportion of identical outputs is not lower than a predetermined condition (e.g., 50%), then outputs with such identical proportions are considered corrected text. In other words, the correction process stops and no further iterations are performed. For example, after 5 iterations, 6 outputs are obtained, including 3 instances of "VAGATION", 2 instances of "VAGATION", and 1 instance of "VATION". The number of times "VAGATION" is output is not less than 50%, thus "VAGATION" is identified as the corrected text.

[0100] Figure 4 This is a structural block diagram of an apparatus according to another embodiment of the present invention. The solution of this embodiment can be applied to any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, tablets, etc.), and PCs. For example, in the model training phase, a computing device (e.g., a data center) configured with a CPU (processor unit) + GPU (acceleration unit) architecture can be used to train the encoder-decoder model based on training samples. The computing device, such as a data center, can be deployed in cloud servers such as dedicated clouds, private clouds, or hybrid clouds. Correspondingly, in the inference phase, a computing device configured with a CPU (processor unit) + GPU (acceleration unit) architecture can also be used for inference operations.

[0101] The model training device in this embodiment includes:

[0102] The acquisition module 410 acquires the image features and text features of the target image.

[0103] The fusion module 420 fuses the image features and text features to obtain the image and text features of the target image.

[0104] Training module 430 trains a text correction model based on the image and text features and the corrected text labels of the text features. The text correction model is used to correct the text features.

[0105] In the embodiment of the present invention, the image and text features are fused together, so that the fused features include more contextual semantic factors represented by the image features. Therefore, based on the fused features, the text features are corrected, which improves the correction capability of the text correction model and thus improves the text recognition effect.

[0106] In other examples, the acquisition module is specifically used to include: extracting features from the target image to obtain image features; and performing text recognition on the image features to obtain text features.

[0107] In other examples, the fusion module is specifically used to: concatenate the dimensional representations of the image features and the dimensional representations of the text features to obtain the image-text features of the target image.

[0108] In other examples, the text correction model includes a context fusion layer and a correction layer, with the output of the context fusion layer connected to the input of the correction layer. The training module is specifically used to train the text correction model based on the image-text features as input to the context fusion layer and the corrected text label as output to the correction layer. The context fusion layer is used to perform contextual fusion of the text features and the image features, and the correction layer is used to correct the context-fused image-text features.

[0109] In other examples, the correction layer includes a deletion operation module and an insertion operation module. The output of the deletion operation module is connected to the input of the insertion operation module. The input of the deletion operation module serves as the input of the correction layer, and the output of the insertion operation module serves as the output of the correction layer. The deletion operation module is used to delete characters from the context-fused image-text features. The insertion operation module is used to insert characters into the image-text features containing the deleted characters.

[0110] In other examples, the insertion operation module includes a placeholder addition module and a character prediction module. The input of the deletion operation module serves as the input of the insertion operation module, the output of the deletion operation module is connected to the input of the placeholder addition module, and the output of the placeholder addition module serves as the output of the insertion operation module. The placeholder addition module is used to add placeholders to the graphic features of the deleted character, and the character prediction module is used to predict the characters of the placeholders in the graphic features with added placeholders.

[0111] The apparatus of this embodiment is used to implement the corresponding methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0112] Figure 5 This is a structural block diagram of a device according to another embodiment of the present invention. The text recognition device of this embodiment includes:

[0113] The extraction module 510 extracts features from the image to be identified to obtain the image features of the image to be identified.

[0114] The fusion module 520 fuses the image features and the text features of the image to be identified to obtain the image-text features of the image to be identified.

[0115] The correction module 530 inputs the image and text features into the text correction model to obtain the corrected text features. The text correction model is trained using a model training method.

[0116] In other examples, the correction module is specifically used to: input the image-text features into a text correction model to obtain the initial corrected text of the text features, and update the image-text features by replacing the text features with the initial corrected text.

[0117] In other examples, the text recognition device further includes a feature recognition module, which performs text recognition based on the image features of the image to be recognized to obtain the text features of the image to be recognized.

[0118] The apparatus of this embodiment is used to implement the corresponding methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0119] Reference Figure 6 The diagram shows a schematic of an electronic device according to another embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the electronic device.

[0120] like Figure 6 As shown, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0121] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0122] Communication interface 604 is used for communication with other electronic devices or servers.

[0123] The processor 602 is used to execute program 610, specifically to perform the relevant steps in the above method embodiments.

[0124] Specifically, program 610 may include program code that includes computer operation instructions.

[0125] Processor 602 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0126] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0127] Specifically, program 610 can be used to cause processor 602 to perform the following operations: acquire image features and text features of a target image; fuse the image features and text features to obtain image-text features of the target image; and train a text correction model based on the image-text features and the corrected text labels of the text features, wherein the text correction model is used to correct the text features.

[0128] Alternatively, program 610 may specifically be used to cause processor 602 to perform the following operations: extracting features from the image to be recognized to obtain image features of the image to be recognized; fusing the image features and text features of the image to be recognized to obtain image-text features of the image to be recognized; inputting the image-text features into a text correction model to obtain corrected text of the text features, wherein the text correction model is trained using a model training method.

[0129] Furthermore, the specific implementation of each step in procedure 610 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0130] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of the present invention can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present invention.

[0131] The methods described above according to embodiments of the present invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0132] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the embodiments of the present invention.

[0133] The above embodiments are only used to illustrate the embodiments of the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A model training method, comprising: obtaining an image feature and a text feature of a target image; fusing the image feature and the text feature to obtain a feature-text feature of the target image, the fusing being used to represent splicing of the image feature and the text feature; training a text correction model based on a correction text label of the feature-text feature and the text feature until a preset iteration condition is met, the text correction model being used to correct the text feature, wherein the preset iteration condition comprises one of the following: a previous output of the text correction model being the same as a current output, and a difference between the previous output and the current output of the text correction model being less than a predetermined condition; the text correction model comprising a context fusion layer and a correction layer, an output of the context fusion layer being connected to an input of the correction layer, the correction layer comprising a deletion operation module and an insertion operation module, an output of the deletion operation module being connected to an input of the insertion operation module, the input of the deletion operation module serving as an input of the correction layer, and an output of the insertion operation module serving as an output of the correction layer; the deletion operation module being used to perform character deletion on the context-fused feature-text feature according to the correction text label, and the insertion operation module being used to perform character insertion on the feature-text feature with deleted characters according to the correction text label.

2. The method of claim 1, wherein, the obtaining of the image feature and the text feature of the target image comprises: performing feature extraction on the target image to obtain the image feature; performing text recognition on the image feature to obtain the text feature.

3. The method of claim 1, wherein, the fusing of the image feature and the text feature to obtain the feature-text feature of the target image comprises: splicing a dimension representation of the image feature and a dimension representation of the text feature to obtain the feature-text feature of the target image.

4. The method of claim 1, wherein the training of the text correction model based on the correction text label of the feature-text feature and the text feature comprises: training the text correction model based on the feature-text feature as an input of the context fusion layer and based on the correction text label as an output of the correction layer, wherein the context fusion layer is used to perform context fusion on the text feature and the image feature, and the correction layer is used to correct the context-fused feature-text feature.

5. The method of claim 1, wherein, the insertion operation module comprises a placeholder adding module and a character predicting module, the input of the deletion operation module serving as an input of the insertion operation module, the output of the deletion operation module being connected to an input of the placeholder adding module, and an output of the placeholder adding module serving as an output of the insertion operation module, wherein the placeholder adding module is used to perform placeholder addition on the feature-text feature with deleted characters, and the character predicting module is used to perform character prediction on the placeholders in the feature-text feature with added placeholders.

6. A text recognition method, comprising: performing feature extraction on a to-be-recognized image to obtain an image feature of the to-be-recognized image; fusing the image feature and a text feature of the to-be-recognized image to obtain a feature-text feature of the to-be-recognized image; obtaining the corrected text of the text feature based on inputting the image-text feature into a text correction model, wherein the text correction model is trained by the method according to any one of claims 1-5.

7. The method of claim 6, wherein, The method further comprises: obtaining the corrected text of the text feature based on inputting the image-text feature into a text correction model, wherein the text correction model is trained by the method according to any one of claims 1-5. obtaining the corrected text of the text feature based on inputting the image-text feature into a text correction model, wherein the text correction model is trained by the method according to any one of claims 1-5.

8. The method of claim 6, wherein, replacing the text feature with the initial corrected text, and updating the image-text feature. The method further comprises:

9. The method of claim 6, wherein, performing text recognition based on the image feature of the to-be-recognized image to obtain a text feature of the to-be-recognized image. The method further comprises: obtaining an identification service request, wherein the identification service request comprises the to-be-recognized image; in response to the identification service request, returning the corrected text of the text feature.

10. A text recognition method, comprising: obtaining a trigger instruction input in a human-computer interaction interface for a to-be-recognized image; in response to the trigger instruction, generating an identification service request based on the to-be-recognized image; sending the identification service request to perform text recognition based on the text recognition method according to any one of claims 7-9; obtaining the corrected text of the text feature.

11. A model training apparatus, comprising: an obtaining module configured to obtain an image feature and a text feature of a target image; a fusion module configured to fuse the image feature and the text feature to obtain an image-text feature of the target image; a training module configured to train a text correction model based on the image-text feature and a corrected text label of the text feature until a preset iteration condition is met, wherein the text correction model is used to correct the text feature, and the preset iteration condition comprises one of the following: the previous output of the text correction model is the same as the current output, or a difference between the previous output of the text correction model and the current output is less than a predetermined condition; the text correction model comprises a context fusion layer and a correction layer, an output of the context fusion layer is connected to an input of the correction layer, the correction layer comprises a deletion operation module and an insertion operation module, an output of the deletion operation module is connected to an input of the insertion operation module, an input of the deletion operation module is taken as an input of the correction layer, and an output of the insertion operation module is taken as an output of the correction layer; the deletion operation module is configured to perform character deletion on the context-fused image-text feature according to the text correction label, and the insertion operation module is configured to perform character insertion on the image-text feature with deleted characters according to the text correction label.

12. A text recognition apparatus, comprising: an extraction module configured to perform feature extraction on a to-be-recognized image to obtain an image feature of the to-be-recognized image; a fusion module configured to fuse the image feature and a text feature of the to-be-recognized image to obtain an image-text feature of the to-be-recognized image; a correction module configured to obtain a corrected text of the text feature based on inputting the image-text feature into a text correction model, wherein the text correction model is trained by the method according to any one of claims 1-5.

13. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of claims 1-9.

Images