A text image recognition method, device, equipment and storage medium

Abstract

The embodiments of the present application relate to the field of data processing technology, and specifically to a text image recognition method, apparatus, device and storage medium, which are intended to shorten the development cycle of multi-language recognition tasks and improve multi-language recognition performance. The method comprises: extracting features of a text image to be recognized through a shared backbone network to obtain a shared feature map; performing text line detection on the shared feature map through a text line detection branch to obtain position information of the text line; extracting features from the shared feature map based on the position information of the text line through the shared backbone network to obtain text features; classifying the text features by language through a language classification branch to obtain language category information corresponding to the text features; and recognizing the text features through the corresponding text line recognition branch based on the language category information to obtain text recognition results.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically, to a text image recognition method, device, equipment and storage medium. Background Art

[0002] In the field of optical character recognition, multilingual recognition is a cutting-edge hotspot and a difficult point. Multilingual recognition is to recognize text in different languages ​​in images. It has a wide range of application scenarios in various fields such as production, life, teaching and training, such as its use in bilingual menus and bilingual teaching. There are two existing multilingual recognition methods. One is to use three independent branches, first perform text line detection, then classify the text into languages, and then recognize the classified text. The other is to train a unified multilingual recognition model, first perform text detection, and then directly input it into a unified multilingual recognition model for recognition.

[0003] In the prior art, the three branches of the first method are independent of each other and trained separately without interaction, and the end-to-end advantage is not utilized, resulting in poor recognition efficiency and effect. The second method requires the use of multilingual samples to train the model in advance. When a new language is added, the entire model needs to be retrained, and the data between languages ​​is unbalanced, resulting in a long model development cycle and poor recognition effect. Summary of the invention

[0004] The embodiments of the present application provide a text image recognition method, apparatus, device and storage medium, aiming to shorten the development cycle of multilingual recognition tasks and improve multilingual recognition performance.

[0005] A first aspect of an embodiment of the present application provides a text image recognition method, the method comprising:

[0006] The shared backbone network is used to extract features of the text image to be recognized, and a shared feature map is obtained;

[0007] Performing text line detection on the shared feature map through a text line detection branch to obtain position information of the text line;

[0008] By using the shared backbone network, extracting features from the shared feature map according to the position information of the text line to obtain text features;

[0009] Classify the text features into different languages ​​through a language classification branch to obtain language category information corresponding to the text features;

[0010] According to the language category information, the text features are recognized through the corresponding text line recognition branch to obtain a text recognition result.

[0011] Optionally, the method is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include:

[0012] Using the shared backbone network as a feature extraction network of the multilingual text image recognition network;

[0013] Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network;

[0014] Adding the language classification branch to the text recognition network;

[0015] Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions;

[0016] The multilingual text image recognition network is trained as a whole using the multi-task loss function to obtain the trained multilingual text image recognition network.

[0017] Optionally, selecting appropriate loss functions for different branches to train the multilingual text image recognition network includes:

[0018] Collect multiple pictures containing multiple languages ​​and characters, and put the multiple pictures containing multiple languages ​​and characters into the same set to obtain a training set;

[0019] Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are obtained in advance;

[0020] The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result;

[0021] The text line position, language category and text recognition result are compared with the text line position information, language information and text information, the difference obtained by the comparison is passed into the multi-task loss function, the parameters of the multilingual text image recognition network are adjusted, and the trained multilingual text image recognition network is obtained.

[0022] Optionally, when a new language needs to be recognized, the method further includes:

[0023] Adding a text line recognition branch corresponding to the new language in the text recognition network;

[0024] While ensuring that the existing model parameters remain unchanged, the language classification branch and the text line recognition branch are trained using pictures containing the new language to obtain the trained multilingual text image recognition network.

[0025] Optionally, while ensuring that the existing model parameters remain unchanged, the language classification branch and the text line recognition branch are trained using pictures containing the new language to obtain the trained multilingual text image recognition network, including:

[0026] When the new language is a language with more characters, a larger number of pictures are selected to train the language classification branch and the text line recognition branch;

[0027] When the new language is a language with fewer characters, a smaller number of pictures are selected to train the language classification branch and the text line recognition branch.

[0028] A second aspect of an embodiment of the present application provides a multi-text image recognition device, the device comprising:

[0029] An image recognition module is used to extract features of the text image to be recognized through a shared backbone network to obtain a shared feature map;

[0030] A text line detection module, used to perform text line detection on the shared feature map through a text line detection branch to obtain position information of the text line;

[0031] A feature extraction module, used to extract features from the shared feature map according to the position information of the text line through the shared backbone network to obtain text features;

[0032] A language classification module, used to classify the text features into languages ​​through language classification branches to obtain language category information corresponding to the text features;

[0033] The text line recognition module is used to recognize the text features through the corresponding text line recognition branch according to the language category information to obtain a text recognition result.

[0034] Optionally, the method is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include:

[0035] Using the shared backbone network as a feature extraction network of the multilingual text image recognition network;

[0036] Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network;

[0037] Adding the language classification branch to the text recognition network;

[0038] Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions;

[0039] The multilingual text image recognition network is trained as a whole using the multi-task loss function to obtain the trained multilingual text image recognition network.

[0040] Optionally, the multilingual text image recognition network is trained as a whole by using the multi-task loss function to obtain the trained multilingual text image recognition network, including:

[0041] Collect multiple pictures containing multiple languages ​​and texts, and put the multiple pictures containing multiple languages ​​and texts into the same set to obtain a training set;

[0042] Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are obtained in advance;

[0043] The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result;

[0044] The text line position, language category and text recognition result are compared with the text line position information, language information and text information, the difference obtained by the comparison is passed into the multi-task loss function, the parameters of the multilingual text image recognition network are adjusted, and the trained multilingual text image recognition network is obtained.

[0045] Optionally, the device further comprises:

[0046] A text line recognition branch adding submodule, used to add a text line recognition branch corresponding to the new language in the text recognition network;

[0047] The model training submodule is used to train the language classification branch and the text line recognition branch using pictures containing the new language while ensuring that the existing model parameters remain unchanged, so as to obtain the trained multilingual text image recognition network.

[0048] Optionally, the model training submodule includes:

[0049] A first model training submodule is used for selecting a large number of pictures to train the language classification branch and the text line recognition branch when the new language is a language with a large number of characters;

[0050] The second model training submodule is used to select a smaller number of pictures to train the language classification branch and the text line recognition branch when the new language is a language with fewer characters.

[0051] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0052] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0053] Using the text image recognition method provided in the present application, the shared backbone network is used to extract features from the text image to be recognized, and a shared feature map is obtained; the text line detection branch is used to detect the text line on the shared feature map, and the position information of the text line is obtained; the shared backbone network is used to extract features from the shared feature map according to the position information of the text line, and text features are obtained; the language classification branch is used to classify the text features, and the language category information corresponding to the text features is obtained; according to the language category information, the text features are recognized by the corresponding text line recognition branch to obtain text recognition results. In the present application, the text line detection branch, the language classification branch, and the text line recognition branch are put into the same network for end-to-end learning, which fully utilizes the advantages of end-to-end multi-task complementarity. When adding a new language, only the corresponding text line recognition branch needs to be added and the language classification branch needs to be modified, and the entire network does not need to be retrained, thereby shortening the development cycle of multi-language recognition tasks and improving the performance of multi-language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 is a flowchart of a text image recognition method proposed in an embodiment of the present application;

[0056] Figure 2 This is a flowchart of constructing a multilingual text image recognition network proposed in one embodiment of the present application;

[0057] Figure 3is a schematic diagram of a text image recognition device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] refer to Figure 1 , Figure 1 It is a flowchart of a text image recognition method proposed in one embodiment of the present application.

[0060] like Figure 1 As shown, the method comprises the following steps:

[0061] S11: Extract features of the text image to be recognized through the shared backbone network to obtain a shared feature map.

[0062] In this embodiment, the text image to be recognized is a picture containing text, and the text is multilingual. In one picture, multiple texts in different languages ​​can exist simultaneously. The shared backbone network is used to extract text features from the text image to be recognized. The shared feature map is a feature map extracted from the text image to be recognized, which can accurately reflect the regional position of the text in the text image to be recognized.

[0063] In this embodiment, the shared backbone network is a shared CNN (convolutional neural network) skeleton, which is equivalent to the backbone and skeleton of the entire multilingual text image recognition network. The shared backbone network extracts the text image to be recognized to obtain a shared feature map, analyzes it, and determines the location of the area containing the text. The network can identify the range of text in all languages ​​on the image and perform further feature extraction.

[0064] For example, the text image to be recognized may contain Chinese, English, Russian, etc., and the shared CNN skeleton may use networks such as resnet50 and resnet101.

[0065] S12: Perform text line detection on the shared feature map through a text line detection branch to obtain position information of the text line.

[0066] In this embodiment, the text line detection branch uses a text line recognition algorithm to identify the position of the text line in the shared feature map as the position information of the text line. The text line detection algorithm is a type of OCR (optical character recognition) algorithm.

[0067] Exemplarily, the text line detection algorithm can use the Bezier curve regression algorithm used in ABCnet, etc.

[0068] S13: Through the shared backbone network, feature extraction is performed on the shared feature map according to the position information of the text line to obtain text features.

[0069] In this embodiment, the shared backbone network will perform feature extraction on the text line at the detected position according to the position information of the text line detected by the text line detection branch, obtain the roi features of the text line on the CNN backbone, that is, the image features of the text line cropped from the picture, and obtain the feature vector of the text in this part.

[0070] Exemplarily, if 3 lines of text are cropped out in a picture, the shared backbone network will extract the picture features of the 3 cropped lines of text to obtain the text feature vectors of these 3 lines of text.

[0071] S14: Through the language classification branch, language classification is performed on the text features to obtain the language category information corresponding to the text features.

[0072] In this embodiment, the language classification branch classifies the text features through a language classification algorithm to obtain the language category information corresponding to the text features. This language category information shows which language the input text features belong to. When the text features contain multiple languages, these features can be classified according to the corresponding languages.

[0073] Exemplarily, the language classification algorithm can select the softmax cross-entropy loss algorithm. When the text corresponding to the text features contains "你好" and "hello", the language classification branch obtains through the language classification algorithm that "你好" belongs to Chinese and "hello" belongs to English.

[0074] S15: According to the language category information, the text features are recognized through the corresponding text line recognition branch to obtain the text recognition result.

[0075] In this embodiment, the text line recognition branch recognizes the text features through a text line recognition algorithm to obtain the text recognition result. The text line recognition algorithm is also a type of OCR recognition algorithm. For different languages, there is a separate text line recognition algorithm, that is, for different languages, there is a separate text line recognition branch.

[0076] Exemplarily, the text line recognition algorithm can adopt the CRNN_CTC algorithm. When the text features contain the Chinese "你好" and the English "hello", the text line recognition algorithm uses the Chinese recognition branch to recognize the text in the text image as "你好", and uses the English branch to recognize the text in the text image as "hello".

[0077] In this embodiment, feature extraction is performed through a suitable CNN skeleton, and then the text position, language information and text recognition results are obtained through the text line detection branch, language classification branch and text line recognition branch. The CNN network is used as the skeleton, and a branch network is added to it. In an end-to-end manner, multi-language text line detection, language classification, and text line recognition are placed in the same deep learning network for solution, achieving complementarity between the various sub-networks and improving the accuracy of text recognition.

[0078] In another embodiment of the present application, the method is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include:

[0079] S21: Using the shared backbone network as a feature extraction network of the multilingual text image recognition network.

[0080] In this embodiment, the multilingual text image recognition network is an end-to-end deep learning network. Multilingual text images are input into the network, and the text line position, language category, and text line recognition results in the image are output.

[0081] In this embodiment, after the image data is input into the network, the first thing it enters is the shared backbone network in the network. The shared backbone network is a subnetwork of the network, and the shared backbone network extracts the feature map in the input image. The shared backbone network can select the resnet50 network.

[0082] S22: Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network.

[0083] In this embodiment, the text line detection branch and the text line recognition branch are set in the same network, that is, the text recognition network. The text recognition network is a subnetwork in the multilingual text image recognition network. The text recognition network will perform text line detection and text line recognition on the input image.

[0084] For example, the text line detection adopts the Bezier curve regression algorithm in ABCnet, and the text recognition branch adopts the CRNN_CTC algorithm.

[0085] In this embodiment, when the network is constructed, the number of text line recognition branches can be set according to the number of languages ​​on the text image to be recognized. For example, if the number of languages ​​on the text image to be recognized is 3, then 3 text line recognition branches are added to the text recognition network, and each branch is responsible for text line recognition of one language.

[0086] S23: Adding the language classification branch to the text recognition network.

[0087] In this embodiment, a language classification branch is added to the text recognition network, and the language classification branch can adopt a softmax cross entropy loss containing a residual module.

[0088] S24: Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions.

[0089] In this embodiment, it is necessary to select a suitable loss function for each branch to achieve the best training effect for each branch. These loss functions can be summed up with a certain coefficient to obtain a multi-task loss function.

[0090] For example, the text line detection branch uses the L_det_loss loss function, the language classification branch uses the L_cls_loss loss function, and the text line recognition branch uses the L_CTC_loss loss function. The multi-task loss function is total_loss = 0.5*L_det_loss (text line detection loss function) + 0.1*L_cls_loss (language classification loss function) + 1.0*L_CTC_loss (text line recognition loss function). Among them, 0.5, 1, 1.0 are coefficient values ​​obtained through experiments.

[0091] S25: Performing overall training on the multilingual text image recognition network through the multi-task loss function to obtain the trained multilingual text image recognition network.

[0092] In this embodiment, after the basic structure of the model is constructed, the multilingual text image recognition network needs to be trained as a whole. The multilingual text image recognition network is trained using training samples combined with a multi-task loss function to obtain the trained multilingual text image recognition network.

[0093] In this embodiment, the text line detection branch, language classification branch, and text line recognition branch are set in the same network for end-to-end learning, which improves the performance of text recognition. A suitable loss function is selected for each branch and a multi-task loss function is constructed, achieving better training results and improving the performance of the model.

[0094] like Figure 2 As shown, Figure 2This is a flowchart of constructing a multilingual text image recognition network proposed in an embodiment of the present application. In the figure, a suitable shared CNN skeleton is first selected as a shared backbone network, and then a suitable text line detection algorithm is selected as a text line detection branch. The shared backbone network extracts text line features based on the text line position information detected by the text line detection branch, obtains the ROI features of the text line on the CNN skeleton, adds the language classification algorithm to the language classification branch to set the language classification branch, and adds the text line recognition algorithm to the text line recognition branch to set the text line recognition branch.

[0095] In another embodiment of the present application, the multilingual text image recognition network is trained as a whole by using the multi-task loss function to obtain the trained multilingual text image recognition network, including:

[0096] S31: Collect multiple pictures containing multiple languages ​​and texts, and put the multiple pictures containing multiple languages ​​and texts into the same set to obtain a training set.

[0097] In this embodiment, the pictures containing multiple languages ​​in the training set can be collected from the Internet, and the collected data are put into the same set to obtain the training set. In addition, in order to verify the effect of the model, a small amount of samples can also be put into another set as a test set.

[0098] For example, a text image simulation tool is used to obtain 100,000 lines of multilingual samples, which are stored in the form of pictures, to obtain 100,000 pictures containing multilingual text. 95,000 pictures are randomly selected as the training set, and the remaining 5,000 pictures are used as the test set.

[0099] S32: Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are pre-acquired.

[0100] In this embodiment, the training set is input into the multilingual text image recognition network. At this time, the model is untrained and needs to adjust its own parameters through the samples in the training set. The text line position information, language information and text information on the image in the training set are information obtained in advance during the text image simulation.

[0101] S33: The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result.

[0102] S34: Compare the text line location, language category and text recognition result with the text line location information, language information and text information, pass the difference obtained by the comparison into the multi-task loss function, adjust the parameters of the multilingual text image recognition network, and obtain the trained multilingual text image recognition network.

[0103] In this embodiment, under the TensorFlow deep learning framework, the network parameters of the multilingual text image recognition network can be adjusted by a multi-task loss function to obtain the trained multilingual text image recognition network.

[0104] After the network is completed, the images in the test set are input into the trained multilingual text image recognition network for training, and the training results are evaluated.

[0105] In another embodiment of the present application, when a new language needs to be recognized, the method further includes:

[0106] S41: Adding a text line recognition branch corresponding to the new language in the text recognition network.

[0107] In this embodiment, when a new language needs to be recognized, a text line recognition branch corresponding to the new language can be added to the text recognition network in the existing multilingual text image recognition network.

[0108] For example, when two new languages ​​need to be recognized, such as Arabic and German, text line recognition branches corresponding to the two new languages ​​are added to the text recognition network.

[0109] S42: While ensuring that the existing model parameters remain unchanged, use pictures containing the new language to train the language classification branch and the text line recognition branch to obtain the trained multilingual text image recognition network.

[0110] In this embodiment, while ensuring that the parameters of the trained model remain unchanged, the language classification branch and the two newly added text line recognition branches can be trained separately using pictures containing text in the new language. After the training is completed, the update of the multilingual text image recognition network is completed, and the updated multilingual text recognition network has the ability to recognize the newly added language.

[0111] For example, if the newly added languages ​​are Arabic and German, the pictures containing Arabic and German will be input into the multilingual text image recognition network. The language classification branch in the network will be trained to classify Arabic and German, and the two text line recognition branches in the network will be trained to recognize Arabic and German respectively.

[0112] In this embodiment, when a new language needs to be recognized, two independent text line recognition branches are added to the network. The text line recognition branch and the language classification branch are trained separately using new training data. The original parameters of the model will not be changed, and there is no need to train the model uniformly, thereby shortening the development cycle of multi-language recognition tasks.

[0113] In another embodiment of the present application, while ensuring that the existing model parameters remain unchanged, the language classification branch and the text line recognition branch are trained using pictures containing the new language to obtain the trained multilingual text image recognition network, including:

[0114] S51: When the new language is a language with more characters, a larger number of pictures are selected to train the language classification branch and the text line recognition branch.

[0115] S52: When the new language is a language with fewer characters, select a smaller number of pictures to train the language classification branch and the text line recognition branch.

[0116] In this embodiment, when the new language is a language with more characters, in order to ensure the performance of the model, more training samples need to be collected when training the text line recognition branch and the language classification branch, so that the model can learn as many characters as possible and ensure the recognition effect. When the new language is a language with fewer characters, a small amount of training samples is collected, which can also ensure the training effect.

[0117] For example, when the new language is Chinese, because Chinese is a language with more characters, more training samples are collected to train the text line recognition branch and the language classification branch, such as 20,000 samples containing Chinese. When the new language is English, because English only contains 26 letters and some characters, a small number of samples can be collected to train the text line recognition branch and the language classification branch, such as 5,000 samples containing English.

[0118] In this embodiment, a different text line recognition branch is used for each language. Each branch can focus on learning the language sample corresponding to the branch, which ensures the efficiency of training and avoids the problem of sample imbalance, thereby improving the performance of multi-language recognition.

[0119] Based on the same inventive concept, an embodiment of the present application provides a text image recognition device. Figure 3 , Figure 3 FIG. 3 is a schematic diagram of a text image recognition device 300 proposed in an embodiment of the present application. Figure 3 As shown, the device comprises:

[0120] The image recognition module 301 is used to extract features of the text image to be recognized through a shared backbone network to obtain a shared feature map;

[0121] A text line detection module 302, configured to perform text line detection on the shared feature map through a text line detection branch to obtain position information of the text line;

[0122] A feature extraction module 303 is used to extract features from the shared feature graph according to the position information of the text line through the shared backbone network to obtain text features;

[0123] A language classification module 304 is used to classify the text features into languages ​​through language classification branches to obtain language category information corresponding to the text features;

[0124] The text line recognition module 305 is used to recognize the text features through the corresponding text line recognition branch according to the language category information to obtain a text recognition result.

[0125] Optionally, the method is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include:

[0126] Using the shared backbone network as a feature extraction network of the multilingual text image recognition network;

[0127] Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network;

[0128] Adding the language classification branch to the text recognition network;

[0129] Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions;

[0130] The multilingual text image recognition network is trained as a whole using the multi-task loss function to obtain the trained multilingual text image recognition network.

[0131] Optionally, the multilingual text image recognition network is trained as a whole by using the multi-task loss function to obtain the trained multilingual text image recognition network, including:

[0132] Collect multiple pictures containing multiple languages ​​and texts, and put the multiple pictures containing multiple languages ​​and texts into the same set to obtain a training set;

[0133] Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are obtained in advance;

[0134] The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result;

[0135] The text line position, language category and text recognition result are compared with the text line position information, language information and text information, the difference obtained by the comparison is passed into the multi-task loss function, the parameters of the multilingual text image recognition network are adjusted, and the trained multilingual text image recognition network is obtained.

[0136] Optionally, the device further comprises:

[0137] A text line recognition branch adding submodule, used to add a text line recognition branch corresponding to the new language in the text recognition network;

[0138] The model training submodule is used to train the language classification branch and the text line recognition branch using pictures containing the new language while ensuring that the existing model parameters remain unchanged, so as to obtain the trained multilingual text image recognition network.

[0139] Optionally, the model training submodule includes:

[0140] A first model training submodule is used for selecting a large number of pictures to train the language classification branch and the text line recognition branch when the new language is a language with a large number of characters;

[0141] The second model training submodule is used to select a smaller number of pictures to train the language classification branch and the text line recognition branch when the new language is a language with fewer characters.

[0142] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the text image recognition method as described in any of the above embodiments of the present application.

[0143] Based on the same inventive concept, another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the text image recognition method described in any of the above embodiments of the present application.

[0144] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0146] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0147] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0151] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0152] The text image recognition method, device, equipment and storage medium provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A text image recognition method, characterized in that: The method comprises: The shared backbone network is used to extract features of the text image to be recognized, and a shared feature map is obtained; Performing text line detection on the shared feature map through a text line detection branch to obtain position information of the text line; By using the shared backbone network, extracting features from the shared feature map according to the position information of the text line to obtain text features; Classify the text features into different languages ​​through a language classification branch to obtain language category information corresponding to the text features; According to the language category information, the text features are recognized by a corresponding text line recognition branch to obtain a text recognition result; The method is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include: Using the shared backbone network as a feature extraction network of the multilingual text image recognition network; Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network; Adding the language classification branch to the text recognition network; Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions; Performing overall training on the multilingual text image recognition network by using the multi-task loss function to obtain the trained multilingual text image recognition network; The multilingual text image recognition network is trained as a whole by using the multi-task loss function to obtain the trained multilingual text image recognition network, including: Collect multiple pictures containing multiple languages ​​and texts, and put the multiple pictures containing multiple languages ​​and texts into the same set to obtain a training set; Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are obtained in advance; The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result; The text line position, language category and text recognition result are compared with the text line position information, language information and text information, the difference obtained by the comparison is passed into the multi-task loss function, the parameters of the multilingual text image recognition network are adjusted, and the trained multilingual text image recognition network is obtained.

2. The method according to claim 1, characterized in that When a new language needs to be recognized, the method further includes: Adding a text line recognition branch corresponding to the new language in the text recognition network; While ensuring that the existing model parameters remain unchanged, the language classification branch and the text line recognition branch are trained using pictures containing the new language to obtain the trained multilingual text image recognition network.

3. The method according to claim 2, characterized in that While ensuring that the existing model parameters remain unchanged, the language classification branch and the text line recognition branch are trained using pictures containing the new language to obtain the trained multilingual text image recognition network, including: When the new language is a language with more characters, a larger number of pictures are selected to train the language classification branch and the text line recognition branch; When the new language is a language with fewer characters, a smaller number of pictures are selected to train the language classification branch and the text line recognition branch.

4. A text image recognition device, characterized in that: The device comprises: An image recognition module is used to extract features of the text image to be recognized through a shared backbone network to obtain a shared feature map; A text line detection module, used to perform text line detection on the shared feature map through a text line detection branch to obtain position information of the text line; A feature extraction module, used to extract features from the shared feature map according to the position information of the text line through the shared backbone network to obtain text features; A language classification module, used to classify the text features into languages ​​through language classification branches to obtain language category information corresponding to the text features; A text line recognition module, used to recognize the text features through the corresponding text line recognition branch according to the language category information to obtain a text recognition result; The device is implemented based on a multilingual text image recognition network, and the steps of constructing the multilingual text image recognition network include: Using the shared backbone network as a feature extraction network of the multilingual text image recognition network; Using the text line detection branch and the text line recognition branch as a text recognition network of the multilingual text image recognition network; Adding the language classification branch to the text recognition network; Select appropriate loss functions for the text line detection branch, the language classification branch, and the text line recognition branch, respectively, and construct a multi-task loss function based on the above functions; Performing overall training on the multilingual text image recognition network by using the multi-task loss function to obtain the trained multilingual text image recognition network; The multilingual text image recognition network is trained as a whole by using the multi-task loss function to obtain the trained multilingual text image recognition network, including: Collect multiple pictures containing multiple languages ​​and texts, and put the multiple pictures containing multiple languages ​​and texts into the same set to obtain a training set; Inputting the training set into the multilingual text image recognition network, wherein the text line position information, language information and text information on the image in the training set are obtained in advance; The multilingual text image recognition network identifies the location of the text line on the image in the training set, the language category and the text recognition result; The text line position, language category and text recognition result are compared with the text line position information, language information and text information, the difference obtained by the comparison is passed into the multi-task loss function, the parameters of the multilingual text image recognition network are adjusted, and the trained multilingual text image recognition network is obtained.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 3 are implemented.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

Images