Text recognition model training method, text recognition method, electronic device, and storage medium

Abstract

The present application provides a training method for a text recognition model, a text recognition method, an electronic device, and a computer-readable storage medium. The method includes: obtaining multiple corpus data of target text; the multiple corpus data include first corpus data corresponding to RTL text and second corpus data corresponding to LTR text; combining the first corpus data and the second corpus data according to multiple combination methods to obtain combined corpus data corresponding to each combination method; generating sample images and labels corresponding to each combined corpus data, and constructing sample data based on the sample images and labels; training a deep learning model based on the sample data to obtain a text recognition model. The present application solution can train a text recognition model that accurately recognizes bidirectional text by generating sample data of complex bidirectional text.

Description

Technical Field

[0001] The present application relates to the technical field of text recognition, and in particular to a text recognition model training method, a text recognition method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the development of deep learning technology, OCR (Optical Character Recognition) technology has achieved remarkable results in the field of Chinese and English recognition. Both Chinese and English are LTR (Left To Right) text, and the reading and writing order is from left to right. In actual scenarios, there are also RTL (Right To Left) texts, such as Uyghur and Arabic, which are read and written from right to left. Some RTL texts are different from LTR texts due to their characteristics, making it impossible for OCR technology generally applicable to Chinese and English to process them normally. Taking Uyghur as an example, each Uyghur letter has 2 to 8 writing variations depending on how and where it appears (single, at the beginning of a word, in the middle of a word, at the end of a word), and there are a total of 126 writing variations for the 32 Uyghur letters. See Figure 1 , which is a schematic diagram of the characteristics of Weiwen provided in one embodiment of the present application, such as Figure 1 As shown, letters The writing style at the end of a word is completely different from that in the middle of a word. In addition, Uyghur is a continuous writing language, and each letter is deformed when it is written and connected with the letters before and after it (if any). Figure 1 In the example, (b) is the letter-split form of word (a).

[0003] In related technologies, in order to realize the recognition of RTL text, a deep learning model can be trained based on sample data. The sample data includes sample images containing RTL text and labels corresponding to the text. The trained deep learning model can subsequently recognize the RTL text in the image and output a character sequence. In actual scenarios, RTL text and LTR text may be mixed to form bidirectional text. Figure 1 For example, a text contains Uyghur, English, and numbers. To enable the model to recognize bidirectional text, sample data corresponding to RTL text and sample data corresponding to LTR text can be mixed and then used to train the deep learning model.

[0004] However, the deep learning model training process lacks training data corresponding to actual bidirectional text, and the subsequent recognition effect of bidirectional text is often not ideal. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a text recognition model training method, a text recognition method, an electronic device, and a computer-readable storage medium, which are used to train a model with high accuracy in bidirectional text recognition.

[0006] In one aspect, the present application provides a method for training a text recognition model, comprising:

[0007] Acquire multiple pieces of corpus data of the target text; wherein the multiple pieces of corpus data include first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text;

[0008] combining the first corpus data and the second corpus data according to a plurality of combination modes to obtain combined corpus data corresponding to each combination mode;

[0009] Generate sample images and string labels corresponding to each combined corpus data, and construct sample data based on the sample images and string labels;

[0010] The deep learning model is trained according to the sample data to obtain a text recognition model.

[0011] In one embodiment, the step of obtaining multiple pieces of corpus data of the target text includes:

[0012] Acquire text data from a data source of the target text;

[0013] Extracting candidate corpus data of the RTL character and candidate corpus data of the LTR character from the character data;

[0014] The candidate corpus data of the RTL characters and the candidate corpus data of the LTR characters are cut respectively to obtain first corpus data and second corpus data that meet character length restriction conditions.

[0015] In one embodiment, generating a sample image and a string label corresponding to each combined corpus data includes:

[0016] For each combined corpus data, determining a character string label corresponding to a target character string in the combined corpus data;

[0017] For each combined corpus data, a target character string in the combined corpus data is drawn onto a background image to obtain a sample image corresponding to the combined corpus data.

[0018] In one embodiment, determining the string label corresponding to the target string in the combined corpus data includes:

[0019] For the first corpus data in the combined corpus data, read a plurality of characters from the first corpus data and sort them in a specified order to determine a first tag sequence; wherein the specified order is opposite to the order in which the characters of the first corpus data are read;

[0020] For the second corpus data in the combined corpus data, read a plurality of characters from the second corpus data and sort them in the character reading order of the second corpus data to determine a second label sequence;

[0021] According to the combination mode corresponding to the combined corpus data, the first label sequence and the second label sequence corresponding to the combined corpus data are concatenated to obtain the character string label.

[0022] In one embodiment, determining the string label corresponding to the target string in the combined corpus data includes:

[0023] For the second corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the second corpus data to obtain a first substring;

[0024] splicing the first substring and the second substring of the first corpus data in the combined corpus data according to a splicing order opposite to the splicing order corresponding to the combined corpus data, and performing a sequence reversal operation on the spliced ​​strings; wherein the second substring is a string included in the first corpus data;

[0025] For the character string after the sequence reversal operation, characters are read one by one in order from left to right and corresponding character labels are determined, and a third label sequence consisting of multiple character labels is used as the character string label.

[0026] In one embodiment, drawing the target character string in the combined corpus data onto a background image to obtain a sample image corresponding to the combined corpus data includes:

[0027] For the first corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the first corpus data to obtain a third substring;

[0028] concatenating the third substring with a fourth substring of the second corpus data in the combined corpus data according to a combination mode corresponding to the combined corpus data; wherein the fourth substring is a string included in the second corpus data;

[0029] A plurality of characters are read from the concatenated character string, and are drawn onto the background image one by one to obtain the sample image.

[0030] In one embodiment, drawing the target character string in the combined corpus data onto a background image to obtain a sample image corresponding to the combined corpus data includes:

[0031] For the second corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the second corpus data to obtain a first substring;

[0032] splicing the first substring and the second substring of the first corpus data in the combined corpus data according to a splicing order opposite to the splicing order corresponding to the combined corpus data, and performing a sequence reversal operation on the spliced ​​strings; wherein the second substring is a string included in the first corpus data;

[0033] A plurality of characters are read from the character string after the sequence reversal operation, and are drawn onto the background image one by one to obtain the sample image.

[0034] In one embodiment, the method further comprises:

[0035] Before drawing the target character string on the background image, a background image is randomly generated for the combined corpus data.

[0036] In one embodiment, the first corpus data is Uyghur corpus data, and the method further includes:

[0037] Before drawing the target character string on the background image, each Uyghur character in the target character string is converted into Arabic characters.

[0038] In one embodiment, the deep learning model includes MobileNetV3 and BiLSTM;

[0039] The deep learning model is trained according to the sample data to obtain a text recognition model, including:

[0040] The MobileNetV3 and the BiLSTM are trained end-to-end according to the sample data to obtain the text recognition model.

[0041] On the other hand, the present application also provides a text recognition method, the method comprising:

[0042] Acquire an image to be recognized, wherein the image to be recognized contains RTL text and LTR text;

[0043] The image to be recognized is input into a text recognition model to obtain a character recognition result output by the text recognition model; wherein the text recognition model is trained by the above-mentioned text recognition model training method.

[0044] On the other hand, the present application also provides a text recognition model training device, comprising:

[0045] An acquisition module, configured to acquire a plurality of corpus data of the target text; wherein the plurality of corpus data includes first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text;

[0046] a combining module, configured to combine the first corpus data and the second corpus data according to a plurality of combining modes, to obtain combined corpus data corresponding to each combining mode;

[0047] A generation module, configured to generate sample images and labels corresponding to each combined corpus data, and construct sample data based on the sample images and labels;

[0048] The training module is used to train the deep learning model based on the sample data to obtain a text recognition model.

[0049] Furthermore, the present application also provides an electronic device, comprising:

[0050] processor;

[0051] a memory for storing processor-executable instructions;

[0052] The processor is configured to execute the above-mentioned text recognition model training method or text recognition method.

[0053] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the above-mentioned text recognition model training method or text recognition method.

[0054] The present application scheme combines the first corpus data corresponding to the RTL text and the second corpus data corresponding to the LTR text to obtain combined corpus data corresponding to a variety of combinations, and generates corresponding sample images and string labels based on the combined corpus data, thereby constructing sample data, and can train a text recognition model based on the sample data;

[0055] By generating sample data of complex bidirectional text, a text recognition model that can accurately recognize bidirectional text can be trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application.

[0057] Figure 1 A schematic diagram of the characteristics of Weiwen provided in one embodiment of the present application;

[0058] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0059] Figure 3 A flowchart of a method for training a text recognition model provided in one embodiment of the present application;

[0060] Figure 4 A flowchart of a method for acquiring corpus data provided in one embodiment of the present application;

[0061] Figure 5 A flowchart of a method for generating labels and sample images provided in one embodiment of the present application;

[0062] Figure 6 A schematic diagram of bidirectional text word order ambiguity provided in one embodiment of the present application;

[0063] Figure 7 A flowchart of a method for determining a string label according to an embodiment of the present application is provided;

[0064] Figure 8 A schematic diagram of a method for determining a string label provided in one embodiment of the present application;

[0065] Figure 9 A flowchart of a method for determining a string label provided in another embodiment of the present application;

[0066] Figure 10 A schematic diagram of a method for determining a string label provided in another embodiment of the present application;

[0067] Figure 11 A schematic diagram of a flow chart of a method for generating a sample image according to an embodiment of the present application;

[0068] Figure 12 A schematic flow chart of a method for generating a sample image according to another embodiment of the present application;

[0069] Figure 13 A schematic diagram of a method for generating a sample image according to an embodiment of the present application;

[0070] Figure 14 A schematic diagram of a method for generating a sample image according to another embodiment of the present application;

[0071] Figure 15 A schematic diagram of a method for determining a string label provided in one embodiment of the present application;

[0072] Figure 16 A schematic diagram of a method for generating a sample image according to an embodiment of the present application;

[0073] Figure 17 A block diagram of a text recognition model training device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0075] Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0076] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 2 In the example, a processor 11 is used. Processor 11 and memory 12 are connected via bus 10. Memory 12 stores instructions executable by processor 11. Processor 11 executes these instructions, enabling electronic device 1 to perform all or part of the method described in the following embodiments. In one embodiment, electronic device 1 may be a server or host computer configured to execute a text recognition model training method or a text recognition method.

[0077] The memory 12 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0078] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the text recognition model training method or text recognition method provided in the present application.

[0079] See also Figure 3 , is a flowchart of a method for training a text recognition model according to an embodiment of the present application, such as Figure 3As shown, the method may include the following steps 310 to 340.

[0080] Step 310: Acquire multiple pieces of corpus data of the target text; wherein the multiple pieces of corpus data include first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text.

[0081] The target text is the text that the text recognition model will subsequently recognize. The target text is bidirectional text, consisting of a first corpus containing RTL text and a second corpus containing LTR text. For example, the target text can be a text containing both Uyghur and English corpus data; alternatively, the target text can be a text containing both Arabic and numeric corpus data; or alternatively, the target text can be a text containing both Hebrew and English corpus data.

[0082] Step 320: Combine the first corpus data and the second corpus data according to a plurality of combination modes to obtain combined corpus data corresponding to each combination mode.

[0083] The combination mode is used to indicate the concatenation order of the first corpus data and the second corpus data. Exemplarily, the combination mode may include "LTR+RTL", "RTL+LTR", "RTL+LTR+RTL", "LTR+RTL+LTR", etc.

[0084] After obtaining multiple pieces of first corpus data and multiple pieces of second corpus data, the host can select one piece of first corpus data and one piece of second corpus data, and directly splice them into combined corpus data in "LTR+RTL" or "RTL+LTR".

[0085] When any first corpus data includes at least two words, the host can randomly select a space in the first corpus data for segmentation, thereby obtaining two segments of corpus data. After inserting a second corpus data into these two segments of corpus data, combined corpus data in the form of "RTL+LTR+RTL" can be obtained.

[0086] When any second corpus data includes at least two words, the host can randomly select a space in the second corpus data for segmentation, thereby obtaining two segments of corpus data. After inserting a piece of the first corpus data into these two segments of corpus data, combined corpus data in the form of "LTR+RTL+LTR" can be obtained.

[0087] The host combines the first corpus data and the second corpus data in a variety of combinations to obtain a plurality of combined corpus data corresponding to any combination.

[0088] Step 330: Generate sample images and string labels corresponding to each combined corpus data, and construct sample data based on the sample images and string labels.

[0089] Step 340: Train the deep learning model based on the sample data to obtain a text recognition model.

[0090] After obtaining multiple combined corpus data, for each combined corpus data, the host can generate a string label and a sample image of the combined corpus data based on the character string in the combined corpus data, and the sample image includes the above-mentioned character string. Here, the string label can be a label sequence composed of the character label corresponding to each character of the character string in the sample image. After the host labels the sample image with a string label, the sample data can be obtained. In addition, the host can also generate a sample image and string label corresponding to the first corpus data, and construct sample data containing only RTL text. The host can generate a sample image and string label corresponding to the second corpus data, and construct sample data containing only LTR text. The host can generate corresponding sample images and string labels based on the corpus data corresponding to punctuation marks, and construct sample data containing punctuation marks.

[0091] After constructing multiple pieces of sample data, the host can use the sample data to train a deep learning model. Here, the deep learning model can be a combination of CNN (Convolutional Neural Networks) and RNN (Recurrent Neural Network).

[0092] After inputting sample data into the deep learning model, the model generates predictions. The model's loss function is used to evaluate the difference between the predictions and the labels, and the model parameters are adjusted based on this difference. The loss function can be a CTC (Connectionist Temporal Classification) loss. After repeated iterations, when the loss function stabilizes, the deep learning model is considered converged, and a text recognition model is obtained.

[0093] Through the above measures, sample data of more complex bidirectional text can be generated. The sample data contains a mixture of LTR text and RTL text, which can make the trained text recognition model more accurate in recognizing bidirectional text.

[0094] In one embodiment, Figure 4 A flow chart of a method for obtaining corpus data provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method may include the following steps 311 to 313.

[0095] Step 311: Obtain text data from a data source of the target text.

[0096] The data source of the target text can be pre-configured, and can be the storage address of the corpus data corresponding to the target text, or the URL of a website containing the target text.

[0097] For example, the target text is a mixture of Uyghur, numbers, and English. The source of the target text can include websites using Uyghur (such as People's Daily Online, Xinhua News Agency, Northwest China Regional Website, etc.) and English websites. The host can crawl content from the above websites to obtain text data.

[0098] Step 312: Extract candidate corpus data of RTL characters and candidate corpus data of LTR characters from the character data.

[0099] Among them, the candidate corpus data is the corpus data obtained through preliminary extraction and has not been further processed.

[0100] The host can extract candidate corpus data for RTL characters using a regular expression corresponding to RTL characters; and extract candidate corpus data for LTR characters using a regular expression corresponding to LTR characters. Furthermore, the host can also extract candidate corpus data corresponding to expression symbols using a regular expression corresponding to punctuation marks.

[0101] Step 313: Cut the candidate corpus data of the RTL characters and the candidate corpus data of the LTR characters respectively to obtain first corpus data and second corpus data that meet the character length restriction condition.

[0102] Here, the character length restriction can be pre-configured based on experience to maximize the effect of subsequent model training. For example, the character length restriction for the first corpus data is that each corpus does not exceed 10 characters; the character length restriction for the second corpus data is that each corpus does not exceed 20 characters.

[0103] The host can check whether the character length of each candidate corpus data satisfies the character length restriction. On the one hand, if it does, the candidate corpus data can be directly used as the first corpus data or the second corpus data. On the other hand, if it does not, the candidate corpus data can be segmented into word units, and the resulting segmented corpus data can be used as the first corpus data or the second corpus data.

[0104] By means of the above measures, first corpus data and second corpus data that meet the character length restriction condition and contain characters of a single word order can be obtained, which facilitates subsequent splicing and combination to form combined corpus data.

[0105] In one embodiment, see Figure 5 , is a flow chart of a method for generating string labels and sample images provided in an embodiment of the present application, such as Figure 5 As shown, the method may include steps 510 to 520.

[0106] Step 510: For each combined corpus data, determine a string label corresponding to a target string in the combined corpus data.

[0107] The target character string is the character string indicated by the combined corpus data.

[0108] For each piece of combined corpus data, the host can read multiple characters in the combined corpus data, determine the corresponding character tag for each character, and construct a string tag corresponding to the target string based on the character tags of the multiple characters. The string tag includes multiple character tags, and the reading order of the multiple character tags is not affected by the text type.

[0109] Step 520: For each combined corpus data, draw the target character string in the combined corpus data onto the background image to obtain a sample image corresponding to the combined corpus data.

[0110] For each piece of combined corpus data, after the host determines the target string in the combined corpus data, it can use tools such as pillow and opencv to draw it onto the background image, thereby obtaining a sample image corresponding to the combined corpus data.

[0111] In this application, the combined corpus data contains RTL characters and LTR characters, and the main order of the corpus cannot be determined, which may cause problems when generating string tags. Figure 6 , is a schematic diagram of bidirectional text word order ambiguity provided by an embodiment of the present application, such as Figure 6 As shown, the bidirectional text "2020 "The text can be read in either a LTR-based or RTL-based character reading order, resulting in different reading results. Since it is not possible to read in a fixed character reading order, bidirectional text ambiguity will occur, making it impossible to obtain correct string labels or generate correct sample images.

[0112] In this case, it is necessary to perform targeted processing on the target strings in the combined corpus data to obtain string labels. See the following description for details.

[0113] In one embodiment, see Figure 7 , is a flow chart of a method for determining a string label provided in an embodiment of the present application, such as Figure 7As shown, the method may include the following steps 511A to 513A.

[0114] Step 511A: For the first corpus data in the combined corpus data, read a plurality of characters from the first corpus data and sort them in a specified order to determine a first tag sequence; wherein the specified order is opposite to the order in which the characters of the first corpus data are read.

[0115] For the first corpus data corresponding to the RTL text, the host can read multiple characters in sequence according to the character reading order from right to left of the first corpus data, and sort the multiple characters read in a specified order from left to right. For each character after sorting, the host can determine the character label corresponding to the character one by one, thereby obtaining multiple character labels. Here, the character label can be in the form of a code, a digital serial number, etc.; the arrangement order of the multiple character labels is the same as the specified order, and the multiple character labels constitute a first label sequence. The reading order of each character label in the first label sequence is not affected by the reading order of the corresponding RTL characters.

[0116] See also Figure 8 , which is a schematic diagram of a method for determining a tag provided in an embodiment of the present application, such as Figure 8 As shown, the first corpus data in the combined corpus data is Uyghur corpus data. After reading out multiple characters in the character reading order, they are sorted in the specified order. The sorted multiple characters are the same as the character order in the first corpus data. At this time, the corresponding character label can be determined character by character, thereby forming a first label sequence.

[0117] Step 512A: For the second corpus data in the combined corpus data, read a plurality of characters from the second corpus data and sort them in the character reading order of the second corpus data to determine a second tag sequence.

[0118] For the second corpus data corresponding to the LTR text, the host can read multiple characters from the second corpus data in sequence according to the character reading order from left to right, and sort the read multiple characters in the character reading order from left to right. For each character after sorting, the host can determine the character label corresponding to the character one by one, thereby obtaining multiple character labels. The multiple character labels constitute a second label sequence. Here, the character reading order of each character label in the second label sequence is not affected by the character reading order corresponding to the LTR.

[0119] by Figure 8 For example, the second corpus data in the combined corpus data is English corpus data, and according to the character reading order ( Figure 8After reading out a plurality of characters, the plurality of characters are arranged in a character reading order from left to right, and corresponding character labels are determined character by character, thereby forming a second label sequence.

[0120] Step 513A: Based on the combination mode corresponding to the combined corpus data, the first label sequence and the second label sequence corresponding to the combined corpus data are concatenated to obtain a character string label.

[0121] The host splices the first label sequence and the second label sequence in a combined manner according to the splicing order indicated, thereby obtaining a string label. Here, if the combined corpus data includes at least two first corpus data or at least two second corpus data, then at least two first label sequences or at least two second label sequences are spliced.

[0122] like Figure 8 As shown, the combination mode of the combined corpus data is "LTR+RTL", the second label sequence corresponding to the English corpus data is arranged in front, and the first label sequence corresponding to the Uyghur corpus data is arranged in the back, to obtain a string label.

[0123] When reading character tags in a string tag, the reading order is not affected by the character reading order of the text type. Figure 8 For example, the reading order of each character label (including the character labels corresponding to Uyghur characters) in the string label is from left to right.

[0124] In one embodiment, Figure 9 A flowchart of a tag determination method provided in another embodiment of the present application is shown in FIG. Figure 9 As shown, the method may include the following steps 511B to 513B.

[0125] 511B: For the second corpus data in the combined corpus data, perform a sequence reversal operation on the character strings in the second corpus data to obtain a first substring.

[0126] For the character strings in the second corpus data, the host can perform a sequence reversal operation on the entire string without reading each character one by one, thereby obtaining a first substring. Here, the reading order of each character in the first substring is affected by the reading order of the characters corresponding to the LTR.

[0127] See also Figure 10 , which is a schematic diagram of a method for determining a label provided by another embodiment of the present application, such as Figure 10 As shown, the second corpus data in the combined corpus data is English corpus data, and the host can directly perform a sequence reversal operation on the English character string to obtain the first substring.

[0128] 512B: Concatenate the first substring and the second substring of the first corpus data in the combined corpus data according to the concatenation order opposite to the concatenation order corresponding to the combined corpus data, and perform a sequence reversal operation on the concatenated string, wherein the second substring is a string included in the first corpus data.

[0129] The host concatenates the first substring and the second substring of the first corpus data according to a concatenation order opposite to the combination method to obtain a concatenated string. Further, the host performs a sequence reversal operation on the concatenated string.

[0130] by Figure 10 For example, the combination is "LTR+RTL". The host concatenates the first substring and the second substring in the order of "RTL+LTR" to obtain a concatenated string, and reverses the operation to obtain a string in the form of "LTR+RTL".

[0131] 513B: For the character string after the sequence reversal operation, read the characters one by one in order from left to right and determine the corresponding character labels, and use a third label sequence consisting of multiple character labels as the character string label.

[0132] After obtaining the string after the sequence reversal operation, the host can read each character in the string one by one in a left-to-right character reading order and determine the character label corresponding to the character, thereby obtaining multiple character labels. A third label sequence can be formed based on the multiple character labels. The host can use the third label sequence as the string label.

[0133] like Figure 10 As shown, after obtaining the string after the order reversal operation in the form of "LTR+RTL", the English characters are read first and then the Uyghur characters are read in order from left to right; when reading the English characters, they are read one by one from left to right and converted into character labels; when reading the Uyghur characters, they are read one by one from right to left and converted into character labels, and then the third label sequence composed of multiple character labels is used as the string label.

[0134] The multiple character tags in a string tag are independent of each other. When reading the character tags in a string tag, the reading order will not be affected by the reading order of the characters corresponding to the text type. Figure 10 For example, the reading order of each character label in the label (including the character labels corresponding to the Uyghur characters) is from left to right.

[0135] In one embodiment, see Figure 11 , is a flow chart of a method for generating a sample image provided by an embodiment of the present application, such as Figure 11 As shown, the method may include the following steps 521A to 523A.

[0136] Step 521A: for the first corpus data in the combined corpus data, perform a sequence reversal operation on the character strings in the first corpus data to obtain a third substring.

[0137] For the character string in the first corpus data, the host can perform a sequence reversal operation on the entire string without reading each character one by one, thereby obtaining a third substring. Here, the reading order of each character in the third substring is affected by the reading order of the corresponding characters in RTL.

[0138] Step 522A: Based on the combination mode corresponding to the combined corpus data, concatenate the third substring with the fourth substring of the second corpus data in the combined corpus data, wherein the fourth substring is a string contained in the second corpus data.

[0139] The host concatenates the third substring and the fourth substring in the concatenation order indicated in the combined manner, thereby obtaining a concatenated string.

[0140] Step 523A: Read multiple characters from the concatenated character string and draw them onto the background image one by one to obtain a sample image.

[0141] The reading order of the concatenated string is affected by the character reading order of the text type. The host reads multiple characters from the string using the character reading order corresponding to LTR text and the character reading order corresponding to RTL text, and then draws each character to obtain a sample image corresponding to the combined corpus data.

[0142] In one embodiment, see Figure 12 , is a flow chart of a method for generating a sample image provided by another embodiment of the present application, such as Figure 12 As shown, the method may include the following steps 521B to 523B.

[0143] Step 521B: for the second corpus data in the combined corpus data, perform a sequence reversal operation on the character strings in the second corpus data to obtain a first substring.

[0144] For the character strings in the second corpus data, the host can perform a sequence reversal operation on the entire string without reading each character one by one, thereby obtaining a first substring. Here, the reading order of each character in the first substring is affected by the reading order of the characters corresponding to the LTR.

[0145] Step 522B: Concatenate the first substring with the second substring of the first corpus data in the combined corpus data according to the concatenation order that is opposite to the concatenation order corresponding to the combined corpus data, and perform a sequence reversal operation on the concatenated string, wherein the second substring is the string contained in the first corpus data.

[0146] The host concatenates the first substring and the second substring of the first corpus data according to a concatenation order opposite to the combination method to obtain a concatenated string. Further, the host performs a sequence reversal operation on the concatenated string.

[0147] Step 523B: Read multiple characters from the string after the sequence reversal operation, and draw them one by one onto the background image to obtain a sample image.

[0148] The string after the order reversal operation is a regular string, so the reading order is affected by the character reading order corresponding to the text type. The host reads multiple characters from the string using the character reading order corresponding to LTR text and the character reading order corresponding to RTL text, and then draws them character by character to obtain a sample image corresponding to the combined corpus data.

[0149] In one embodiment, the first corpus data is Uyghur corpus data. Because Uyghur characters are only encoded in their own form in some encoding standards (such as Unicode), but lack corresponding encodings for their written forms, tools that draw sample images cannot properly draw images. To address this issue, before drawing the target string on the background image, each Uyghur character in the target string can be converted to Arabic characters. This measure allows tools based on encoding standards such as Unicode to correctly draw sample images.

[0150] In one embodiment, before drawing the target string on the background image, the host can randomly generate a background image for the combined corpus data. In this case, multiple background images can be generated for a single piece of combined corpus data, each with different sizes and scenes. Furthermore, when drawing the target string, a variety of fonts can be used to obtain sample images with diverse morphologies, which helps improve model robustness.

[0151] See also Figure 13 , is a schematic diagram of a method for generating a sample image according to an embodiment of the present application, which indicates the sample image generation process from step 521A to step 532A. Figure 13 As shown, the first corpus data in the combined corpus data is Uyghur corpus data, which can be converted into Arabic corpus data. Further, the Arabic corpus data is subjected to a sequence reversal operation to obtain a third substring.

[0152] exist Figure 13 In the combined corpus data, the concatenation method is "LTR+RTL". In the concatenated string, the fourth substring is placed first, and the third substring is placed last. Characters are read one by one from the concatenated string and drawn onto the background image to produce a sample image. English characters are drawn onto the background image from left to right, while Uyghur characters are drawn from right to left. In this case, the string in the drawn sample image is exactly the same as the target string in the combined corpus data.

[0153] See also Figure 14 , is a schematic diagram of a method for generating a sample image provided by another embodiment of the present application, which indicates the sample image generation process from step 521B to step 523B. Figure 14 As shown, the second corpus data in the combined corpus data is English corpus data. After the sequence reversal operation, the first substring "oisicedeht" can be obtained.

[0154] exist Figure 14 In the example, the combination is "LTR+RTL". The host concatenates the first and second substrings in the order "RTL+LTR" to obtain the concatenated string. Since the concatenated string contains Uyghur corpus data, it can be converted to Arabic corpus data and the operation is reversed to obtain a string in the "LTR+RTL" format.

[0155] Characters are read from the reversed string from left to right and drawn. English characters are drawn from left to right on the background image, while Uyghur characters are drawn from right to left. In this case, the character string in the drawn sample image is exactly the same as the target character string in the combined corpus data.

[0156] See also Figure 15 , is a schematic diagram of a tag determination method provided in an embodiment of the present application, such as Figure 15 As shown, the label is generated for the combined corpus data of "RTL+LTR+RTL". The first corpus data in the combined corpus is the Uyghur corpus After randomly selecting spaces for segmentation, we get two pieces of corpus and The host reads each character from the two corpora in the character reading order corresponding to the RTL text (which is also the character input order), sorts the read characters in the specified order, and assigns character labels to each sorted character, thereby obtaining two label sequences. The second corpus data is the English, numeric, and symbol corpus w.KL178.co. The host reads each character in the character reading order, assigns character labels to each sorted character, and obtains a label sequence. Figure 15The character labels in the label sequence corresponding to the "Get Label" column are independent of each other and are not affected by the order of the text type when reading. The host concatenates the three label sequences in "RTL+LTR+RTL" format to obtain the string labels for the combined corpus data.

[0157] See also Figure 16 , is a schematic diagram of a method for generating a sample image provided by an embodiment of the present application, such as Figure 16 As shown in the figure, the sample image is generated for the combined corpus data with the combination mode of "RTL+LTR+RTL". The first corpus data in the combined corpus is the Uyghur corpus. After randomly selecting spaces for segmentation, we get two pieces of corpus and The second corpus data consists of the English, numeric, and symbol corpus w.KL178.co. The host performs a sequence reversal on this data, resulting in the string oc.871LK.w. The host then concatenates this reversed string with the strings from the two Uyghur corpora in the reverse order of their combination. The Uyghur characters are then converted to Arabic characters and the sequence is further reversed across the entire corpus. A random background image is generated for this combined corpus data and drawn using a random font to produce a sample image.

[0158] In one embodiment, the deep learning model is a combination of MobileNetV3 and BiLSTM (Bi-directional Long Short-Term Memory). MobileNetV3 is used to extract convolutional features from sample images, while BiLSTM is used to further extract sequence features from the convolutional features and output character recognition results.

[0159] The host can perform end-to-end training on a deep learning model composed of MobileNetV3 and BiLSTM based on sample data to generate a text recognition model. CTC loss is used as the loss function during training to address character misalignment. This text recognition model is compact and fast, making it easy to deploy on various platforms (such as mobile phones, tablets, host computers, and webcams).

[0160] This application provides a text recognition method that can be implemented using a text recognition model. The text recognition model can be trained using the aforementioned model training method. The text recognition model can be pre-trained and used directly when executing the text recognition method; alternatively, the text recognition model can be trained online before executing the text recognition method. The text recognition model can recognize images containing bidirectional mixed text.

[0161] The host may obtain an image to be recognized, wherein the image to be recognized includes RTL text and LTR text.

[0162] The host can input the image to be recognized into the text recognition model, thereby obtaining a character recognition result output by the text recognition model. The character recognition result indicates a string of RTL text and a string of LTR text. Exemplarily, if the text recognition model includes a CNN network and an RNN network, the CNN network can extract image features (e.g., feature maps) from the image to be recognized, and the RNN network can determine sequence features from the image features, and then determine the string of RTL text and LTR text based on the sequence features.

[0163] Figure 17 is a block diagram of a training device for a text recognition model according to an embodiment of the present invention. Figure 17 As shown, the device may include:

[0164] An acquisition module 1610 is configured to acquire a plurality of corpus data of a target text; wherein the plurality of corpus data includes first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text;

[0165] A combining module 1620 is configured to combine the first corpus data and the second corpus data according to a plurality of combining modes to obtain combined corpus data corresponding to each combining mode;

[0166] A generating module 1630 is configured to generate a sample image and a string label corresponding to each combined corpus data, and construct sample data based on the sample image and the string label;

[0167] The training module 1640 is used to train the deep learning model according to the sample data to obtain a text recognition model.

[0168] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the training method of the above-mentioned text recognition model, and will not be repeated here.

[0169] In several embodiments provided in this application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented using a dedicated hardware-based system that performs the specified functions or actions, or may be implemented using a combination of dedicated hardware and computer instructions.

[0170] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0171] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A training method for a text recognition model, characterized in that: include: Acquire multiple pieces of corpus data of the target text; wherein the multiple pieces of corpus data include first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text; combining the first corpus data and the second corpus data according to a plurality of combination modes to obtain combined corpus data corresponding to each combination mode; Generate sample images and string labels corresponding to each combined corpus data, and construct sample data based on the sample images and string labels; Training a deep learning model based on the sample data to obtain a text recognition model; The multiple combinations include "LTR+RTL", "RTL+LTR", "RTL+LTR+RTL", and "LTR+RTL+LTR". The step of obtaining multiple pieces of corpus data of the target text includes: Acquire text data from a data source of the target text; Extracting candidate corpus data of the RTL character and candidate corpus data of the LTR character from the character data; The candidate corpus data of the RTL character and the candidate corpus data of the LTR character are respectively cut to obtain first corpus data and second corpus data that meet a character length restriction condition; The step of generating a sample image and a string label corresponding to each combined corpus data includes: For each combined corpus data, determining a character string label corresponding to a target character string in the combined corpus data; For each combined corpus data, drawing the target character string in the combined corpus data onto the background image to obtain a sample image corresponding to the combined corpus data; The step of determining a character string label corresponding to a target character string in the combined corpus data includes: For the first corpus data in the combined corpus data, read a plurality of characters from the first corpus data and sort them in a specified order to determine a first tag sequence; wherein the specified order is opposite to the order in which the characters of the first corpus data are read; For the second corpus data in the combined corpus data, read a plurality of characters from the second corpus data and sort them in the character reading order of the second corpus data to determine a second label sequence; According to the combination mode corresponding to the combined corpus data, the first label sequence and the second label sequence corresponding to the combined corpus data are concatenated to obtain the character string label.

2. The method according to claim 1, characterized in that The determining of the character string label corresponding to the target character string in the combined corpus data further includes: For the second corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the second corpus data to obtain a first substring; splicing the first substring and the second substring of the first corpus data in the combined corpus data according to a splicing order opposite to the splicing order corresponding to the combined corpus data, and performing a sequence reversal operation on the spliced ​​strings; wherein the second substring is a string included in the first corpus data; For the character string after the sequence reversal operation, characters are read one by one in order from left to right and corresponding character labels are determined, and a third label sequence consisting of multiple character labels is used as the character string label.

3. The method according to claim 1, characterized in that Drawing the target character string in the combined corpus data onto the background image to obtain a sample image corresponding to the combined corpus data includes: For the first corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the first corpus data to obtain a third substring; concatenating the third substring with a fourth substring of the second corpus data in the combined corpus data according to a combination mode corresponding to the combined corpus data; wherein the fourth substring is a string included in the second corpus data; A plurality of characters are read from the concatenated character string, and are drawn onto the background image one by one to obtain the sample image.

4. The method according to claim 1, wherein Drawing the target character string in the combined corpus data onto the background image to obtain a sample image corresponding to the combined corpus data includes: For the second corpus data in the combined corpus data, performing a sequence reversal operation on the character strings in the second corpus data to obtain a first substring; splicing the first substring and the second substring of the first corpus data in the combined corpus data according to a splicing order opposite to the splicing order corresponding to the combined corpus data, and performing a sequence reversal operation on the spliced ​​strings; wherein the second substring is a string included in the first corpus data; A plurality of characters are read from the character string after the sequence reversal operation, and are drawn onto the background image one by one to obtain the sample image.

5. The method according to claim 1, 3 or 4, characterized in that: The method further comprises: Before drawing the target character string on the background image, a background image is randomly generated for the combined corpus data.

6. The method according to claim 1, 3 or 4, characterized in that: The first corpus data is Uyghur corpus data, and the method further includes: Before drawing the target character string on the background image, each Uyghur character in the target character string is converted into Arabic characters.

7. The method according to claim 1, characterized in that The deep learning model includes MobileNetV3 and BiLSTM; The deep learning model is trained according to the sample data to obtain a text recognition model, including: The MobileNetV3 and the BiLSTM are trained end-to-end according to the sample data to obtain the text recognition model.

8. A text recognition method, characterized in that: The method comprises: Acquire an image to be recognized, wherein the image to be recognized contains RTL text and LTR text; The image to be recognized is input into a text recognition model to obtain a character recognition result output by the text recognition model; wherein the text recognition model is trained by the method according to any one of claims 1 to 7.

9. A training device for a text recognition model, characterized in that: include: An acquisition module, configured to acquire a plurality of corpus data of the target text; wherein the plurality of corpus data includes first corpus data corresponding to the RTL text and second corpus data corresponding to the LTR text; a combining module, configured to combine the first corpus data and the second corpus data according to a plurality of combining modes, to obtain combined corpus data corresponding to each combining mode; A generation module, configured to generate sample images and string labels corresponding to each combined corpus data, and construct sample data based on the sample images and string labels; A training module, configured to train a deep learning model based on the sample data to obtain a text recognition model; The multiple combinations include "LTR+RTL", "RTL+LTR", "RTL+LTR+RTL", and "LTR+RTL+LTR". The step of obtaining multiple pieces of corpus data of the target text includes: Acquire text data from a data source of the target text; Extracting candidate corpus data of the RTL character and candidate corpus data of the LTR character from the character data; The candidate corpus data of the RTL character and the candidate corpus data of the LTR character are respectively cut to obtain first corpus data and second corpus data that meet a character length restriction condition; The step of generating a sample image and a string label corresponding to each combined corpus data includes: For each combined corpus data, determining a character string label corresponding to a target character string in the combined corpus data; For each combined corpus data, drawing the target character string in the combined corpus data onto the background image to obtain a sample image corresponding to the combined corpus data; The step of determining a character string label corresponding to a target character string in the combined corpus data includes: For the first corpus data in the combined corpus data, read a plurality of characters from the first corpus data and sort them in a specified order to determine a first tag sequence; wherein the specified order is opposite to the order in which the characters of the first corpus data are read; For the second corpus data in the combined corpus data, read a plurality of characters from the second corpus data and sort them in the character reading order of the second corpus data to determine a second label sequence; According to the combination mode corresponding to the combined corpus data, the first label sequence and the second label sequence corresponding to the combined corpus data are concatenated to obtain the character string label.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the text recognition model training method described in any one of claims 1 to 7 or the text recognition method described in claim 8.

11. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which can be executed by a processor to complete the text recognition model training method described in any one of claims 1 to 7 or the text recognition method described in claim 8.

Images