A text recognition method and device, electronic equipment and computer storage medium
By decomposing Chinese characters into radicals and structural information, and constructing radical and structural encoding sequences for similarity matching, the problem of low accuracy in Chinese text recognition is solved, and more efficient Chinese character recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA (CHINA) CO LTD
- Filing Date
- 2023-06-06
- Publication Date
- 2026-05-12
AI Technical Summary
Existing Chinese text recognition methods suffer from low accuracy, especially the Latin-based One-Hot encoding strategy, which performs poorly in Chinese text recognition.
The method involves decomposing Chinese characters into radicals and structural information to construct character encoding sequences. Chinese character recognition is performed by combining radical encoding sequences and structural encoding sequences. The recognition result is determined by matching the similarity between the predicted encoding sequence and the character encoding sequence.
By representing Chinese characters with fine granularity, the accuracy of recognition results is improved, the number of model parameters is reduced, and it is suitable for mobile devices with limited computing resources.
Smart Images
Figure CN116665235B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a text recognition method, apparatus, electronic device, and computer storage medium. Background Technology
[0002] OCR (Optical Character Recognition) refers to the technology of converting text in paper documents into image files using optical methods, and then using recognition software to convert the text in the images into text format for further editing by word processing software. Chinese text recognition is an important research direction in the field of optical character recognition, and it has broad application prospects in document digitization, card recognition, autonomous driving, and other fields.
[0003] Most current Chinese text recognition methods inherit the One-Hot encoding strategy from Latin. Specifically, One-Hot encoding is performed on a certain range of Chinese characters. For a specific Chinese character category, a valid bit is used to represent that category. Then, a deep learning model is used to predict the probability that the sample image belongs to each Chinese character category, and the category with the highest probability is selected as the prediction result.
[0004] The aforementioned Chinese text recognition methods suffer from low recognition accuracy. Summary of the Invention
[0005] In view of this, embodiments of this application provide a text recognition scheme to at least partially solve the above-mentioned problems.
[0006] According to a first aspect of the embodiments of this application, a text recognition method is provided, comprising:
[0007] Obtain the target image containing Chinese characters;
[0008] The target image is input into a pre-trained text recognition model, and the text recognition model outputs a predicted encoding sequence.
[0009] Based on the similarity between the predicted encoding sequence and each character encoding sequence in the character encoding sequence set, the target character encoding sequence is determined from the character encoding sequence set; wherein, one character encoding sequence corresponds to one Chinese character, and the character encoding sequence is composed of a radical encoding sequence and a structure encoding sequence; the radical encoding sequence corresponds to the radical in the Chinese character, and the structure encoding sequence corresponds to the structural information in the Chinese character;
[0010] The Chinese characters corresponding to the target character encoding sequence are identified as the recognition result.
[0011] According to a second aspect of the embodiments of this application, a text recognition device is provided, comprising:
[0012] The target image acquisition module is used to acquire target images containing Chinese characters.
[0013] The encoding prediction module is used to input the target image into a pre-trained text recognition model and output a predicted encoding sequence through the text recognition model;
[0014] The determination module is used to determine the target character encoding sequence from the character encoding sequence set based on the similarity between the predicted encoding sequence and each character encoding sequence in the character encoding sequence set; wherein, one character encoding sequence corresponds to one Chinese character, and the character encoding sequence is composed of a radical encoding sequence and a structure encoding sequence; the radical encoding sequence corresponds to the radical in the Chinese character, and the structure encoding sequence corresponds to the structural information in the Chinese character;
[0015] The recognition result determination module is used to determine the Chinese characters corresponding to the target character encoding sequence as the recognition result.
[0016] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.
[0017] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0018] According to the text recognition scheme provided in this application, a text recognition network is used to predict the encoding of a target image containing Chinese characters, obtaining the predicted encoding sequence corresponding to the Chinese characters in the target image. Then, the predicted encoding sequence is matched with the character encoding sequences corresponding to different Chinese characters to obtain the target character encoding sequence. The Chinese characters corresponding to the target character encoding sequence are then determined as the recognition result. In this application embodiment, when constructing the character encoding sequence corresponding to Chinese characters, the hierarchical information of the Chinese characters is used. The Chinese characters are decomposed into radicals and structural information, and the radicals and structural information are encoded into radical encoding sequences and structural encoding sequences, respectively. The complete character encoding result is obtained by combining the radical encoding sequences and structural encoding sequences: the character encoding sequence. Therefore, the character encoding sequence constructed in the above manner can more effectively and finely represent Chinese characters. Furthermore, similarity matching and text recognition based on the above character encoding sequence can effectively improve the accuracy of the recognition result. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0020] Figure 1 This is a flowchart illustrating the steps of a text recognition method according to Embodiment 1 of this application;
[0021] Figure 2 A diagram illustrating the hierarchical decomposition of characters;
[0022] Figure 3 for Figure 1 The overall system block diagram of the embodiment shown;
[0023] Figure 4 for Figure 1 The illustrated embodiment is a schematic diagram of the text recognition process.
[0024] Figure 5 A schematic diagram illustrating the process of obtaining a character encoding sequence;
[0025] Figure 6 This is a structural block diagram of a text recognition device according to Embodiment 2 of this application;
[0026] Figure 7 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this application. Detailed Implementation
[0027] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.
[0028] Example 1
[0029] Reference Figure 1 , Figure 1 This is a flowchart illustrating the steps of a text recognition method according to Embodiment 1 of this application. Specifically, the text recognition method provided in this embodiment includes the following steps:
[0030] Step 102: Obtain the target image containing Chinese characters.
[0031] The target image is the image to be recognized as Chinese characters. The target image contains Chinese characters to be recognized. In this embodiment of the application, the number of Chinese characters contained in the target image is not limited. For example, it can be a single Chinese character or a text line composed of multiple Chinese characters.
[0032] Step 104: Input the target image into the pre-trained text recognition model, and output the predicted encoding sequence through the text recognition model.
[0033] In this embodiment, the specific structure of the text recognition model is not limited. It can be any machine learning model that can extract features from the target image and encode the extracted features to output a predicted encoded sequence.
[0034] The training process of the text recognition model in this application embodiment may include:
[0035] Obtain the sample image and sample label for recognition; the sample image contains a text line composed of multiple Chinese characters;
[0036] Input the sample image into the initial text recognition model, and the initial text recognition model outputs the sample encoding sequence;
[0037] Based on the similarity between each character encoding sequence in the sample encoding sequence and character encoding sequence set, and the identification sample label, a time-series classification CTC loss function is constructed, and the loss value is calculated based on the CTC loss function;
[0038] The initial text recognition model is trained based on the loss value, resulting in a fully trained text recognition model.
[0039] The aforementioned identification sample label is composed of the character encoding sequence corresponding to each Chinese character in the aforementioned text line.
[0040] Specifically, when calculating the similarity between the sample encoded sequence and each character encoded sequence in the character encoded sequence set, the inner product of the sample encoded sequence and the matrix composed of each character encoded sequence in the character encoded sequence set can be used as the aforementioned similarity.
[0041] Furthermore, the loss value can be calculated using the following formula:
[0042]
[0043] d(C i ,)= i ·
[0044] Where L is the loss value, and k is the number of Chinese characters contained in the sample image; l iTo identify the character encoding sequence corresponding to the i-th Chinese character in the sample image; C i To identify the sample encoding sequence corresponding to the i-th Chinese character in the sample image; B is a matrix composed of the character encoding sequences in the character encoding sequence set; d(C i ,) represents the characterization of C i The inner product of the similarities between the character encoding sequences in B and the character encoding sequences in B; p(l i |d(C i ,)) represents the relationship between B and C i The character encoding sequence with the highest similarity is l i The probability of.
[0045] In this embodiment of the application, the sample image contains a text line composed of multiple Chinese characters. During the training of the text recognition model, a CTC loss function is constructed based on the similarity between the sample encoding sequence and each character encoding sequence in the character encoding sequence set. Then, the loss value is obtained based on the constructed CTC loss function for model training. Since the CTC loss function can align sequences and labels of different lengths, the text recognition model obtained by the above model training method provided in this embodiment of the application can recognize both single characters and text lines in an image.
[0046] Step 106: Based on the similarity between the predicted encoding sequence and each character encoding sequence in the character encoding sequence set, determine the target character encoding sequence from the character encoding sequence set; one character encoding sequence corresponds to one Chinese character.
[0047] The character encoding sequence is composed of a radical encoding sequence and a structure encoding sequence; the radical encoding sequence corresponds to the radical in Chinese characters, and the structure encoding sequence corresponds to the structural information in Chinese characters.
[0048] In this embodiment of the application, the predicted encoding sequence and each character encoding sequence in the character encoding sequence set have the same length, and this length value can be set in advance according to the actual situation.
[0049] Optionally, in some embodiments, both the predicted encoding sequence and the character encoding sequence are binary encoding sequences containing multiple 1s; the length of both the predicted encoding sequence and the character encoding sequence is less than the total number of Chinese characters corresponding to the character encoding sequence set. That is, multi-hot encoding can be used to encode each character in a preset Chinese character set to obtain the character encoding sequence corresponding to each Chinese character.
[0050] In the embodiments of the present application, a radical refers to the component obtained after splitting (decomposing) Chinese characters according to a preset splitting rule (or hierarchical order). A radical is the basic unit that composes Chinese characters. The radical in the embodiments of the present application has a wider range compared to the concept of the radical and stroke of Chinese characters. That is to say, the radical in the embodiments of the present application can include the radical and stroke, and can also include some relatively simple strokes outside the radical and stroke, such as: the left-falling stroke, the right-falling stroke, etc. Structure information refers to the relative position relationship between the radicals in Chinese characters. The above splitting rule can be set according to the structural characteristics of specific Chinese characters and actual situations. For example: the radical splitting can be performed in the order of from left to right first, and then from top to bottom; it can also be performed in the order of from top to bottom first, and then from left to right; it can also be performed in the order of from the inside to the outside, etc. The above splitting rule can also be obtained according to an existing or publicly available Chinese character splitting sequence table. For each Chinese character in this sequence table, there is a corresponding splitting rule. The above splitting rule can also be obtained by taking certain optimization measures based on the existing Chinese character splitting sequence table. For example: refer to Figure 2 , Figure 2 the schematic diagram of the hierarchical decomposition information of characters. For the Chinese character "zhi", performing hierarchical decomposition in the order of from left to right first, and then from top to bottom, can obtain Figure 2 the binary tree composed of radicals and structure information shown in the figure. Based on the above binary tree, the splitting result can be obtained as: "wood", "mouth" and "eight". Among them, represents "left-right structure"; represents "up-down structure". Therefore, the radicals of the Chinese character "zhi" include: "wood", "mouth" and "eight", and the structure information includes: "left-right structure" and "up-down structure".
[0051] For a character coding sequence, the radical coding sequence therein can include at least one radical coding subsequence, and one radical coding subsequence corresponds to one radical; the structure coding sequence can also include at least one structure coding subsequence, and one structure coding subsequence corresponds to one structure information.
[0052] Among them, the length of the radical coding subsequence can be a preset fixed value, and the length of the structure coding subsequence can also be a preset fixed value. In the embodiments of the present application, the specific values of the length of the radical coding subsequence and the length of the structure coding subsequence are not limited and can be set according to actual situations. As mentioned above, the length of the coding sequence determines the number of model parameters. Therefore, to reduce the model parameters, on the premise of ensuring that different radicals (or structure information) correspond to different radical coding subsequences (structure coding subsequences), the length of the radical coding subsequence (structure coding subsequence) can be reduced as much as possible.
[0053] Optionally, in some embodiments of this application, the radical encoding subsequence is a binary encoding sequence containing multiple 1s, and the length of the radical encoding subsequence is less than the total number of radicals; the structure encoding subsequence is also a binary encoding sequence containing multiple 1s, and the length of the structure encoding subsequence is less than the total number of structural information. In the embodiments of this application, the specific encoding methods for the radical encoding subsequence and the structure encoding subsequence are not limited. For example, they can be obtained by manual setting or by using a preset encoding model, etc.
[0054] In addition, since the total number of existing structural information is relatively small (about ten types), while the total number of radicals is relatively large, a structural encoding subsequence can be represented by fewer binary bits, such as 4 bits; while a radical encoding subsequence can be represented by more binary bits, such as 36 bits.
[0055] Typically, a Chinese character can be decomposed into a binary tree composed of radicals and structural information with a depth of no more than 5 levels. Furthermore, a Chinese character usually contains no more than 9 radicals. Therefore, the length of the character encoding sequence can be set according to the number of radicals and structural information corresponding to a 5-level binary tree. Specifically: a 5-level binary tree contains 15 pieces of structural information; assuming one structural encoding subsequence is 4 bits, the structural encoding sequence occupies a total of 4 × 15 = 60 bits. A Chinese character can contain a maximum of 9 radicals; assuming one radical encoding subsequence is 36 bits, the radical encoding sequence occupies a total of 9 × 36 = 324 bits. Therefore, the length of a character encoding sequence can be set to 60 + 324 = 384 bits. For a specific Chinese character, if the number of structural information is less than 15, the corresponding structural encoding sequence can be padded with zeros at the end to make it 60 binary bits. Similarly, if the number of radicals is less than 9, the corresponding radical encoding sequence can be padded with zeros at the end to make it 324 binary bits, so that the length of the final character encoding sequence is 384 binary bits.
[0056] Text recognition models typically consist of a feature extractor and a classifier. The feature extractor extracts features, while the classifier maps the extracted features to predictive codes. The number of nodes in the classifier equals the product of the number of input features and the number of output features. Therefore, in existing text recognition models based on One-Hot encoding strategies, where the feature extractor remains constant (and the number of input features remains constant), the number of nodes in the classifier equals the product of the number of input features and the total number of Chinese characters in the character set (usually tens of thousands). In this embodiment, the number of nodes in the classifier equals the product of the number of input features and the length of the character encoding sequence. Since the character encoding sequence is obtained using multi-hot encoding, its length (e.g., 384, etc.) is less than the total number of Chinese characters in the character set. Therefore, this embodiment compresses the number of nodes in the classifier, thereby effectively reducing the number of parameters in the text recognition model and achieving a lightweight design. This allows the model to be deployed on mobile devices with limited computing resources for Chinese character recognition.
[0057] Furthermore, in this embodiment, the specific calculation method for calculating the similarity between the predicted coding sequence and the character coding sequence is not limited. It can be set according to the actual situation. For example, the similarity between the predicted coding sequence and the character coding sequence can be represented by the Euclidean distance, the Manhattan distance, or the Chebyshev distance, etc.
[0058] The target character encoding sequence can be selected from the set of character encoding sequences, based on the highest similarity between the target and predicted character encoding sequences.
[0059] Step 108: Determine the Chinese characters corresponding to the target character encoding sequence as the recognition result.
[0060] Since each character encoding sequence corresponds to a Chinese character in the Chinese character set, after determining the target character encoding sequence from the character encoding sequence set, the Chinese character corresponding to the target character encoding sequence can be determined as the recognition result of text recognition of the target image.
[0061] The recognition result in this application embodiment can be a single Chinese character or a text line composed of multiple Chinese characters.
[0062] See Figure 3 , Figure 3 for Figure 1 The overall system block diagram of the embodiment shown is illustrated below. Figure 3 The steps of the text recognition method provided in the embodiments of this application will be explained:
[0063] After acquiring the target image containing Chinese characters, the target image can be input into a text recognition model. The text recognition model performs encoding prediction to obtain the predicted encoding sequence. Simultaneously, the character encoding sequence corresponding to each Chinese character in the Chinese character set, composed of a radical encoding sequence and a structural encoding sequence, can be pre-determined. Figure 3 The character encoding table in the table is used; then the similarity between the predicted encoding sequence and the aforementioned predetermined character encoding sequences is calculated, and the target character encoding sequence is determined based on the calculation results. Finally, the Chinese characters corresponding to the target character encoding sequence are determined as the recognition results.
[0064] According to the text recognition method provided in this application, a text recognition network is used to predict the encoding of a target image containing Chinese characters, obtaining the predicted encoding sequence corresponding to the Chinese characters in the target image. Then, the predicted encoding sequence is matched with the character encoding sequences corresponding to different Chinese characters to obtain the target character encoding sequence. The Chinese characters corresponding to the target character encoding sequence are then determined as the recognition result. In this application embodiment, when constructing the character encoding sequence corresponding to Chinese characters, the hierarchical information of the Chinese characters is used. The Chinese characters are decomposed into radicals and structural information, and the radicals and structural information are encoded into radical encoding sequences and structural encoding sequences, respectively. The complete character encoding result, the character encoding sequence, is obtained by combining the radical encoding sequences and structural encoding sequences. Therefore, the character encoding sequence constructed in the above manner can more effectively and finely represent Chinese characters. Furthermore, similarity matching and text recognition based on the above character encoding sequence can effectively improve the accuracy of the recognition result.
[0065] Optionally, in some embodiments, the text recognition model in Embodiment 1 above may include: a feature extractor and a classifier; the classifier includes a linear layer and an activation function layer. Step 104 above may include:
[0066] The target image is input into the feature extractor, which extracts features from the target image to obtain an initial feature sequence, which is then input into the classifier.
[0067] The initial feature sequence is transformed into an aligned feature sequence through a linear layer in the classifier, and the aligned feature sequence is then activated by an activation function layer to obtain the predicted encoding sequence; the length of the aligned feature sequence is equal to the length of the character encoding sequence.
[0068] Specifically, the aforementioned initial feature sequence can characterize the global features contained in the target image, that is, the overall features of the Chinese characters contained in the target image.
[0069] The activation function layer in this embodiment differs from the conventional activation function layer in the classifier of a text recognition model based on one-hot encoding. The conventional activation function layer typically uses a single-point activation function, such as the softmax activation function. This means that during feature activation, the conventional activation function layer can only activate one binary bit in the initial feature sequence to be 1, while the remaining binary bits can only be 0. In contrast, the activation function layer in this embodiment uses a multi-point activation function. This means that during feature activation, each binary bit in the initial feature sequence can be activated independently, and each binary bit may be activated to be 1. Furthermore, in this embodiment, the activation function layer can use the tanh function.
[0070] In this embodiment of the application, the specific process of obtaining the predicted coding sequence by activating the aligned feature sequence through the activation function layer can be as follows: the activation function layer maps each value in the aligned feature sequence to the interval [-1, 1], and the mapped values are binarized to convert them to -1 or 1. Then, the element values of the elements with a value of -1 are updated to convert the element values from -1 to 0, thereby obtaining a binary predicted coding sequence containing multiple 1s.
[0071] In this embodiment, the selection of the backbone network (feature extractor) in the text recognition model is not restricted, which means that this embodiment can be embedded into the existing text recognition model by replacing the classifier and the encoding method of the character encoding sequence, thus providing greater flexibility.
[0072] See figure. Figure 4 for Figure 1 The illustrated embodiment is a schematic diagram of the text recognition process. Specifically: After obtaining a target image containing Chinese characters, the target image can be input into a text recognition model. The feature extractor in the text recognition model extracts features from the target image to obtain an initial feature sequence. The initial feature sequence is then converted into an aligned feature sequence through a linear layer in the classifier. The aligned feature sequence is then activated through an activation function layer to obtain a predicted encoding sequence. Finally, the target character encoding sequence is determined based on the similarity between the predicted encoding sequence and the character encoding sequences in the character encoding table, thereby obtaining the recognition result.
[0073] As mentioned in the explanation of step 106 above, the specific encoding methods for radical encoding subsequences and structure encoding subsequences can be obtained by manual setting or by using a preset encoding model.
[0074] For radical encoding subsequences, if they are obtained manually, different radical encoding subsequences may lack correlation, and the interpretability of each radical encoding subsequence may be lacking. Therefore, optionally, in some embodiments of this application, a preset encoding model can be used to obtain radical encoding subsequences. This makes the final radical encoding subsequences more interpretable, that is, radicals with similar shapes will have more similar radical encoding subsequences. Specifically, the process of obtaining radical encoding subsequences may include:
[0075] Construct an initial prototype matrix, which consists of an initial radical encoding subsequence and an initial structure encoding subsequence;
[0076] Obtain encoded sample images and encoded sample labels containing a single Chinese character; the encoded sample labels include the one-hot encoded sequence corresponding to the radical of the single Chinese character and the one-hot encoded sequence corresponding to the structural information contained in the single Chinese character;
[0077] Input the encoded sample image into the initial radical encoding model to obtain multiple local feature vectors; each local feature vector corresponds to a radical or structural information of a single Chinese character; the dimension of the local feature vector is the same as the dimension of the initial radical encoding subsequence;
[0078] Similarity is calculated for multiple local feature vectors and the initial prototype matrix, and the loss value is calculated based on the similarity calculation results and the encoded sample labels. The initial radical encoding model is trained based on the loss value and the initial prototype matrix is updated to obtain the trained radical encoding model and the updated prototype matrix containing radical encoding subsequences.
[0079] The above method can simultaneously obtain the updated radical encoding subsequence and the structure code subsequence. In practical applications, the updated radical encoding subsequence and structure code subsequence can be used as the encoding sequence corresponding to the radical and the encoding sequence corresponding to the structure information, respectively. Furthermore, since the number of types of structure information is relatively small (approximately 10), in order to further reduce the length of the encoding sequence, manual encoding can be used for structure information: using fewer binary bits to represent a piece of structure information, such as using 4 bits.
[0080] Specifically, the number of initial radical encoding subsequences contained in the initial prototype matrix can be the total number of radical categories contained in the existing Chinese character set (such as the set of characters included in GB18030-2005, which includes approximately 70,244 characters), and one initial radical encoding subsequence can correspond to one radical category; the number of initial structure encoding subsequences contained in the initial prototype matrix can be the total number of structure information categories contained in the existing Chinese character set, and one initial structure encoding subsequence can correspond to one structure information category.
[0081] According to statistics, the existing Chinese character set contains approximately 700 radical categories and structural information categories in total. Assuming that the existing Chinese character set contains 700 radical categories and structural information categories in total, then the length of the one-hot encoding sequence corresponding to each radical is 700, with only one binary bit being 1 and the rest being 0. Similarly, the length of the one-hot encoding sequence corresponding to each structural information category is also 700, with only one binary bit being 1 and the rest being 0.
[0082] In this embodiment of the application, when calculating the loss value based on the similarity calculation result and the encoded sample label, specifically, the cross-entropy loss can be calculated using the similarity calculation result and the encoded sample label, and then the calculation result can be determined as the loss value.
[0083] Furthermore, the radical encoding model may include a feature extraction module and an attention module; inputting the encoded sample image into the initial radical encoding model, and outputting multiple local feature vectors through the radical encoding model, may include:
[0084] The encoded sample image is input into the radical encoding model, and the global feature vector is obtained by the feature extraction module. The global feature vector is then decoded into multiple local feature vectors by the attention module.
[0085] In addition, after obtaining the global feature vector, the attention module can perform two independent decoding operations on the global feature vector, outputting two types of feature vectors: the aforementioned local feature vector and a candidate local feature vector. The dimension of the candidate local feature vector is equal to the length of the one-hot encoding sequence corresponding to the radical, and the number of candidate local feature vectors is the same as the number of the aforementioned local feature vectors. Both are equal to the total number of radicals and structural information contained in a single character in the encoded sample image, and one candidate local feature vector corresponds to one radical or one piece of structural information contained in a single character in the encoded sample image.
[0086] After outputting the two feature vectors mentioned above through the attention module, the loss value for model training can be calculated as follows: The similarity between the local feature vectors and the initial prototype matrix is calculated, and a cross-entropy loss is calculated based on the similarity calculation result and the encoded sample labels, serving as the first loss value; the cross-entropy loss is calculated based on the aforementioned candidate local feature vectors and encoded sample labels, serving as the second loss value; the similarity between the aforementioned local feature vectors and the radical encoding subsequences and structural encoding subsequences corresponding to the radical and structural information contained in a single character of the encoded sample image in the prototype matrix is calculated, and a third loss value is obtained based on the aforementioned similarity; finally, the first, second, and third loss values are fused to obtain the loss value used for model training. Using this method to obtain the loss value and training the radical encoding model based on this loss value can improve the training accuracy of the radical encoding model.
[0087] In this embodiment, when constructing the character encoding sequence corresponding to Chinese characters, the Chinese characters are decomposed into radicals and structural information, and then the structural encoding sub-sequences corresponding to the structural information are constructed respectively. The radical encoding sub-sequences corresponding to the radicals are determined through the radical encoding model. Therefore, when determining the radical encoding sub-sequences through the radical encoding model, it is not necessary to use all characters in the Chinese character set as sample images. It is only necessary to ensure that the radicals in the character set appear in the sample images. In this way, in the application stage, even if a certain Chinese character is not used as a sample image in the training of the radical encoding model and the determination process of the radical encoding sub-sequence, the text recognition method of this embodiment can still recognize the Chinese character. That is, this embodiment can perform zero-sample recognition.
[0088] Furthermore, the process of obtaining the character encoding sequence may include:
[0089] Determine the hierarchical decomposition information of the target character; the hierarchical decomposition information includes: the target radical contained in the target character, the target structural information contained in the target character, and the combination relationship between the target radical and the target structural information;
[0090] Obtain the target radical encoding subsequence corresponding to the target radical and the target structure encoding subsequence corresponding to the target structure information, respectively;
[0091] According to the combination relationship, the target radical encoding subsequence is combined into the target radical encoding sequence, and the target structure encoding subsequence is combined into the target structure encoding sequence;
[0092] By concatenating the target structure encoding sequence and the target radical encoding sequence, the character encoding sequence corresponding to the target character is obtained.
[0093] Specifically, the target character can be any character in the existing character set.
[0094] The hierarchical decomposition information can be presented in the form of a binary tree as follows Figure 2 shown
[0095] When combining the target radical encoding subsequences into a target radical encoding sequence, the combination can be performed in the order in which each target radical appears in the binary tree. Among them, the target radical encoding subsequence corresponding to the target radical that appears first is in the front, and the target radical encoding subsequence corresponding to the target radical that appears later is in the back. For example, for Figure 2 the character "枳" in, the order of the target radicals is: "wood", "mouth", "eight". Therefore, when combining the target radical encoding sequence, the radical encoding subsequence corresponding to "wood" can be placed in the front, the radical encoding subsequence corresponding to "mouth" in the middle, and the radical encoding subsequence corresponding to "eight" at the back. Correspondingly, when combining the target structure encoding subsequences into a target structure encoding sequence, it can also be combined in the order in which each target structure information appears in the binary tree. Among them, the one that appears first is in the front, and the one that appears later is in the back. This will not be elaborated here
[0096] Refer to Figure 5 , Figure 5 which is a schematic diagram of the process of obtaining the character encoding sequence. The following will elaborate on the process of obtaining the character encoding sequence in combination with Figure 5 as follows
[0097] Input the encoded sample image into the radical encoding model. Through the feature extraction module and the attention module in the radical encoding model, obtain the radical encoding subsequences corresponding to each radical. For example, the radical encoding subsequence corresponding to "mouth" is 0010101101… (a total of 36 bits), the radical encoding subsequence corresponding to "eight" is 1100001011…, the radical encoding subsequence corresponding to "person" is 1110101100…, the radical encoding subsequence corresponding to "氵" is 0111100101…, the radical encoding subsequence corresponding to "扌" is 1010011110…, and so on; at the same time, encode the structure information through manual setting to obtain the structure encoding subsequences corresponding to each structure information. For example the corresponding structure encoding subsequence is: 0110 The corresponding structural coding subsequences are: 1001, etc.; then encode and combine the radical coding subsequences and the structural coding subsequences to obtain the character coding sequences corresponding to each Chinese character. For example, the character coding sequence corresponding to the Chinese character "啊" is: 0110…0010101101…(a total of 384 bits), the character coding sequence corresponding to the Chinese character "把" is: 0110…1010011110…, the character coding sequence corresponding to the Chinese character "彩" is: 0110…0101101011…, the character coding sequence corresponding to the Chinese character "点" is: 11001…1011101101…, the character coding sequence corresponding to the Chinese character "沧" is: 0110…0011010010…, the character coding sequence corresponding to the Chinese character "繁" is: 1001…1101001011…, etc.
[0098] Embodiment 2
[0099] Figure 6 It is a structural block diagram of a text recognition device according to Embodiment 2 of the present application. The text recognition device provided by the embodiment of the present application includes:
[0100] A target image acquisition module 602, configured to acquire a target image containing Chinese characters;
[0101] An encoding prediction module 604, configured to input the target image into a pre-trained text recognition model, and output a predicted coding sequence through the text recognition model;
[0102] A determination module 606, configured to determine a target character coding sequence from the set of character coding sequences based on the similarity between the predicted coding sequence and each character coding sequence in the set of character coding sequences; wherein, one character coding sequence corresponds to one Chinese character, and the character coding sequence is composed of a radical coding sequence and a structural coding sequence; the radical coding sequence corresponds to the radical in the Chinese character, and the structural coding sequence corresponds to the structural information in the Chinese character;
[0103] An identification result determination module 608, configured to determine the Chinese character corresponding to the target character coding sequence as the identification result.
[0104] Optionally, in some embodiments, both the predicted coding sequence and the character coding sequence are binary coding sequences containing multiple 1s; the length of the predicted coding sequence and the length of the character coding sequence are both less than the total number of Chinese characters corresponding to the set of character coding sequences.
[0105] Optionally, in some embodiments, the radical coding sequence includes at least one radical coding subsequence, one radical coding subsequence corresponds to one radical, the radical coding subsequence is a binary coding sequence containing multiple 1s, and the length of the radical coding subsequence is less than the total number of radicals;
[0106] The structure coding sequence includes at least one structure coding subsequence, and one structure coding subsequence corresponds to one structure information. The structure coding subsequence is a binary coding sequence containing multiple 1s, and the length of the structure coding subsequence is less than the total number of structure information.
[0107] Optionally, in some embodiments, the text recognition device further includes:
[0108] The radical encoding subsequence acquisition module is used for:
[0109] Construct an initial prototype matrix, which consists of an initial radical encoding subsequence and an initial structure encoding subsequence;
[0110] Obtain encoded sample images and encoded sample labels containing a single Chinese character; the encoded sample labels include the one-hot encoded sequence corresponding to the radical of the single Chinese character and the one-hot encoded sequence corresponding to the structural information contained in the single Chinese character;
[0111] Input the encoded sample image into the initial radical encoding model to obtain multiple local feature vectors; each local feature vector corresponds to a radical or structural information of a single Chinese character; the dimension of the local feature vector is the same as the dimension of the initial radical encoding subsequence;
[0112] Similarity is calculated for multiple local feature vectors and the initial prototype matrix, and the loss value is calculated based on the similarity calculation results and the encoded sample labels. The initial radical encoding model is trained based on the loss value and the initial prototype matrix is updated to obtain the trained radical encoding model and the updated prototype matrix containing radical encoding subsequences.
[0113] Optionally, in some embodiments, the text recognition device further includes:
[0114] The character encoding sequence acquisition module is used to determine the hierarchical decomposition information of the target character; the hierarchical decomposition information includes: the target radical contained in the target character, the target structural information contained in the target character, and the combination relationship between the target radical and the target structural information;
[0115] Obtain the target radical encoding subsequence corresponding to the target radical and the target structure encoding subsequence corresponding to the target structure information, respectively;
[0116] According to the combination relationship, the target radical encoding subsequence is combined into the target radical encoding sequence, and the target structure encoding subsequence is combined into the target structure encoding sequence;
[0117] By concatenating the target structure encoding sequence and the target radical encoding sequence, the character encoding sequence corresponding to the target character is obtained.
[0118] Optionally, in some embodiments, the text recognition model includes: a feature extractor and a classifier; the classifier includes a linear layer and an activation function layer; the encoding prediction module 604 is specifically used for:
[0119] The target image is input into the feature extractor, which extracts features from the target image to obtain an initial feature sequence, which is then input into the classifier.
[0120] The initial feature sequence is transformed into an aligned feature sequence through a linear layer in the classifier, and the aligned feature sequence is then activated by an activation function layer to obtain the predicted encoding sequence; the length of the aligned feature sequence is equal to the length of the character encoding sequence.
[0121] Optionally, in some embodiments, the text recognition device further includes:
[0122] The text recognition model training module is used for:
[0123] Obtain the sample image and sample label for recognition; the sample image contains a text line composed of multiple Chinese characters;
[0124] Input the sample image into the initial text recognition model, and the initial text recognition model outputs the sample encoding sequence;
[0125] Based on the similarity between each character encoding sequence in the sample encoding sequence and character encoding sequence set, and the identification sample label, a time-series classification CTC loss function is constructed, and the loss value is calculated based on the CTC loss function;
[0126] The initial text recognition model is trained based on the loss value, resulting in a fully trained text recognition model.
[0127] Optionally, in some embodiments, when the text recognition model training module performs the steps of calculating the similarity between each character encoding sequence in the sample encoding sequence and character encoding sequence set, constructing a temporal classification CTC loss function based on the identified sample labels, and calculating the loss value based on the CTC loss function, it is specifically used for:
[0128] The loss value is obtained through the loss value calculation formula, which is:
[0129]
[0130] d(C i ,)= i ·
[0131] Where L is the loss value, and k is the number of Chinese characters contained in the sample image; l i To identify the character encoding sequence corresponding to the i-th Chinese character in the sample image; Ci To identify the sample encoding sequence corresponding to the i-th Chinese character in the sample image; B is a matrix composed of the character encoding sequences in the character encoding sequence set; d(C i ,) represents the characterization of C i The inner product of the similarities between the character encoding sequences in B and the character encoding sequences in B; p(l i |d(C i ,)) represents the relationship between B and C i The character encoding sequence with the highest similarity is l i The probability of.
[0132] The text recognition device of this embodiment is used to implement the corresponding text recognition methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the text recognition device of this embodiment can refer to the description of the corresponding parts in the foregoing method embodiments, which will also not be repeated here.
[0133] Example 3
[0134] Reference Figure 7 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.
[0135] like Figure 7 As shown, the electronic device may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.
[0136] in:
[0137] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.
[0138] Communication interface 704 is used to communicate with other electronic devices or servers.
[0139] The processor 702 is used to execute program 710, specifically to perform the relevant steps in the above-described text recognition method embodiment.
[0140] Specifically, program 710 may include program code that includes computer operation instructions.
[0141] The processor 702 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.
[0142] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0143] Program 710 may include multiple computer instructions. Specifically, program 710 may use multiple computer instructions to cause processor 702 to perform the operation corresponding to the text recognition method described in any of the foregoing multiple method embodiments.
[0144] The specific implementation of each step in program 710 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.
[0145] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.
[0146] This application also provides a computer program product, including computer instructions that instruct a computing device to perform the operation corresponding to the text recognition method in the above method embodiments.
[0147] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0148] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.
[0149] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.
[0150] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0151] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.
Claims
1. A text recognition method, comprising: Obtain the target image containing Chinese characters; The target image is input into a pre-trained text recognition model, and the text recognition model outputs a predicted encoding sequence. Based on the similarity between the predicted encoding sequence and each character encoding sequence in the character encoding sequence set, a target character encoding sequence is determined from the character encoding sequence set; wherein, one character encoding sequence corresponds to one Chinese character, and the character encoding sequence is composed of a radical encoding sequence and a structure encoding sequence; the radical encoding sequence corresponds to the radical in the Chinese character, and the structure encoding sequence corresponds to the structural information in the Chinese character, the radical encoding sequence includes at least one radical encoding subsequence, one radical encoding subsequence corresponds to one radical, the radical encoding subsequence is a binary encoding sequence containing multiple 1s, and the length of the radical encoding subsequence is less than the total number of radicals; The Chinese characters corresponding to the target character encoding sequence are identified as the recognition result.
2. The method according to claim 1, wherein, Both the predicted encoding sequence and the character encoding sequence are binary encoding sequences containing multiple 1s; the length of the predicted encoding sequence and the length of the character encoding sequence are both less than the total number of Chinese characters corresponding to the character encoding sequence set.
3. The method according to claim 1 or 2, wherein, The structure encoding sequence includes at least one structure encoding subsequence, and one structure encoding subsequence corresponds to one structure information; the structure encoding subsequence is a binary encoding sequence containing multiple 1s, and the length of the structure encoding subsequence is less than the total number of structure information.
4. The method according to claim 3, wherein, The process of obtaining the radical encoding subsequence includes: Construct an initial prototype matrix, which consists of an initial radical encoding subsequence and an initial structure encoding subsequence; Obtain encoded sample images and encoded sample labels containing a single Chinese character; the encoded sample labels include the one-hot encoded sequence corresponding to the radical of the single Chinese character and the one-hot encoded sequence corresponding to the structural information contained in the single Chinese character; The encoded sample image is input into the initial radical encoding model to obtain multiple local feature vectors; each local feature vector corresponds to a radical or structural information of a single Chinese character; the dimension of the local feature vector is the same as the dimension of the initial radical encoding subsequence. Similarity is calculated between the multiple local feature vectors and the initial prototype matrix, and a loss value is calculated based on the similarity calculation result and the encoded sample label; the initial radical encoding model is trained based on the loss value and the initial prototype matrix is updated to obtain the trained radical encoding model and the updated prototype matrix containing radical encoding subsequences.
5. The method according to claim 3, wherein, The process of obtaining the character encoding sequence includes: Determine the hierarchical decomposition information of the target character; the hierarchical decomposition information includes: the target radical contained in the target character, the target structural information contained in the target character, and the combination relationship between the target radical and the target structural information; Obtain the target radical encoding subsequence corresponding to the target radical and the target structure encoding subsequence corresponding to the target structure information, respectively; According to the combination relationship, the target radical encoding subsequence is combined into a target radical encoding sequence, and the target structure encoding subsequence is combined into a target structure encoding sequence; By concatenating the target structure encoding sequence and the target radical encoding sequence, the character encoding sequence corresponding to the target character is obtained.
6. The method according to claim 2, wherein, The text recognition model includes a feature extractor and a classifier; the classifier includes a linear layer and an activation function layer. The step of inputting the target image into a pre-trained text recognition model and outputting a predicted encoding sequence through the text recognition model includes: The target image is input into the feature extractor, which extracts features from the target image to obtain an initial feature sequence, and then the initial feature sequence is input into the classifier. The initial feature sequence is converted into an aligned feature sequence through a linear layer in the classifier, and the aligned feature sequence is activated by the activation function layer to obtain a predicted encoding sequence; the length of the aligned feature sequence is equal to the length of the character encoding sequence.
7. The method according to claim 1 or 2, wherein, The training process of the text recognition model includes: Obtain a sample image and a sample label for recognition; the sample image contains a text line composed of multiple Chinese characters; The recognition sample image is input into the initial text recognition model, and the initial text recognition model outputs a sample encoding sequence. Based on the similarity between the sample encoding sequence and each character encoding sequence in the character encoding sequence set, and the identification sample label, a time-series classification CTC loss function is constructed, and the loss value is calculated based on the CTC loss function; The initial text recognition model is trained based on the loss value to obtain the trained text recognition model.
8. The method according to claim 7, wherein, The step of constructing a temporal classification CTC loss function based on the similarity between the sample encoding sequence and each character encoding sequence in the character encoding sequence set, and the identified sample label, and calculating the loss value based on the CTC loss function, includes: The loss value is obtained through the loss value calculation formula, which is: in, This is the loss value. To identify the number of Chinese characters contained in the sample image; To identify the character encoding sequence corresponding to the i-th Chinese character in the sample image; To identify the sample encoding sequence corresponding to the i-th Chinese character in the sample image; It is a matrix composed of the character encoding sequences in the character encoding sequence set; To characterize and The inner product of the similarities between the character encoding sequences in the code; Characterization Zhongyu The character encoding sequence with the highest similarity is The probability of.
9. A text recognition device, comprising: The target image acquisition module is used to acquire target images containing Chinese characters. The encoding prediction module is used to input the target image into a pre-trained text recognition model and output a predicted encoding sequence through the text recognition model; The determining module is used to determine a target character encoding sequence from the character encoding sequence set based on the similarity between the predicted encoding sequence and each character encoding sequence in the character encoding sequence set; wherein, one character encoding sequence corresponds to one Chinese character, and the character encoding sequence is composed of a radical encoding sequence and a structure encoding sequence; the radical encoding sequence corresponds to the radical in the Chinese character, the structure encoding sequence corresponds to the structural information in the Chinese character, the radical encoding sequence includes at least one radical encoding subsequence, one radical encoding subsequence corresponds to one radical, the radical encoding subsequence is a binary encoding sequence containing multiple 1s, and the length of the radical encoding subsequence is less than the total number of radicals; The recognition result determination module is used to determine the Chinese characters corresponding to the target character encoding sequence as the recognition result.
10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-8.
11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-8.
12. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-8.