Decoding method, text recognition method and device

Abstract

The present disclosure relates to a decoding method and a text recognition method and device, which are used for decoding a classification result obtained by processing an input by an end-to-end model, the classification result being used to represent probabilities of different characters in a character solution space corresponding to characters at different time steps in the input. The decoding method comprises, for at least one time step, iteratively performing the following steps: popping a top element from a large root heap; for any one character prefix and character combination corresponding to the top element, splicing a second character prefix in the character prefix and the character combination with a second character, splicing a third character prefix with a first character in the character prefix and the character combination, and adding the spliced character prefix as a heap element to the large root heap. Thus, compared with CTC Prefix Beam Search, the time complexity of decoding for a single time step can be reduced from O(N 2 ) to O(N*log(N)).

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a decoding method, a text recognition method, and an apparatus. Background Technology

[0002] An end-to-end (E2E) model can directly obtain the corresponding output from the input.

[0003] Applications of end-to-end models can include, but are not limited to, speech recognition and optical character recognition. Speech recognition refers to recognizing the text corresponding to spoken words, while optical character recognition refers to recognizing the text contained in an image.

[0004] A typical example of an E2E model is the Connectionist Temporal Classification (CTC) model. The decoding process in a CTC model is extremely crucial. Taking the CTC decoding process in speech recognition as an example, assuming there is a feature sequence of length T, the model infers a result of T×d (where d represents the number of modeling units for each character). The decoding goal is to decode each T... i The character category is determined at each step, ultimately yielding the optimal character sequence of length T or less (blank or consecutive characters are removed). The most intuitive approach is to combine all possibilities, obtain the probability of each path, merge paths with the same result, sum the probabilities, and take the prefix with the largest probability as the final result. While this method guarantees the optimal answer, it is extremely costly, requiring the listing of all combinations to be d. T This possibility is generally unacceptable.

[0005] Another simple approach uses a greedy algorithm. That is, for each T... i The result is obtained by taking the character corresponding to the index with the highest probability value in the d-dimensional dimension at any given time, but the result is often suboptimal.

[0006] The most common approach currently is to use a pruning search algorithm, which merges prefixes with the same prefix at each time step, and then expands to retain only the Top-N candidates, called CTC Prefix Beam Search. This method is between brute-force combination and greedy algorithms, achieving a balance between decoding accuracy and speed by setting the parameter N.

[0007] For the time complexity of CTC Prefix Beam Search, please refer to the following analysis.

[0008] The current decoding time is t. N prefixes have been retained so far. There are d possible options at time t. Since d is often very large (e.g., around ten thousand), only the top-M characters with the highest probabilities are selected. Each character is then concatenated with all prefixes to obtain M×N new prefixes. Identical prefixes are further merged, their probabilities are summed, and the prefixes are reordered, retaining the top-N new prefixes. This process continues until the next decoding time t+1. It's important to note that the process of concatenating each character with a prefix involves token propagation and other mechanisms, making it quite complex. Therefore, the complexity of selecting the top-M characters can be ignored. Thus, the time complexity of decoding a single time step is O(N×M). For simplicity, we take M=N, meaning the time complexity is O(N...). 2 ).

[0009] The analysis above shows that the larger the value of N, the closer the decoding result is to the optimal value, but the decoding speed will decrease exponentially. Therefore, to obtain better decoding results, the performance loss is significant.

[0010] Therefore, a CTC decoding scheme that can reduce time complexity is needed to achieve better decoding results while ensuring performance. Summary of the Invention

[0011] One of the technical problems this disclosure aims to solve is how to reduce the time complexity of CTC decoding in order to achieve better decoding results while ensuring performance.

[0012] According to the first aspect of this disclosure, a decoding method is provided for decoding classification results obtained by processing input by an end-to-end model. The classification results are used to characterize the probability values ​​of characters corresponding to different time steps in the input as different characters in the character solution space. The decoding method includes: iteratively performing the following steps for at least one time step: popping the top element from a max-heap, the max-heap including at least one heap element, any heap element corresponding to a first character prefix, the value of the heap element being used to characterize the cumulative probability value of the first character prefix, the first character prefix being composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations including a second character prefix and a first character, the second character prefix being a character prefix in the current set of character prefixes, and the first character... The character is the character corresponding to one probability value in the probability values ​​corresponding to the current time step in the classification result. The current character prefix set is the character prefix set corresponding to the decoding result of the previous time step. For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the probability values ​​corresponding to the current time step in the classification result that is after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is after the second character prefix.

[0013] Optionally, the decoding method further includes: taking the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step, and iteratively executing the following steps for at least one time step, including: traversing all time steps except the first time step in chronological order, and iteratively executing the following steps for any time step, wherein the character prefix set composed of the popped top elements that satisfy the iteration termination condition is the decoding result of the current time step.

[0014] Optionally, the iteration terminates when the number of popped top elements is greater than or equal to the second number.

[0015] Optionally, before iterating through at least one time step to perform the following steps, the decoding method further includes: concatenating the character prefix with the largest cumulative probability value in the current character prefix set with the character corresponding to the character with the largest probability value in the classification results corresponding to the current time step, and adding the concatenated character prefix as a heap element to a max-heap.

[0016] Optionally, before iterating through the following steps for at least one time step, the decoding method further includes: determining one or more sets of target character prefixes and character combinations based on the characters corresponding to the probability values ​​of the current character prefix set and the classification results corresponding to the current time step; any set of target character prefixes and character combinations includes a target character and a target character prefix, wherein the target character is the same as the last character of the target character prefix, or the target character is a whitespace character; adding the character prefix obtained by concatenating the target character with the target character prefix in all sets of target character prefixes and character combinations as a heap element to a max-heap, wherein the value of the heap element is the sum of the cumulative probability values ​​of all character prefixes that are the same as the character prefix corresponding to the heap element.

[0017] Optionally, based on the characters corresponding to the probability values ​​in the current character prefix set and classification results corresponding to the current time step, determine one or more sets of target character prefixes and character combinations, including: constructing a prefix dictionary and a suffix dictionary, where the keys of elements in the prefix dictionary are used to represent character prefixes, and the values ​​of elements in the prefix dictionary are used to represent the order of character prefixes in the current character prefix set; the keys of elements in the suffix dictionary are used to represent characters, and the values ​​of elements in the suffix dictionary are used to represent the order of character prefixes ending with that character in the current character prefix set; traversing the characters corresponding to at least some of the probability values ​​in the classification results corresponding to the current time step, and determining one or more sets of target character prefixes and character combinations based on the prefix dictionary and suffix dictionary.

[0018] Optionally, before iteratively performing the following steps for at least one time step, the decoding method further includes: retaining only the third number of the largest probability values ​​for the probability values ​​in the classification results corresponding to the current time step.

[0019] Optionally, the decoding method further includes: recording the character prefixes and character combinations corresponding to the character prefixes added to the max-heap; wherein concatenating the second character prefix and the second character in the character prefixes and character combinations, concatenating the third character prefix and the first character in the character prefixes and character combinations, and adding the concatenated character prefix as a heap element to the max-heap includes: determining whether the second character prefix and the second character, and the third character prefix and the first character in the character prefixes and character combinations belong to the already recorded character prefixes and character combinations; if the second character prefix and the second character in the character prefixes and character combinations do not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the second character prefix and the second character in the character prefixes and character combinations is added to the max-heap, and / or if the first character in the third character prefix and the character prefix and character combinations does not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the third character prefix and the first character in the character prefix and character combinations is added to the max-heap.

[0020] Optionally, the cumulative probability value of the character prefix includes a first cumulative probability value and a second cumulative probability value. The first cumulative probability value is the cumulative probability value when the character prefix ends with a whitespace character, and the second cumulative probability value is the cumulative probability value when the character prefix ends with a non-whitespace character.

[0021] Optionally, if the first character is a blank character, then the character prefix obtained by concatenating the second character prefix with the first character is the same as the second character prefix. The first cumulative probability value of the concatenated character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-blank character and is the same as the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix includes two character prefixes. One character prefix is ​​the same as the second character prefix, and the other character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the character prefix that is the same as the second character prefix is ​​equal to the product of the second cumulative probability value of the second character prefix and the probability value of the first character. The second cumulative probability value of the new character prefix is ​​equal to the product of the first cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-blank character and is different from the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the new character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character.

[0022] According to a second aspect of this disclosure, a text recognition method is also provided for recognizing text contained in an input, comprising: processing the input using an end-to-end model to obtain a classification result, the classification result being used to characterize the probability values ​​of characters corresponding to different time steps in the input as different characters in the character solution space; taking the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step; traversing all remaining time steps in chronological order, and iteratively executing the following steps for any time step: popping the top element from a max-heap, the max-heap including at least one heap element, any heap element corresponding to a first character prefix, the value of the heap element being used to characterize the cumulative probability value of the first character prefix, the first character prefix being composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations including a second character prefix and a first character, the second character prefix being the current... A character prefix in the character prefix set is defined as follows: the first character is the character corresponding to a probability value in the classification result corresponding to the current time step; the current character prefix set is the character prefix set corresponding to the decoding result of the previous time step. For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the classification result corresponding to the current time step, which is after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is after the second character prefix. The character sequence represented by the first popped top element of the heap at the last time step is taken as the input text recognition result.

[0023] According to a third aspect of this disclosure, a decoding device is also provided for decoding a classification result obtained by processing an end-to-end model for an input. The classification result is used to characterize the probability values ​​of characters corresponding to different time steps in the input as different characters in the character solution space. The decoding device includes an iterative processing module, which iteratively executes the following steps for at least one time step: popping the top element from a max-heap, the max-heap including at least one heap element, any heap element corresponding to a first character prefix, the value of the heap element being used to characterize the cumulative probability value of the first character prefix, the first character prefix being composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations including a second character prefix and a first character, the second character prefix being a character in the current character prefix set. The prefix is ​​defined as follows: the first character is the character corresponding to one probability value in the classification result corresponding to the current time step, and the current character prefix set is the character prefix set corresponding to the decoding result of the previous time step. For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​then added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the classification result corresponding to the current time step after the probability value corresponding to the first character, and the third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is after the second character prefix.

[0024] Optionally, the decoding device further includes: a first determining module, used to take the character corresponding to the first number of the largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step; and an iterative processing module traversing all time steps except the first time step in sequence, and iteratively executing the following steps for any time step, wherein the character prefix set composed of the popped top elements that satisfy the iteration termination condition is the decoding result of the current time step.

[0025] Optionally, the iteration terminates when the number of popped top elements is greater than or equal to the second number.

[0026] Optionally, the decoding device further includes: a first adding module, configured to concatenate the character prefix with the largest cumulative probability value in the current character prefix set with the character corresponding to the character with the largest probability value in the classification result corresponding to the current time step before iteratively performing the following steps for at least one time step, and add the concatenated character prefix as a heap element to a max-heap.

[0027] Optionally, the decoding device further includes: a second determining module and a second adding module. Before the iterative processing module iteratively executes the following steps for at least one time step, the second determining module determines one or more sets of target character prefixes and character combinations based on the current character prefix set and the characters corresponding to the probability values ​​of the current time step in the classification results. Any set of target character prefixes and character combinations includes a target character and a target character prefix. The target character is the same as the last character of the target character prefix, or the target character is a blank character. The second adding module concatenates the target character and the target character prefix in all sets of target character prefixes and character combinations to obtain character prefixes, and adds them as heap elements to a max-heap. The value of the heap element is the sum of the cumulative probability values ​​of all character prefixes that are the same as the character prefixes corresponding to the heap element.

[0028] Optionally, the second determining module includes: a construction module for constructing a prefix dictionary and a suffix dictionary, wherein the keys of elements in the prefix dictionary are used to represent character prefixes, and the values ​​of elements in the prefix dictionary are used to represent the order of character prefixes in the current character prefix set; the keys of elements in the suffix dictionary are used to represent characters, and the values ​​of elements in the suffix dictionary are used to represent the order of character prefixes ending with that character in the current character prefix set; and a traversal module for traversing the characters corresponding to at least some of the probability values ​​in the classification results corresponding to the current time step, and determining one or more sets of target character prefixes and character combinations based on the prefix dictionary and the suffix dictionary.

[0029] Optionally, the decoding device further includes a filtering module, which, before the iterative processing module iteratively performs the following steps for at least one time step, retains only the third number of the largest probability values ​​of the probability values ​​corresponding to the current time step in the classification results.

[0030] Optionally, the decoding device further includes: a recording module, used to record the character prefixes and character combinations corresponding to the character prefixes added to the max-heap, wherein the iterative processing module is further used to determine whether the second character prefix and the second character, and the third character prefix and the first character in the character prefix and character combination, belong to the already recorded character prefixes and character combinations; if the second character prefix and the second character in the character prefix and character combination do not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the second character prefix and the second character in the character prefix and character combination is added to the max-heap, and / or if the first character in the third character prefix and character prefix and character combination does not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the third character prefix and the first character in the character prefix and character combination is added to the max-heap.

[0031] Optionally, the cumulative probability value of the character prefix includes a first cumulative probability value and a second cumulative probability value. The first cumulative probability value is the cumulative probability value when the character prefix ends with a whitespace character, and the second cumulative probability value is the cumulative probability value when the character prefix ends with a non-whitespace character.

[0032] Optionally, if the first character is a blank character, then the character prefix obtained by concatenating the second character prefix with the first character is the same as the second character prefix. The first cumulative probability value of the concatenated character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-blank character and is the same as the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix includes two character prefixes. One character prefix is ​​the same as the second character prefix, and the other character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the character prefix that is the same as the second character prefix is ​​equal to the product of the second cumulative probability value of the second character prefix and the probability value of the first character. The second cumulative probability value of the new character prefix is ​​equal to the product of the first cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-blank character and is different from the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the new character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character.

[0033] According to a fourth aspect of this disclosure, a text recognition device is also provided for recognizing text contained in an input, comprising: a first processing module for processing the input using an end-to-end model to obtain a classification result, the classification result being used to characterize the probability values ​​of characters corresponding to different time steps in the input as different characters in the character solution space; a second processing module for taking the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step; and a third processing module for traversing all remaining time steps in sequential order, and iteratively executing the following steps for any given time step: popping the top element from a max-heap, the max-heap including at least one heap element, any heap element corresponding to a first character prefix, the value of the heap element being used to characterize the cumulative probability value of the first character prefix, the first character prefix being composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations including a second character prefix and a first character. The second character prefix is ​​a character prefix in the current character prefix set, and the first character is the character corresponding to a probability value in the classification result corresponding to the current time step. The current character prefix set is the character prefix set corresponding to the decoding result of the previous time step. For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the classification result corresponding to the probability value of the current time step that is after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is after the second character prefix. The determination module is used to take the character sequence represented by the first popped top element of the heap in the last time step as the input text recognition result.

[0034] According to a fifth aspect of this disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the methods described in the first or second aspect above.

[0035] According to a sixth aspect of this disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored, which, when executed by a processor of an electronic device, causes the processor to perform the method described in the first or second aspect above.

[0036] Therefore, this disclosure utilizes a heap data structure for decoding at at least one time step, thereby reducing the time complexity of decoding at a single time step from O(N) compared to CTCPrefix Beam Search.2 The value is reduced to O(N*log(N)). Attached Figure Description

[0037] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof taken in conjunction with the accompanying drawings, wherein like reference numerals generally denote like parts.

[0038] Figure 1 A schematic flowchart of a decoding method according to an embodiment of the present disclosure is shown.

[0039] Figure 2 A schematic flowchart of a text recognition method according to an embodiment of the present disclosure is shown.

[0040] Figure 3 A schematic diagram of a decoding apparatus according to an embodiment of the present disclosure is shown.

[0041] Figure 4 A schematic diagram of the structure of a text recognition device according to an embodiment of the present disclosure is shown.

[0042] Figure 5 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown, which can be used to implement the above-described decoding method or text recognition method. Detailed Implementation

[0043] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0044] This disclosure aims to reduce the time complexity of the decoding process and improve the decoding speed by optimizing the decoding process in the processing flow based on a general end-to-end model.

[0045] The decoding process refers to decoding the classification results obtained by the end-to-end model from processing the input, in order to obtain the output corresponding to the input. Here, the input can also be called the input sequence, and the output can also be called the output sequence.

[0046] The application scenarios of this disclosure can be, but are not limited to, speech recognition and optical character recognition (OCR) scenarios. Among them, optical character recognition can also be called text recognition.

[0047] In speech recognition, the input can be audio data, such as an audio feature sequence; the output is the speech recognition result. In optical character recognition, the input can be image data, such as an image feature sequence; the output is the text recognition result obtained by recognizing the text in the image, which can be a character sequence or a text sequence.

[0048] End-to-end models can be based on Connectionist Temporal Classification (CTC). CTC models are trained end-to-end, and during inference, the input can be directly translated into the final character or text sequence, offering simplicity and efficiency. CTC is an algorithm applicable to fields such as speech recognition and optical character recognition, capable of resolving issues such as inconsistent input and output sequence lengths and alignment problems.

[0049] It should be noted that the focus of this disclosure is on optimizing the decoding process, while the encoding process (i.e., the process of processing the input to obtain the classification result) can be found in the prior art.

[0050] For example, an end-to-end model can be based on an RNN (Recurrent Neural Network) and CTC (Concurrent Traceability) model structure. That is, the RNN is used to extract features and generate probabilities from the input to obtain a classification result. Then, a CTC-based decoding method is used to decode the classification result to obtain the output corresponding to the input.

[0051] The classification result represents the probability value of the character corresponding to (or contained in) different time steps in the input being a different character in the character solution space. The character solution space can refer to the set of all possible characters (including whitespace). The classification result can be viewed as a probability matrix. The number of rows in the probability matrix represents the size of the character solution space, i.e., the number of characters. The number of columns in the probability matrix represents the number of time steps. Each column vector in the probability matrix represents the probability distribution of the characters contained in a given time step in the input. Each element in the column vector corresponds to a character, and the value of the element represents the probability value that the character corresponding to that time step is the character corresponding to that element.

[0052] When the input is speech (i.e., audio data), a time step refers to a time segment within the speech input. The classification result then represents the probability that the character corresponding to the audio data (or audio features) at different time segments (e.g., 25ms) in the speech input is a different character in the character solution space. The input to an end-to-end speech recognition model typically transforms a one-dimensional temporal speech signal into two-dimensional time-frequency features. Therefore, one time step can correspond to a column of features in the input sequence, and each column of features is obtained from audio data at a fixed time segment, such as 25ms.

[0053] When the input is an image (i.e., image data), the time step refers to a position or region in the image input that contains a character. In this case, the classification result is used to characterize the probability value of the character contained in different positions or regions of the image input as different characters in the character solution space.

[0054] The classification result can be denoted as T×d. Here, T represents the step size or number of steps, i.e., the number of time steps. d represents the number of modeling units for each character. T×d represents the probability value that the character corresponding to different time steps in step T is a different character among d characters. The decoding goal is to determine the probability of each time step T. i The character categories are determined, and the optimal character sequence with a length less than or equal to T is obtained (blank or consecutive characters are removed).

[0055] Decoding at a certain time step means concatenating (combining) the character prefixes with the characters based on the cumulative probability values ​​of different character prefixes in the decoding result of the previous time step (i.e., the current character prefix set mentioned below) and the probability values ​​of the characters corresponding to the current time step being different characters in the character solution space. This results in a certain number of concatenated character prefixes with the highest cumulative probability values, which are then used as the decoding result of the current time step and participate in the decoding process of the next time step.

[0056] Therefore, decoding the time step can be simplified to the problem of adding two ordered arrays pairwise and finding the Top-N.

[0057] If we use the CTC Prefix Beam Search algorithm to find all possible combinations, then the time complexity can be written as O(N). 2 Therefore, this disclosure proposes that a heap data structure (mainly a max-heap) can be used for decoding to reduce the original N values ​​while ensuring consistent performance with the CTC Prefix Beam Search algorithm. 2 The complexity is reduced to O(N*log(N)).

[0058] Figure 1 A schematic flowchart of a decoding method according to an embodiment of the present disclosure is shown.

[0059] This disclosure can be executed iteratively for at least one time step. Figure 1 Steps S110 and S120 are shown.

[0060] For the first time step, the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification results can be directly used as the character prefix. The resulting set of character prefixes is the decoding result of the first time step. Therefore, at least one time step can refer to any one or more time steps other than the first time step. For example, all time steps except the first time step can be traversed in chronological order, and the decoding can be performed iteratively for any given time step. Figure 1 Steps S110 and S120 are shown.

[0061] Alternatively, it can be executed for the first time step. Figure 1 The steps S110 and S120 are shown. At this point, the decoding result of the previous time step can be regarded as a blank character with a cumulative probability value of 1.

[0062] See Figure 1 In step S110, the top element of the max-heap is popped.

[0063] A max-heap, also known as a max-heap or max-top heap, is a large heap.

[0064] A max-heap is a binary heap where the root node (also called the top of the heap) has the largest value among all nodes in the heap. In a max-heap, the value of each node is greater than or equal to the value of its child nodes.

[0065] A max-heap contains at least one heap element (i.e., a node in the heap). Each heap element corresponds to a first character prefix. The value of the heap element represents the cumulative probability value of the first character prefix. The first character prefix is ​​formed by concatenating at least one set of character prefixes and character combinations, including a second character prefix and a first character. The second character prefix is ​​a character prefix from the current set of character prefixes, and the first character is the character corresponding to one of the probability values ​​in the classification result corresponding to the current time step.

[0066] It should be noted that the first character prefix can be formed by concatenating one set of character prefixes and character combinations, or by concatenating multiple sets of character prefixes and character combinations. When it is formed by concatenating multiple sets of character prefixes and character combinations, it means that the first character prefix will be generated repeatedly by these multiple sets of character prefixes and character combinations. In this case, the cumulative probability value of the first character prefix will be the sum of the cumulative probabilities of the first character prefix generated by the multiple sets of character prefixes and character combinations.

[0067] For example, multiple character prefixes and combinations such as {abcd, d}, {abcdd, d}, and {abcdd, whitespace} can all generate the character prefix "abcdd". Therefore, when the character prefix "abcdd" is added as a heap element to a max-heap, the character prefix "abcdd" can be formed by concatenating these multiple character prefixes and combinations, and the cumulative probability value of the character prefix "abcdd" is equal to the sum of the cumulative probability values ​​(the first cumulative probability value and / or the second cumulative probability value mentioned below) of these multiple character prefixes and combinations generating the character prefix "abcdd".

[0068] The current character prefix set is the character prefix set corresponding to the decoding result of the previous time step. For example, it could be an iterative execution based on the previous time step. Figure 1 The method shown is a set of character prefixes consisting of all popped top elements of the heap, provided that the iteration terminates.

[0069] Each character prefix in the current character prefix set has a cumulative probability value. The cumulative probability value of a character prefix represents its probability of occurrence. For example, the cumulative probability value of a character prefix can be the sum of the probability values ​​of all decoding paths corresponding to that character prefix. The probability value of a decoding path is equal to the product of the probability values ​​of the characters at all time steps along that decoding path. Here, the decoding path refers to the decoding path from the first time step to the time step corresponding to the current character prefix, not the decoding path across all time steps.

[0070] The first character prefix corresponding to the top element in a max-heap refers to the character prefix with the highest cumulative probability value among all possible character prefixes generated in the current situation, excluding the previously popped top element.

[0071] In step S120, for any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, the third character prefix is ​​concatenated with the first character in the character prefix and character combination, and the concatenated character prefix is ​​added as a heap element to the max-heap.

[0072] The second character is the character corresponding to the first probability value in the classification result that is located after the probability value corresponding to the first character in the probability value corresponding to the current time step. The third character prefix is ​​the first character prefix in the current character prefix set that is located after the second character prefix corresponding to the top element of the heap in the cumulative probability value.

[0073] As mentioned above, the first character prefix corresponding to the top element in a max-heap refers to the character prefix with the highest cumulative probability value among all possible character prefixes generated in the current situation, excluding the previously popped top element. The first character prefix corresponding to the top element can be generated from one or more character prefixes and character combinations. After popping the top element from the max-heap, the next possible character prefix with the highest cumulative probability value will be generated from the character prefixes corresponding to the two child nodes of the original top node (i.e., the top node) in the max-heap, as well as from two combinations: "the second character prefix and the second character in the character prefix and character combination" and "the third character prefix and the first character in the character prefix and character combination". Therefore, for any character prefix and character combination corresponding to the top element, the second character prefix and the second character in the character prefix and character combination can be concatenated, and the third character prefix can be concatenated with the first character in the character prefix and character combination. Then, the resulting character prefixes (new character prefixes not appearing in the max-heap) are added as heap elements to the max-heap.

[0074] In each iteration of the process, such as Figure 1 After executing steps S110 and S120 (or after step S110 and before step S120), it can be determined whether the iteration termination condition is met. If the iteration termination condition is met, the iteration stops; otherwise, the iteration process of steps S110 and S120 continues. The iteration termination condition can be, but is not limited to, the number of popped top elements being greater than or equal to a second quantity. The second quantity can be the same as or different from the first quantity mentioned above. In addition, the iteration termination condition can also be that the number of iterations or the execution time reaches a predetermined threshold.

[0075] Therefore, by iteratively executing steps S110 and S120, a set of character prefixes (i.e., the decoding result) sorted in descending order of cumulative probability values ​​can be obtained for the current time step. This set of character prefixes can serve as the current set of character prefixes for the next time step. Steps S110 and S120 are iteratively executed for the next time step until decoding is completed for all time steps. The character sequence represented by the first popped top element in the final set of character prefixes can be used as the final decoding result. The final decoding result is also the text recognition result obtained by recognizing the input, and thus the final output text recognition result.

[0076] Analyzing the time complexity of a single time step in the decoding process described above reveals that pairing a combination in the heap data structure (i.e., adding a heap element) has a time complexity of log(N), retrieving N top-heap combinations involves a time complexity of O(N), and creating a maximum of 2N heap elements results in a time complexity of O(2N)*log(N). In summary, the complexity ranges from O(N)*log(N). 2 The time complexity of this decoding method can be reduced from O(N*log(N)) to O(N*log(N)). Therefore, compared with CTC Prefix Beam Search, the time complexity of each time step can be reduced from O(N*log(N)) while ensuring the same effect. 2 The value is reduced to O(N*log(N)).

[0077] Initially, it can be assumed that concatenating the character prefix with the highest cumulative probability value in the current character prefix set with the character corresponding to the character with the highest probability value among the probability values ​​corresponding to the current time step in the classification results will result in the character prefix with the highest cumulative probability value. Therefore, during iterative execution for at least one time step... Figure 1 Before steps S110 and S120, this disclosure can concatenate the character prefix with the largest cumulative probability value in the current character prefix set with the character corresponding to the character with the largest probability value in the classification results corresponding to the current time step, and add the concatenated character prefix as a heap element to a max-heap.

[0078] When combining character prefixes with characters, there are cases where different character prefixes and character combinations can produce the same character prefix. On one hand, repeatedly adding the same character prefix as a heap element to a max-heap may result in multiple top elements corresponding to the same character prefix being popped. On the other hand, because the cumulative probability value of the same character prefix is ​​calculated separately, the character prefix corresponding to the top element of the max-heap (i.e., the first character prefix) may not be the character prefix with the highest cumulative probability value among all possible character prefixes in the current situation, affecting the accuracy of the decoding result.

[0079] Therefore, this disclosure proposes that, during iterative execution for any given time step... Figure 1 Before steps S110 to S130, all character prefixes and combinations that may produce the same character prefix can be pre-calculated and placed into a max-heap. Generally, the number of character prefixes and combinations involved is small, so it has almost no impact on the complexity.

[0080] Since whitespace characters are removed from the character sequence, the character prefix includes two cases: one ending with a whitespace character, and the other ending with a non-whitespace character. That is, these two cases share the same character prefix. Correspondingly, the cumulative probability value of the character prefix is ​​also divided into two types: a first cumulative probability value and a second cumulative probability value. The first cumulative probability value is the cumulative probability value when the character prefix ends with a whitespace character. The second cumulative probability value is the cumulative probability value when the character prefix ends with a non-whitespace character.

[0081] The character prefix resulting from concatenating a whitespace character with any character prefix remains unchanged; it is still the original character prefix. Since the character added after the prefix is ​​a whitespace character, the concatenated character prefix only has a first cumulative probability value. This first cumulative probability value is equal to the product of the cumulative probability value of the original character prefix (i.e., the sum of the first and second cumulative probability values) and the probability value of the whitespace character.

[0082] There are two scenarios when a non-whitespace character is combined with a character prefix. One is that the non-whitespace character is the same as the last character in the character prefix, and the other is that the non-whitespace character is different from the last character in the character prefix.

[0083] Since characters ending with whitespace and those ending with non-whitespace share the same character prefix, when a non-whitespace character is the same as the last character in the prefix, concatenating the non-whitespace character with the prefix generally produces two types of prefixes. The first type adds a non-whitespace character to the end of the prefix, while the second type combines the non-whitespace character with the last non-whitespace character in the prefix. This second type is formally equivalent to the original prefix. Both types of prefixes have only one second cumulative probability value. The second cumulative probability value of the first type is equal to the product of the first cumulative probability value of the prefix being concatenated and the probability value of the character being concatenated. The second cumulative probability value of the second type is also equal to the product of the second cumulative probability value of the prefix being concatenated and the probability value of the character being concatenated.

[0084] When a non-whitespace character is different from the last character in a character prefix, concatenating the non-whitespace character with the character prefix will result in a new character prefix. This new character prefix is ​​obtained by adding a non-whitespace character after the original character prefix. The character prefix obtained from this concatenation only has a second cumulative probability value, which is equal to the product of the cumulative probability values ​​of the character prefixes involved in the concatenation (i.e., the sum of the first and second cumulative probability values) and the probability value of the non-whitespace character.

[0085] According to the above rules for concatenating character prefixes and characters, for a character prefix and character combination consisting of a second character prefix and a first character, when concatenating the second character prefix and the first character: if the first character is a whitespace character, then the character prefix obtained by concatenating the second character prefix and the first character is the same as the second character prefix, and the first cumulative probability value of the concatenated character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character; if the first character is a non-whitespace character, and the first character is the same as the last character in the second character prefix, then the character prefix obtained by concatenating the first character and the second character prefix includes two character prefixes, one of which is the same as the second character prefix, and the other character... The prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the character prefix that is the same as the second character prefix is ​​equal to the product of the second cumulative probability value of the second character prefix and the probability value of the first character. The second cumulative probability value of the new character prefix is ​​equal to the product of the first cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-whitespace character and the first character is different from the last character in the second character prefix, then the character prefix obtained by concatenating the first character and the second character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the new character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character.

[0086] Based on the above rules for concatenating character prefixes and characters, if a character is different from the last character of any prefix in the current character prefix set, then this character will not repeat after concatenation with any prefix, meaning there is no possibility of merging and it does not need to be placed in a max-heap beforehand. However, if a character is the same as the last character of some prefixes in the current character prefix set, then this character may produce the same result with these prefixes, meaning that concatenation may result in duplicate prefixes.

[0087] The occurrence of duplicate character prefixes mainly includes the following two situations.

[0088] One type is where character prefixes, after removing the last character, have the same prefix. For example, prefix A is "abcdd" and prefix B is "abcd". When character 'd' is concatenated with prefix A, two results are produced: "abcdd" and "abcddd". Similarly, when character 'd' is concatenated with prefix B, two results are also produced: "abcd" and "abcdd". Here, "abcdd" represents the repeated prefix.

[0089] Another possibility is that the character is a whitespace character. Taking the character prefix A as an example again, the character prefix A concatenated with a whitespace character will still result in the character prefix abcdd, which is the same as one of the character prefixes abcdd obtained by concatenating the character d with the character prefix A (or the character prefix B).

[0090] Based on the above analysis of the recurring character prefixes, this disclosure proposes that, during iterative execution for at least one time step... Figure 1 Before steps S110 and S120, one or more sets of target character prefixes and character combinations can be determined based on the characters corresponding to the probability values ​​of the current character prefix set and classification results at the current time step. Any set of target character prefixes and character combinations includes a target character and a target character prefix. The target character is the same as the last character of the target character prefix, or the target character is a whitespace character. Target character prefixes and character combinations are those that may produce the same character prefix. Then, the target character and target character prefix in all sets of target character prefixes and character combinations are concatenated. All concatenated character prefixes are added as heap elements to a max-heap. The value of each heap element is the sum of the cumulative probability values ​​of all concatenated character prefixes that are the same as the corresponding character prefix of that pair. For details on the concatenation rules and the calculation method of the cumulative probability value, please refer to the relevant description above.

[0091] To facilitate the determination of target character prefixes and character combinations, a prefix dictionary and a suffix dictionary can be constructed. The keys of elements in the prefix dictionary represent character prefixes in the current character prefix set, and the values ​​represent the order of character prefixes in the current character prefix set. The keys of elements in the suffix dictionary represent characters, and the values ​​represent the order of character prefixes ending with that character in the current character prefix set. The system iterates through the probability values ​​in the classification results corresponding to at least some of the probability values ​​at the current time step, determining one or more sets of target character prefixes and character combinations based on the prefix and suffix dictionaries. For example, for a given character, if the character is non-whitespace and there is no corresponding key in the suffix dictionary, it can be determined that concatenating this character with any character prefix will not produce the same character prefix. If the character is non-whitespace and there is a corresponding key in the suffix dictionary, then based on the key value and the prefix dictionary, all character prefixes ending with this character, the character itself, and whitespace characters can be considered combinations that may produce the same character prefix. The character prefixes generated by these combinations and their cumulative probability values ​​are calculated and added to a max-heap.

[0092] To avoid duplicate additions, this disclosure can also record the character prefixes and character combinations corresponding to the character prefixes added to the max-heap. For example, the status of the character prefixes and character combinations corresponding to the character prefixes added to the max-heap can be marked as "visited". During step S120, it can be determined whether the second character prefix and second character, and the third character prefix and the first character in the character prefix and character combination, belong to the already recorded character prefixes and character combinations. If the second character prefix and second character in the character prefix and character combination do not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the second character prefix and second character in the character prefix and character combination is added to the max-heap; conversely, if the second character prefix and second character in the character prefix and character combination belong to the already recorded character prefixes and character combinations, then the concatenation of the second character prefix and second character in the character prefix and character combination is not performed. Accordingly, if the first character in the third character prefix and the character prefix and character combination does not belong to the already recorded character prefix and character combination, then the character prefix obtained by concatenating the first character in the third character prefix and the character prefix and character combination is added to the max heap; otherwise, if the first character in the third character prefix and the character prefix and character combination belongs to the already recorded character prefix and character combination, then the first character in the third character prefix and the character prefix and character combination is no longer concatenated.

[0093] Iterative execution for at least one time step Figure 1 Before steps S110 and S120, it is possible to retain only the third number of the largest probability values ​​corresponding to the current time step in the classification results, in order to speed up the decoding process. The third number can be the same as or different from the first number mentioned above.

[0094] Application examples

[0095] The following example uses a speech recognition scenario to illustrate the decoding process.

[0096] In this embodiment, the input for decoding can be the classification result P of the CTC model, with shape T×d, representing the probability value of belonging to different characters among d characters at each time point in step T, where the 0th character can represent the blank character. The decoding output can be the optimal character sequence.

[0097] The detailed decoding process is as follows.

[0098] 1. Traverse all T in order i At any given time, for each T i Decoding operations are performed continuously, and the specific operations are as follows.

[0099] 2. Select the top-N largest values ​​P from the current d-dimensional probabilities. n ={P1,P2,…,P n}, and record the corresponding index I. n ={I1,I2,…,I n}. Where N corresponds to the third quantity mentioned above.

[0100] 3. If T i At the initial moment, i.e., when i = 0, the corresponding character can be directly used as a prefix (i.e., a character prefix), and the total probability of this prefix is ​​recorded, denoted by hyps = [{prefix1, (prob_blank, prob_no_blank, prob_sum)}, ...]. Here, prob_blank (the first cumulative probability value mentioned above) represents the overall probability when the prefix ends with "blank," prob_no_blank (the second cumulative probability value mentioned above) represents the integer probability when the prefix ends with a non-"blank" character (because the "blank" character is removed from the character sequence, the prefixes represented by these two values ​​are shared), and prob_sum represents the total cumulative probability of the prefix, which is equal to prob_blank plus prob_no_blank. All prefixes are then sorted in descending order of their total probability (i.e., prob_sum).

[0101] 4. If T i If it's a non-initial time, i.e., i>0, then we need to first find the prefixes that might be repeated during combination:

[0102] a) Construct a prefix dictionary, using the prefix in the hyps of the previous time step as the key and its order in the hyps as the value;

[0103] b) Construct a suffix dictionary, using the last character of each prefix in the hyps as the key and the value as a list [idx_1, idx_i, ...], where idx_i indicates that the last character of the i-th prefix is ​​the key;

[0104] c) Traverse the current Top-N characters I n Based on the suffix dictionary and the prefix dictionary, prefixes that may produce the same prefix and I are grouped together. i The combinations are put together, and the new prefixes generated by these combinations are calculated. Then, these new prefixes and their probabilities are inserted into the heap structure. In addition, the combined prefix and character idx combination (prefix, idx) needs to be marked as visited to prevent duplicate additions later.

[0105] d) The probability of combining the first prefix of hyps with I1 is the highest globally (because hyps and I1...). n Since they are all ordered, if this combination (prefix_1, I1) is not marked as visited, we need to first calculate new_prefix for this combination, insert the combination into the heap, and then mark it as visited.

[0106] e) Next, we will use the current heap to perform N loops, each time performing the following operation:

[0107] i. Take the top element of the heap. It is the element with the highest probability of being the new_prefix among the remaining elements. Then pop this top element from the heap. In addition to the new_prefix information, this will also contain all the (hyps_id, I) that generated the new_prefix. i The combination is denoted as visited_dicts.

[0108] ii. Iterate through all combinations of visited_dicts, and if (hyps_id+1, I i ) or (hyps_id, I i+1 If a combination has not been marked as visited beforehand, its combination result (new_prefix) will be calculated and inserted into the heap.

[0109] iii. Repeat the previous two steps until the current T. i We obtain Top-N new prefixes at any time, and they are sorted in descending order.

[0110] 5. Iterate through the next moment until the last moment.

[0111] 6. In the final moment, we can obtain the Top-N prefixes, among which the first prefix is ​​the result with the highest probability and can be used as the output.

[0112] The table below compares the decoding time of the proposed decoding algorithm and the CTC Prefix Beam Search algorithm on two different test sets, ensuring consistent performance. Datasets A and B in the table represent test data from two different sources. As shown in the table, the proposed decoding method effectively improves decoding speed when the search range increases compared to the CTC Prefix Beam Search algorithm.

[0113]

[0114]

[0115] Figure 2 A schematic flowchart of a text recognition method according to an embodiment of the present disclosure is shown. The text recognition method is used to recognize text contained in an input. The input may be audio data or image data.

[0116] See Figure 2 In step S210, the input is processed using an end-to-end model to obtain the classification result.

[0117] End-to-end models can be, but are not limited to, connection-based temporal classification models. The classification results are used to characterize the probability values ​​of characters at different time steps in the input being different characters in the character solution space.

[0118] In step S220, the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result is taken as the character prefix, and the resulting character prefix set is the decoding result of the first time step.

[0119] In step S230, all remaining time steps are traversed in chronological order, and steps S2310 and S2320 are executed iteratively for any given time step. Step S2310 is the same as step S110 described above, and step S2320 is the same as step S120 described above. For details on steps S2310 and S2320, please refer to the relevant descriptions above.

[0120] After decoding at one time step is completed, decoding at the next time step can begin, until decoding at all time steps is completed. In step S240, the character sequence (i.e., character prefix) represented by the first popped top element of the heap at the last time step is used as the input text recognition result.

[0121] The decoding method disclosed herein can also be implemented as a decoding device. Figure 3 A schematic diagram of a decoding apparatus according to an embodiment of the present disclosure is shown. The functional units of the decoding apparatus can be implemented by hardware, software, or a combination of hardware and software that implement the principles of the present disclosure. Those skilled in the art will understand that the functional units described below in conjunction with the accompanying drawings can be combined or divided into sub-units to implement the principles of the invention described above. Therefore, the description herein supports any possible combination, division, or further limitation of the functional units described herein.

[0122] The following is a brief description of the functional units that the decoding device can have and the operations that each functional unit can perform. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0123] See Figure 3The decoding device 300 may include an iterative processing module 310. The iterative processing module 310 can iteratively execute for at least one time step. Figure 1 Steps S110 and S120 are shown.

[0124] The decoding device 300 may further include a first determining module. The first determining module is used to take the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step. The iterative processing module traverses all time steps except the first time step in sequence, and iteratively executes the following steps for any time step, wherein the character prefix set composed of the popped top elements that meet the iteration termination condition is the decoding result of the current time step.

[0125] The decoding device 300 may further include a first adding module. The first adding module is used to concatenate the character prefix with the largest cumulative probability value in the current character prefix set with the character corresponding to the character with the largest probability value in the classification result corresponding to the current time step before the iterative processing module 310 iteratively performs the following steps for at least one time step, and add the concatenated character prefix as a heap element to a max-heap.

[0126] The decoding device 300 may further include a second determining module and a second adding module. Before iteratively performing the following steps for at least one time step, the second determining module can determine one or more sets of target character prefixes and character combinations based on the characters corresponding to the probability values ​​of the current character prefix set and the classification results corresponding to the current time step. Any set of target character prefixes and character combinations includes a target character and a target character prefix. The target character is the same as the last character of the target character prefix, or the target character is a blank character. The second adding module can add the character prefix obtained by concatenating the target character with the target character prefix in all sets of target character prefixes and character combinations as a heap element to a max-heap. The value of the heap element is the sum of the cumulative probability values ​​of all character prefixes that are the same as the character prefix corresponding to the heap element.

[0127] The second determining module may include: a construction module for constructing a prefix dictionary and a suffix dictionary, wherein the keys of elements in the prefix dictionary represent character prefixes, and the values ​​of elements in the prefix dictionary represent the order of character prefixes in the current character prefix set; the keys of elements in the suffix dictionary represent characters, and the values ​​of elements in the suffix dictionary represent the order of character prefixes ending with that character in the current character prefix set; and a traversal module for traversing the characters corresponding to at least some of the probability values ​​in the classification results corresponding to the current time step, and determining one or more sets of target character prefixes and character combinations based on the prefix dictionary and the suffix dictionary.

[0128] The decoding device 300 may further include a filtering module. The filtering module is used to retain only the third largest probability values ​​of the probability values ​​corresponding to the current time step in the classification results before the iterative processing module iteratively performs the following steps for at least one time step.

[0129] The decoding device 300 may further include a recording module. The recording module is used to record the character prefixes and character combinations corresponding to the character prefixes added to the max-heap. The iterative processing module 310 may also be used to determine whether the second character prefix and the second character, and the third character prefix and the first character in the character prefix and character combination, belong to the already recorded character prefixes and character combinations. If the second character prefix and the second character in the character prefix and character combination do not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the second character prefix and the second character in the character prefix and character combination is added to the max-heap; and / or if the first character in the third character prefix and character prefix and character combination does not belong to the already recorded character prefixes and character combinations, then the character prefix obtained by concatenating the third character prefix and the first character in the character prefix and character combination is added to the max-heap.

[0130] The text recognition method disclosed herein can also be implemented as a text recognition device. Figure 4 A schematic diagram of a text recognition device according to an embodiment of the present disclosure is shown. The functional units of the text recognition device can be implemented by hardware, software, or a combination of hardware and software that implement the principles of the present disclosure. Those skilled in the art will understand that the functional units described below in conjunction with the accompanying drawings can be combined or divided into sub-units to implement the principles of the invention described above. Therefore, the description herein supports any possible combination, division, or further limitation of the functional units described herein.

[0131] The following is a brief description of the functional units that a text recognition device can have and the operations that each functional unit can perform. For details, please refer to the relevant descriptions above, which will not be repeated here.

[0132] See Figure 4 The text recognition device 400 may include a first processing module 410, a second processing module 420, a third processing module 430, and a determination module 440.

[0133] The first processing module 410 processes the input using an end-to-end model to obtain a classification result, which represents the probability value of the character corresponding to different time steps in the input being a different character in the character solution space. The second processing module 420 takes the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step. The third processing module 430 iterates through all remaining time steps in chronological order and iteratively executes the above-described combination for any given time step. Figure 1 Steps S110 and S120 are shown. The determining module 440 is used to take the character sequence represented by the first popped top element of the heap in the last time step as the text recognition result of the input. For details on the decoding process, please refer to the relevant description above.

[0134] Figure 5 A schematic diagram of a computing device according to an embodiment of the present disclosure is shown, which can be used to implement the above-described decoding method or text recognition method.

[0135] See Figure 5 The computing device 500 includes a memory 510 and a processor 520.

[0136] Processor 520 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 520 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 520 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0137] Memory 510 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 520 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 510 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 510 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0138] The memory 510 stores executable code. When the executable code is processed by the processor 520, the processor 520 can execute the decoding method or text recognition method described above.

[0139] The decoding method, text recognition method, apparatus and device according to this disclosure have been described in detail above with reference to the accompanying drawings.

[0140] Furthermore, the method according to this disclosure can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing the steps defined in the above-described method of this disclosure.

[0141] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to this disclosure.

[0142] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A decoding method for decoding classification results obtained by processing an input using an end-to-end model, wherein the classification results characterize the probability values ​​of characters at different time steps in the input being different characters in the character solution space, characterized in that, The decoding method includes: iteratively performing the following steps for at least one time step. Pop the top element from the max-heap, wherein the max-heap includes at least one heap element, each heap element corresponds to a first character prefix, the value of the heap element is used to characterize the cumulative probability value of the first character prefix, the first character prefix is ​​composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations include a second character prefix and a first character, the second character prefix is ​​a character prefix in the current character prefix set, the first character is the character corresponding to a probability value in the classification result corresponding to the current time step, and the current character prefix set is the character prefix set corresponding to the decoding result of the previous time step; For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​then added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the probability value corresponding to the current time step in the classification result, which is located after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is located after the second character prefix.

2. The decoding method according to claim 1, characterized in that, Also includes: The character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification results is taken as the decoding result of the first time step. For at least one time step, the following steps are performed iteratively: traverse all time steps except the first time step in chronological order, and for any time step, the following steps are performed iteratively, wherein the set of character prefixes formed by the popped top elements of the heap that meet the iteration termination condition is the decoding result of the current time step.

3. The decoding method according to claim 1, characterized in that, The iteration terminates when the number of popped top elements is greater than or equal to the second number.

4. The decoding method according to claim 1, characterized in that, Before performing the following steps iteratively for at least one time step, the decoding method further includes: The character prefix with the highest cumulative probability value in the current character prefix set is concatenated with the character corresponding to the highest probability value in the classification result corresponding to the current time step. The concatenated character prefix is ​​then added as a heap element to a max-heap.

5. The decoding method according to claim 4, characterized in that, Before performing the following steps iteratively for at least one time step, the decoding method further includes: Based on the current character prefix set and the character corresponding to the probability value corresponding to the current time step in the classification result, determine one or more sets of target character prefixes and character combinations. Any set of target character prefixes and character combinations includes a target character and a target character prefix. The target character is the same as the last character of the target character prefix, or the target character is a blank character. All target character prefixes and the character prefixes obtained by concatenating the target character with the target character prefix in all groups of character prefixes are added as heap elements to the max-heap. The value of the heap element is the sum of the cumulative probability values ​​of all character prefixes that are the same as the character prefix corresponding to the heap element.

6. The decoding method according to claim 5, characterized in that, Based on the current character prefix set and the characters corresponding to the probability values ​​in the classification results at the current time step, determine one or more sets of target character prefixes and character combinations, including: Construct a prefix dictionary and a suffix dictionary. The keys of the elements in the prefix dictionary are used to represent character prefixes, and the values ​​of the elements in the prefix dictionary are used to represent the order of the character prefixes in the current character prefix set. The keys of the elements in the suffix dictionary are used to represent characters, and the values ​​of the elements in the suffix dictionary are used to represent the order of character prefixes ending with that character in the current character prefix set. Traverse the characters corresponding to at least some of the probability values ​​in the classification results corresponding to the current time step, and determine one or more sets of target character prefixes and character combinations based on the prefix dictionary and suffix dictionary.

7. The decoding method according to claim 1, characterized in that, Before performing the following steps iteratively for at least one time step, the decoding method further includes: For the probability values ​​corresponding to the current time step in the classification results, only the third largest probability value is retained.

8. The decoding method according to claim 1, characterized in that, Also includes: Record the character prefixes and character combinations corresponding to the character prefixes added to the max-heap; Specifically, the process involves concatenating the second character prefix from the character prefix and character combination with the second character, concatenating the third character prefix with the first character from the character prefix and character combination, and adding the concatenated character prefix as a heap element to the max-heap. Determine whether the second character prefix and the second character, and the third character prefix and the first character in the character prefix and character combination, belong to the already recorded character prefix and character combination; If the second character prefix and the second character in the character prefix and character combination do not belong to the already recorded character prefix and character combination, then the character prefix obtained by concatenating the second character prefix and the second character in the character prefix and character combination is added to the max-heap, and / or If the third character prefix and the first character in the character prefix and character combination do not belong to the already recorded character prefix and character combination, then the character prefix obtained by concatenating the third character prefix and the first character in the character prefix and character combination is added to the max-heap.

9. The decoding method according to claim 1, characterized in that, The cumulative probability value of the character prefix includes a first cumulative probability value and a second cumulative probability value. The first cumulative probability value is the cumulative probability value when the character prefix ends with a whitespace character, and the second cumulative probability value is the cumulative probability value when the character prefix ends with a non-whitespace character.

10. The decoding method according to claim 9, characterized in that, If the first character is a whitespace character, then the character prefix obtained by concatenating the second character prefix with the first character is the same as the second character prefix. The first cumulative probability value of the concatenated character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-whitespace character, and the first character is the same as the last character in the second character prefix, then the character prefix obtained by concatenating the first character and the second character prefix includes two character prefixes. One character prefix is ​​the same as the second character prefix, and the other character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the character prefix that is the same as the second character prefix is ​​equal to the product of the second cumulative probability value of the second character prefix and the probability value of the first character. The second cumulative probability value of the new character prefix is ​​equal to the product of the first cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-whitespace character and the first character is different from the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the new character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character.

11. A text recognition method for recognizing text contained in an input, characterized in that, include: An end-to-end model is used to process the input to obtain classification results. The classification results are used to characterize the probability values ​​of characters corresponding to different time steps in the input as different characters in the character solution space. The character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification results is taken as the decoding result of the first time step. Iterate through all remaining time steps in chronological order, and for any given time step, perform the following steps iteratively: Pop the top element from the max-heap, wherein the max-heap includes at least one heap element, each heap element corresponds to a first character prefix, the value of the heap element is used to characterize the cumulative probability value of the first character prefix, the first character prefix is ​​composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations include a second character prefix and a first character, the second character prefix is ​​a character prefix in the current character prefix set, the first character is the character corresponding to a probability value in the classification result corresponding to the current time step, and the current character prefix set is the character prefix set corresponding to the decoding result of the previous time step; For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​then added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the probability value corresponding to the current time step in the classification result, which is located after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is located after the second character prefix. The character sequence represented by the first popped top element of the heap at the last time step is used as the text recognition result of the input.

12. A decoding apparatus for decoding a classification result obtained by processing an input using an end-to-end model, wherein the classification result characterizes the probability values ​​of characters at different time steps in the input being different characters in a character solution space, characterized in that, The decoding device includes an iterative processing module, which is configured to iteratively perform the following steps for at least one time step: Pop the top element from the max-heap, wherein the max-heap includes at least one heap element, each heap element corresponds to a first character prefix, the value of the heap element is used to characterize the cumulative probability value of the first character prefix, the first character prefix is ​​composed of at least one set of character prefixes and character combinations, the character prefixes and character combinations include a second character prefix and a first character, the second character prefix is ​​a character prefix in the current character prefix set, the first character is the character corresponding to a probability value in the classification result corresponding to the current time step, and the current character prefix set is the character prefix set corresponding to the decoding result of the previous time step; For any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, and the third character prefix is ​​concatenated with the first character in the character prefix and character combination. The concatenated character prefix is ​​then added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the probability value corresponding to the current time step in the classification result, which is located after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is located after the second character prefix.

13. The decoding device according to claim 12, characterized in that, Also includes: The first determining module is used to take the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step. The iterative processing module traverses all time steps except the first time step in sequence, and performs the following steps iteratively for any time step, wherein the set of character prefixes formed by the top elements of the heap that meet the iteration termination condition is the decoding result of the current time step.

14. The decoding device according to claim 12, characterized in that, The iteration terminates when the number of popped top elements is greater than or equal to the second number.

15. The decoding device according to claim 12, characterized in that, Also includes: The first adding module is used to concatenate the character prefix with the largest cumulative probability value in the current character prefix set with the character corresponding to the character with the largest probability value in the classification result corresponding to the current time step before the iterative processing module iteratively performs the following steps for at least one time step, and add the concatenated character prefix as a heap element to the max-heap.

16. The decoding apparatus according to claim 15, characterized in that, Also includes: Before the second determining module and the second adding module perform the following steps iteratively for at least one time step... The second determining module determines one or more sets of target character prefixes and character combinations based on the current character prefix set and the character corresponding to the probability value corresponding to the current time step in the classification result. Any set of target character prefixes and character combinations includes a target character and a target character prefix. The target character is the same as the last character of the target character prefix, or the target character is a blank character. The second adding module adds the character prefix obtained by concatenating the target character and the target character prefix in all target character prefixes and character combinations as a heap element to the max-heap. The value of the heap element is the sum of the cumulative probability values ​​of all character prefixes that are the same as the character prefix corresponding to the heap element.

17. The decoding apparatus according to claim 16, characterized in that, The second determining module includes: A construction module is used to construct a prefix dictionary and a suffix dictionary. The keys of the elements in the prefix dictionary are used to represent character prefixes, and the values ​​of the elements in the prefix dictionary are used to represent the order of character prefixes in the current character prefix set. The keys of the elements in the suffix dictionary are used to represent characters, and the values ​​of the elements in the suffix dictionary are used to represent the order of character prefixes ending with that character in the current character prefix set. The traversal module is used to traverse the characters corresponding to at least some of the probability values ​​in the classification results corresponding to the current time step, and determine one or more sets of target character prefixes and character combinations based on the prefix dictionary and suffix dictionary.

18. The decoding apparatus according to claim 12, characterized in that, Also includes: The filtering module is used to retain only the third largest probability value of the probability value corresponding to the current time step in the classification result before the iterative processing module iteratively performs the following steps for at least one time step.

19. The decoding apparatus according to claim 12, characterized in that, Also includes: The recording module is used to record the character prefixes and character combinations corresponding to the character prefixes added to the max-heap. The iterative processing module is further configured to determine whether the second character prefix and the second character, and the third character prefix and the first character in the character prefix and character combination, belong to the already recorded character prefix and character combination; if the second character prefix and the second character in the character prefix and character combination do not belong to the already recorded character prefix and character combination, then the character prefix obtained by concatenating the second character prefix and the second character in the character prefix and character combination is added to the max-heap, and / or if the third character prefix and the first character in the character prefix and character combination do not belong to the already recorded character prefix and character combination, then the character prefix obtained by concatenating the third character prefix and the first character in the character prefix and character combination is added to the max-heap.

20. The decoding device according to claim 12, characterized in that, The cumulative probability value of the character prefix includes a first cumulative probability value and a second cumulative probability value. The first cumulative probability value is the cumulative probability value when the character prefix ends with a whitespace character, and the second cumulative probability value is the cumulative probability value when the character prefix ends with a non-whitespace character.

21. The decoding apparatus according to claim 20, characterized in that, If the first character is a whitespace character, then the character prefix obtained by concatenating the second character prefix with the first character is the same as the second character prefix. The first cumulative probability value of the concatenated character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-whitespace character, and the first character is the same as the last character in the second character prefix, then the character prefix obtained by concatenating the first character and the second character prefix includes two character prefixes. One character prefix is ​​the same as the second character prefix, and the other character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the character prefix that is the same as the second character prefix is ​​equal to the product of the second cumulative probability value of the second character prefix and the probability value of the first character. The second cumulative probability value of the new character prefix is ​​equal to the product of the first cumulative probability value of the second character prefix and the probability value of the first character. If the first character is a non-whitespace character and the first character is different from the last character in the second character prefix, then the character prefix obtained by concatenating the first character with the second character prefix is ​​a new character prefix obtained by adding the first character after the second character prefix. The second cumulative probability value of the new character prefix is ​​equal to the product of the cumulative probability value of the second character prefix and the probability value of the first character.

22. A text recognition device for recognizing text contained in an input, characterized in that, include: The first processing module is used to process the input using an end-to-end model to obtain a classification result. The classification result is used to characterize the probability value of the character corresponding to different time steps in the input as a different character in the character solution space. The second processing module is used to take the character corresponding to the first number of largest probability values ​​among the probability values ​​corresponding to the first time step in the classification result as the decoding result of the first time step; The third processing module is used to traverse all remaining time steps in chronological order, and for any given time step, iteratively execute the following steps: pop the top element from the max-heap, where the max-heap includes at least one heap element, each heap element corresponds to a first character prefix, and the value of the heap element represents the cumulative probability value of the first character prefix. The first character prefix is ​​composed of at least one set of character prefixes and character combinations, including a second character prefix and a first character. The second character prefix is ​​a character prefix in the current character prefix set, and the first character is the character corresponding to a probability value in the classification result corresponding to the current time step. The previous character prefix set is the character prefix set corresponding to the decoding result of the previous time step; for any set of character prefixes and character combinations corresponding to the top element of the heap, the second character prefix in the character prefix and character combination is concatenated with the second character, the third character prefix is ​​concatenated with the first character in the character prefix and character combination, and the concatenated character prefix is ​​added as a heap element to the max-heap. The second character is the character corresponding to the first probability value in the probability value corresponding to the current time step in the classification result, which is located after the probability value corresponding to the first character. The third character prefix is ​​the first character prefix in the current character prefix set whose cumulative probability value is located after the second character prefix. as well as The determination module is used to take the character sequence represented by the first popped top element of the heap at the last time step as the text recognition result of the input.

23. A computing device, comprising: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 11.

24. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 11.

Images