Method, device and storage medium for locating each character in a recognized text line

Abstract

The present application discloses a method and device for locating each character in a recognized text line and a computer-readable storage medium. The method comprises: step S1: marking a core stroke for each character in the text line, wherein the mark indicates which character the stroke belongs to in the text line; step S2: marking unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and step S3: merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, and repeating steps S2 and S3 until all strokes are marked to characters in the text line.

Description

Technical Field

[0001] The present disclosure relates to the field of text recognition, and in particular to locating each character in a recognized text. Background Art

[0002] Word localization algorithm is a long-standing, fundamental and challenging research problem in the field of optical character recognition (OCR) and scene character recognition (STR). A good word localization algorithm can improve the accuracy of character recognition, tagging and classification of text line content.

[0003] So far, many algorithms for single word localization have been proposed. These algorithms can be mainly divided into the following four types: edge-based methods, texture-based methods, connected domain (CC)-based methods, and methods based on deep learning algorithms. Among them, edge-based methods mainly focus on how to use the high contrast between text and background to extract text, but this method will be affected by shadows and highlights. Similarly, texture-based methods use texture analysis algorithms to locate the distribution area of ​​text, and common ones include: spatial changes, Gabor filtering, and wavelet transforms. CC-based methods perform connected domain analysis on text line images to extract possible characters and further aggregate these character areas into a text segment. The above methods mainly focus on how to extract the distribution area of ​​characters from natural scene images, but cannot directly locate the specific area of ​​each character. In recent years, due to the rapid development of deep learning algorithms, neural networks, such as convolutional neural networks (CNNs), have been increasingly used to recognize and locate characters in text lines.

[0004] Furthermore, Convolutional Recurrent Neural Network (CRNN) is a new and very effective recognition algorithm, which is an end-to-end general framework and is very suitable for sequence scenarios of arbitrary length. Therefore, CRNN is widely used in the OCR field due to its high recognition accuracy. However, the single word localization performance of CRNN is not high and often has serious errors, so further improvement is needed. Summary of the invention

[0005] A brief overview of the disclosure is provided below in order to provide a basic understanding of certain aspects of the disclosure. It should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0006] According to one aspect of the present invention, a method for locating each character in a recognized text line is provided. The method comprises: step S1: marking a core stroke for each character in the text line, wherein the mark indicates which character in the text line the stroke belongs to; step S2: marking unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and step S3: merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, and repeating steps S2 and S3 until all strokes are marked to characters in the text line.

[0007] According to another aspect of the present invention, a device for locating each character in an identified text line is provided, comprising: a first marking device, which is configured to mark a core stroke for each character in the text line, the mark indicating which character in the text line the stroke belongs to; a second marking device, which is configured to mark unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and a merging device, which is configured to merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, wherein all strokes are marked to characters in the text line.

[0008] According to other aspects of the present invention, corresponding computer program codes, computer readable storage media and computer program products are also provided.

[0009] The method and device for locating each character in a recognized text line of the present invention can improve the positioning accuracy of a single character, thereby facilitating combination with other recognition engines to further improve text recognition accuracy.

[0010] These and other advantages of the present invention will become more apparent through the following detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to further illustrate the above and other advantages and features of the present disclosure, the specific embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings. The drawings together with the following detailed description are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only describe typical examples of the present disclosure and should not be regarded as limiting the scope of the present disclosure. In the drawings:

[0012] Figure 1 A flow chart showing a method for correcting the end position of each character in a recognized text line before marking core strokes according to one embodiment;

[0013] Figure 2a The soft-max value of each timestamp candidate obtained by CRNN recognition is shown;

[0014] Figure 2b CRNN and over-segmentation results are shown;

[0015] Figure 2c The new ending position of each character after correction is shown;

[0016] Figure 3 A method for locating each character in a recognized text line according to one embodiment of the present invention is shown;

[0017] Figure 4 A flow chart for marking core strokes according to one embodiment is shown;

[0018] Figure 5 A flow chart for marking joined strokes and isolated strokes according to one embodiment is shown;

[0019] Figure 6 A flow chart of merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other according to one embodiment is shown;

[0020] Figure 7a The result of marking the core strokes is shown exemplarily;

[0021] Figure 7b The results of the classification of the sticky strokes are shown exemplarily;

[0022] Figure 7c The results of isolated stroke classification are shown exemplarily;

[0023] Figure 7d The result of merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other is shown as an example;

[0024] Figure 7eThe result that all strokes are marked to the recognized characters is shown as an example;

[0025] Figure 8 is a block diagram of an apparatus for locating each character in a recognized line of text according to one embodiment;

[0026] Fig. 9 is a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or apparatus according to embodiments of the present invention may be implemented. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.

[0028] It is also necessary to explain here that, in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not very relevant to the present disclosure are omitted.

[0029] As mentioned above, existing text recognition methods are not satisfactory in terms of single word localization performance.

[0030] To overcome the deficiencies of the prior art, the present invention proposes a novel single-character localization algorithm based on CRNN and over-segmentation algorithm. First, CRNN is used to recognize the handwritten single-line text line and locate the position of each character. This localization result is relatively rough and can only provide a general character range. At the same time, the over-segmentation algorithm is also used to over-segment the image, that is, the image is segmented into one stroke after another. Then, using the soft-max scores generated during the CRNN recognition process, the end position of each character given by CRNN is appropriately corrected to improve the localization accuracy of CRNN. Next, each recognized character is initialized, and then the core stroke of each character in the text line is searched for and marked, so as to match at least one stroke as accurately as possible for each character in the text line. Finally, based on these core strokes and some prior knowledge, a suitable algorithm is used to mark each unmarked stroke so as to mark each stroke to each character recognized by CRNN, thereby obtaining the strokes corresponding to each character, that is, the distribution area of each character.

[0031] First, the following will be combined with Figure 1 and Figure 2a to Figure 2c to describe how to correct the end position of each character in the recognized text line before marking the core stroke, and this end position is obtained by using CRNN and the over-segmentation algorithm.

[0032] Figure 1 FIG. shows a flowchart of a method for correcting the end position of each character in the recognized text line before marking the core stroke according to an embodiment.

[0033] First, in step 101, the CRNN algorithm is used to recognize the image text line, and the over-segmentation algorithm is used to segment the strokes of the characters in the recognized text line.

[0034] Specifically, in this embodiment, as shown in Figure 2a and Figure 2b , first the CRNN algorithm is used to recognize the image text line to obtain the content of the text line, the possible candidates of each character and their corresponding confidence levels, and the end timestamp position of each character. Figure 2a FIG. shows the top ten candidates predicted at each timestamp and their corresponding confidence values in the 12th to 14th timestamps. Figure 2b The vertical lines in FIG. show the end timestamp positions corresponding to each character given by CRNN, that is, the positions with the highest probability in several adjacent timestamps. For example, in FIG. Figure 2b , the probability of the character "山" is the highest at the 12th timestamp, so the end position of the character "山" is the 12th timestamp. From Figure 2bIt can be seen that CRNN can provide a rough and less plausible end position for each character.

[0035] Next, the over-segmentation algorithm is used to segment each character recognized by CRNN into individual scattered strokes. The result is as follows: Figure 2b as shown in the box in .

[0036] from Figure 2b It can be seen that the end position of the single word provided by CRNN is not accurate. Preferably, the end position provided by CRNN can be slightly corrected in advance to correct some relatively obvious and simple errors.

[0037] Next, in step 102, a character in the text line recognized by the CRNN is selected as the current character, and the process proceeds to step 103 to determine whether the current character is the same as the next character.

[0038] If the current character is different from the next character (i.e., "No" in step 103), proceed to step 1031. In step 1031, find the next timestamp after the end position of the current character, and for the recognition result of the timestamp, determine whether the current character exists in the first P candidates and whether the corresponding confidence is greater than the threshold TH2.

[0039] If yes, proceed to step 1033, that is, re-dividing the next timestamp into the end position of the current character. Then, proceed to step 1036. In step 1036, determine whether the next timestamp of the new end position is the end position of the next character (identified by CRNN). If yes, the end position of the current character has been corrected, and proceed to step 104. If not, return to step 1031 and continue searching for the end position of the current character.

[0040] If the result of step 1031 is no, then proceed to step 1035 without correcting the end position of the current character. Then, proceed to step 104.

[0041] In step 104, it is determined whether the current character is the last character in the characters to be corrected. If so, the process ends, and the end positions of all the characters to be corrected are corrected. If not, the process proceeds to step 105. In step 105, the next character in the characters to be corrected is selected as the new current character and the process returns to step 103, and iterates and determines according to the same logic until the end positions of all the characters to be corrected are corrected.

[0042] Preferably, P = 2, and TH2 = 0.01. However, it should be understood that the values ​​of P and TH2 are not limited thereto, but can be set as required.

[0043] For example, as Figure 2a shown, the 12th timestamp has the highest confidence in the character '山', so this timestamp is also the end position of the character '山' recognized by CRNN. However, as can be seen from Figure 2b , this end position is obviously inaccurate because some strokes of the character '山' are not included. In fact, the 13th timestamp still belongs to the character '山', while the 14th timestamp does not belong to the character '山'.

[0044] To correct the end position of the character '山', in step 103, according to the result recognized by CRNN (as Figure 2a shown), starting from its end timestamp 12, each timestamp is searched sequentially backward. Through the search, it is found that the character '山' exists in the second candidate of the 13th timestamp, and the confidence of this candidate is 0.02, which is higher than the threshold 0.01. Then it is considered that the 13th timestamp still belongs to the character '山'. Further, it can be found that in the 14th timestamp, the confidence of the character '山' is lower than the threshold 0.01. Therefore, the 13th timestamp instead of the 14th timestamp is marked as the end position of the character '山', that is, the end position of the character '山' is extended backward to the 13th timestamp. The corrected end position is as Figure 2c shown.

[0045] When the current character is the same as its next character (i.e., 'yes' in step 103), continue to step 1032. In step 1032, for each timestamp between the end positions of these two identical characters, determine whether the current character exists in the first Q candidates of each timestamp. If so, proceed to step 104 to determine whether the current character is the last character among the characters to be corrected. If so, the process ends, and the end positions of all characters to be corrected have been corrected once. If not, continue with step 105. In step 105, select the next character among the characters to be corrected as the new current character and return to step 103, and perform iterative judgment according to the same logic until the end positions of all characters to be corrected are corrected.

[0046] It should be noted that in order to prevent the wrong confusion effect of overlapping characters on this correction method, it is necessary to detect the text recognized by CRNN in advance. If two identical characters appear consecutively at a certain position, then it is necessary to perform confusion detection, that is, step 1032. If the character always appears in the first Q candidates within all timestamps between the end positions of these two characters, it is considered that the information is confused, and this correction method cannot correct it accurately. Therefore, the end position of the first character (i.e., the current character) in the overlapping characters is not corrected.

[0047] Preferably, Q=5, P and Q are positive integers greater than 1, and P is smaller than Q. However, it should be understood that the values ​​of P and Q are not limited thereto, but can be set as required.

[0048] Through the above correction method, the end position of the characters in the text line recognized by CRNN can be corrected to the correct end position, thereby improving the positioning performance of CRNN.

[0049] Combine the following Figures 3 to 7e A method 300 for locating each character in a recognized text line according to one embodiment of the present invention is described.

[0050] like Figure 3 As shown, first, in step S1, a core stroke is marked for each character in a text line, and the mark indicates to which character in the text line the stroke belongs.

[0051] Preferably, in this embodiment, the text line is a text line recognized by CRNN. ​​However, it should be understood that the present invention is not limited thereto, and the text line may also be a text line recognized by any other existing recognition engine, without or after position correction.

[0052] In order to establish a correspondence between each segmented stroke and the character recognized by CRNN, it is first necessary to find and mark at least one core stroke for each character recognized by CRNN. ​​The so-called core stroke refers to the stroke that is most likely to belong to the character among all the strokes of a character. Since the selection method of the core stroke will seriously affect the performance of the subsequent steps, only those strokes that are most likely to belong to the character are selected as the core strokes in step S1.

[0053] Based on the end position of each character given by CRNN, we can get a rough distribution range of each character, which is called the recognition range of the character. Although this range is often not very accurate in the edge area, it is often more accurate in some core ranges (such as the middle right area). In addition, it is obvious that the higher the overlap area between the stroke and the character recognition range, the more likely the stroke belongs to the character.

[0054] Figure 4 A flowchart of marking core strokes (ie, step S1 ) according to one embodiment is shown.

[0055] In step 401, strokes whose stroke ranges include or are included in the recognition range of a character in a text line and overlap with the core range of the character are marked as core strokes of the character.

[0056] Specifically, in this embodiment, if a stroke overlaps with the core area of ​​a character and the stroke is completely contained in the CRNN recognition range of the character, or conversely, the distribution range of the stroke completely covers the CRNN recognition range of the character, then the stroke is marked as the core stroke of the character.

[0057] Preferably, if the CRNN recognition range of a character is set to 0 to 1 in the horizontal direction, the interval of 0.4 to 0.8 within the recognition range of the character is generally regarded as the core area of ​​the character. It should be understood that the interval value is only an example and the present invention is not limited thereto.

[0058] Next, in step 402, it is determined whether there is a character in the text line that still has no core strokes marked. If yes, then in step 403, the unmarked strokes overlapping the core range of the character are marked as the core strokes of the character. If not, proceed to step S2 which will be described in detail below.

[0059] Specifically, in this embodiment, after step 401 is performed, if there are still characters without core strokes in the text line, the constraint condition needs to be slightly relaxed to search among the remaining unmarked strokes, so as to find a stroke with the highest probability as the core stroke for these characters. Therefore, in step 403, the unmarked strokes overlapping with the core range of the character are marked as the core strokes of the character.

[0060] Next, in step 404, it is determined whether there is a character in the text line that has no core strokes marked. If yes, then in step 405, the unmarked strokes that have the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character. If not, proceed to step S2 which will be described in detail below.

[0061] It should be noted that after step 405, if there are still characters without core strokes marked in the text line, it is considered that CRNN has a serious recognition or positioning error on the character, so the character is deleted and not positioned.

[0062] It should also be noted that in step 403, if a stroke is marked as a core stroke of multiple characters at the same time, the stroke is not marked as a core stroke of any character. Similarly, in steps 401 and 405, if a stroke is marked as a core stroke of multiple characters at the same time, the stroke is marked as the core stroke of the leftmost character.

[0063] Figure 7a Shown by Figure 4 The method shown shows an example of a text line with core strokes marked, where dark and light colors represent core strokes corresponding to different characters, and gray represents unmarked strokes.

[0064] return Figure 3 After marking the core strokes for each character in the text line, proceed to step S2. In step S2, based on the marked strokes, unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes are marked, wherein an isolated stroke refers to the only unmarked stroke between two marked strokes.

[0065] Specifically, in this embodiment, after marking the core strokes of each character as much as possible, it is necessary to mark other unmarked strokes based on these marked strokes. Strokes with the same mark can be regarded as a whole, that is, a set of marked strokes of a category.

[0066] Figure 5 A flow chart for marking glued strokes and isolated strokes according to one embodiment is shown.

[0067] like Figure 5 As shown, in step 501, if an unmarked stroke is adhered to a marked stroke, the unmarked stroke is merged into the set of marked strokes.

[0068] It should be noted that in some special cases, some strokes may be adhered to more than two marked stroke sets. At this time, since it is impossible to judge their categories, they will not be classified and fused for the time being.

[0069] Figure 7b An example of classification of sticky strokes is shown. Figure 7b As shown, all connected strokes are classified into the recognized characters to which they belong.

[0070] Next, in step 502, the unmarked isolated stroke is merged into the marked stroke that is closest to it.

[0071] Specifically, in this embodiment, an isolated stroke refers to the only unmarked stroke between two marked strokes. In this case, the unmarked stroke is called an isolated stroke. Obviously, when an isolated stroke exists, the stroke belongs to either the set to which the left stroke belongs or the set to which the right stroke belongs. Therefore, the isolated strokes can be classified by the following distance formula:

[0072] D=max(s1,c1)-min(s2,c2)+1 (1)

[0073] Among them, s1, s2 represent the pixel coordinates of the left and right edges of the isolated stroke, respectively, and c1, c2 represent the pixel coordinates of the left and right edges of the object from which the distance is calculated.

[0074] It should be noted that the object for calculating the distance can be another marked stroke, or a set of marked strokes, or the recognition range of the characters provided by the CRNN.

[0075] It should also be noted that when D > 1, it indicates that there is no overlapping area between the two objects; when D ≤ 1, it indicates that there is an overlapping area between the two, and the width of the overlapping area is 2 - D pixels.

[0076] It should be understood that due to the certain randomness in writing characters, when the distance between an isolated stroke and two adjacent strokes is very close, that is, the absolute value of the difference between the two distances is less than the threshold TH1, it is impossible to determine the attribution of the isolated stroke only based on the distance information. At this time, new information needs to be introduced. In response to this situation, the method according to this embodiment will calculate the overlapping width between the isolated stroke and the recognition ranges of the two adjacent characters according to formula (1), and assign the isolated stroke to the character category with the largest overlapping width.

[0077] Preferably, the threshold TH1 is 8 pixels. However, it should be understood that the present invention is not limited thereto, but the value of TH1 can be set as needed.

[0078] Finally, in step 503, it is determined whether all the adhered strokes and isolated strokes are marked. If so, continue with step S3 which will be described below. If not, return to step 501.

[0079] Specifically, in this embodiment, since new adhered or isolated strokes may be generated after classifying the adhered strokes and isolated strokes, steps 501 and 502 need to be iterated until no new adhered or isolated strokes are generated.

[0080] Figure 7c An example of classifying isolated strokes is shown. In this example, the stroke "1" in the middle right of the character "息" is closer to the character "息", so it is classified into the category of the character "息".

[0081] Return Figure 3 , and continue with step S3. In step S3, the first N pairs or the first M% pairs of the two adjacent strokes that are closest to each other are merged together, where the two adjacent strokes are not marked to different characters, and where N is an integer greater than or equal to 1, and M is any value between 0 and 100.

[0082] Specifically, in this embodiment, after steps S1 and S2, there may still be some strokes that have not been classified. At this time, only a greedy strategy can be used to classify them.

[0083] Figure 6A flow chart of merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other according to one embodiment is shown.

[0084] First, in step 601, all strokes with the same mark obtained in steps S1 and S2 are merged together and regarded as one stroke.

[0085] Next, in step 602, the distance between two adjacent strokes is calculated. Specifically, in this embodiment, for example, the distance between all adjacent strokes can be calculated according to formula (1).

[0086] Next, in step 603, the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other are merged together, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100. Specifically, in this embodiment, all the distances calculated in step 602 are sorted according to size, the smallest first N or first M% distances are selected, and the adjacent stroke pairs corresponding to the distances are merged into one category, that is, they belong to the same character.

[0087] It should be noted that if the selected adjacent stroke pair has been marked to two different CRNN recognized characters, then this distance is skipped and not selected. In addition, the strategy for classifying and merging strokes is similar to the strategy for classifying isolated strokes, that is, first, based on the distance between the strokes, merge the strokes into the adjacent strokes with a closer distance; if the strokes are close to the two distances of the left and right strokes, then classify the stroke into the character with a greater overlap.

[0088] It should also be noted that when calculating the distance, it is possible that several unlabeled strokes will be merged with each other, thus generating a new category that does not exist in the characters recognized by CRNN. ​​Therefore, this new category is only an intermediate variable during fusion and merging. After multiple iterations in the entire process, these newly generated temporary categories will gradually merge into the recognized character categories generated by CRNN, and the final result category will only correspond to the recognized characters generated by CRNN.

[0089] Figure 7d An example of merging the first N pairs or the first M% pairs of closest strokes is shown. In this example, N is selected as 1, and the stroke "1" is closest to the stroke set "け", so they are marked as the same category here and merged.

[0090] return Figure 3 Finally, in step S4, it is determined whether there are still unmarked strokes. If yes, the process returns to step S2. If no, the method 300 ends.

[0091] Specifically, in this embodiment, it is determined whether there are strokes that have not been marked on the characters recognized by CRNN. ​​If so, return to step S2. After multiple iterative calculations, all strokes are classified and the newly generated temporary categories are merged and disappeared. Thus, all strokes are classified and marked on the characters recognized by CRNN. ​​In other words, the position information corresponding to all recognized characters is accurately located.

[0092] Figure 7e An example in which all strokes are marked to the recognized character is shown. Figure 7e As shown, all strokes corresponding to the recognized characters are found.

[0093] The method 300 according to this embodiment allocates each stroke to its corresponding recognized character with high accuracy based on the end position of the CRNN and the spatial distribution between the strokes, thereby achieving the stroke-level positioning of each character. The positioning result makes it possible to fuse the recognition results between multiple recognition engines.

[0094] By testing the method 300 according to the present embodiment using multiple handwritten Japanese data sets (AIBU, Questionnaire, Cogent and PFU), the recognition accuracy shown in the following Table 1 is obtained. As can be seen from Table 1, the average accuracy of the method 300 is about 95%, thus providing a high degree of precision.

[0095] Dataset (number of images) AIBU(897) Questionnaire(319) Cogent(26) PFU(25) Total(1267) Accuracy 95.0％ 94.2％ 93.5％ 94.9％ 95.0％

[0096] Table 1

[0097] The methods discussed above may be implemented entirely by a computer executable program, or may be implemented partially or entirely by hardware and / or firmware. When implemented by hardware and / or firmware, or when a computer executable program is loaded into a hardware device that can run the program, the device described below for locating each character in a recognized text line is implemented. Below, an overview of these devices is given without repeating some of the details discussed above, but it should be noted that although these devices can perform the methods described above, the methods do not necessarily use those components of the described devices or are not necessarily performed by those components.

[0098] Figure 8A device 800 for locating each character in a recognized text line according to one embodiment is shown, which includes a first marking device 801, a second marking device 802 and a merging device 803. The first marking device 801 is used to mark the core strokes of each character in the text line, and the mark indicates which character the stroke belongs to in the text line. The second marking device 802 is used to mark the unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes. The merging device 803 is used to merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100. After being processed by the second marking device and the merging device, all strokes are marked to the characters in the text line.

[0099] Figure 8 The device 800 shown for locating each character in a recognized text line corresponds to Figure 3 The method 300 for locating each character in a recognized text line is shown in FIG. Therefore, the relevant details of each device in the apparatus 800 for locating each character in a recognized text line have been described in detail. Figure 3 The method 300 for locating each character in a recognized text line is described in detail and will not be repeated here.

[0100] Each component module or unit in the above device can be configured by software, firmware, hardware or a combination thereof. The specific means or methods that can be used for configuration are well known to those skilled in the art and will not be described in detail here. In the case of software or firmware implementation, the data is transmitted from a storage medium or a network to a computer (e.g., a computer) with a dedicated hardware structure. Fig. 9 The general-purpose computer 900 shown in the figure is installed with the programs constituting the software, and when the various programs are installed, the computer can execute various functions, etc.

[0101] Fig. 9 is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or devices according to embodiments of the present invention may be implemented. Fig. 9 As shown, a central processing unit (CPU) 901 performs various processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 to a random access memory (RAM) 903. In the RAM 903, data required when the CPU 901 performs various processes and the like is also stored as needed. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output interface 905 is also connected to the bus 904.

[0102] The following components are connected to the input / output interface 905: an input section 906 (including a keyboard, a mouse, etc.), an output section 907 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage section 908 (including a hard disk, etc.), a communication section 909 (including a network interface card such as a LAN card, a modem, etc.). The communication section 909 performs communication processing via a network such as the Internet. A drive 910 may also be connected to the input / output interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 910 as needed, so that a computer program read out therefrom is installed in the storage section 908 as needed.

[0103] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 911 .

[0104] It should be understood by those skilled in the art that such storage media is not limited to Fig. 9 The removable medium 911 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 911 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 902, a hard disk included in the storage section 908, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0105] The present invention also provides a corresponding computer program code and a computer program product storing a machine-readable instruction code. When the instruction code is read and executed by a machine, the method 300 according to the embodiment of the present invention can be executed.

[0106] Accordingly, storage media configured to carry the program product storing the machine-readable instruction code are also included in the disclosure of the present invention, including but not limited to floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.

[0107] Through the above description, the embodiments of the present disclosure provide the following technical solutions, but are not limited thereto.

[0108] Note 1. A method for locating each character in a recognized text line, comprising:

[0109] Step S1: marking a core stroke for each character in the text line, wherein the mark indicates to which character in the text line the stroke belongs;

[0110] Step S2: based on the marked strokes, marking the unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and

[0111] Step S3: Merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100,

[0112] Steps S2 and S3 are repeated until all strokes are marked to characters in the text line.

[0113] Note 2. According to the method of Note 1, the marking core strokes include:

[0114] Marking the strokes whose stroke ranges include or are included in the recognition range of the character in the text line and overlap with the core range of the character as the core strokes of the character,

[0115] For a character in the text line whose core strokes are still not marked, marking the unmarked strokes overlapping the core range of the character as the core strokes of the character, and

[0116] For the characters in the text line whose core strokes are still not marked, the unmarked strokes having the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character.

[0117] Note 3. According to the method in Note 2,

[0118] If a stroke overlapping with the core range of the character is marked as a core stroke of multiple characters at the same time, the stroke is not marked as a core stroke of any character.

[0119] And wherein, if a stroke range includes or is included in the recognition range of the character and overlaps with the core range of the character, or a stroke that has the largest overlap ratio with the recognition range of the character is marked as the core stroke of multiple characters at the same time, then the stroke is marked as the core stroke of the leftmost character.

[0120] Note 4. According to the method of Note 3, wherein the recognition range is set to 1, then the core range is 0.4 to 0.8.

[0121] Note 5. According to the method of Note 1, marking the unmarked contiguous strokes and unmarked isolated strokes based on the marked strokes comprises:

[0122] If the unmarked sticky stroke is stuck to the marked stroke, merging the unmarked sticky stroke into the marked stroke; and

[0123] Merging the unmarked isolated stroke into the marked stroke that is closest to it; and

[0124] Repeat the above steps until all the glued strokes and isolated strokes are marked.

[0125] Note 6. According to the method of Note 1, step S3 further comprises:

[0126] Merge all strokes with the same mark obtained in step S1 and step S2;

[0127] Calculate the distance between two adjacent strokes; and

[0128] Merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other.

[0129] Note 7. A method according to Note 5 or 6, wherein the distance is based on the difference between the maximum value between the leftmost position of one stroke and the leftmost position of the other stroke of two adjacent strokes and the minimum value between the rightmost position of the one stroke and the rightmost position of the other stroke.

[0130] Note 8. A method according to any one of Notes 1 to 6, wherein:

[0131] If the absolute value of the difference between a stroke and its two adjacent strokes is less than a first threshold, the stroke is marked to the character whose recognition range overlaps the stroke's range the most; and if the absolute value of the difference between a stroke and its two adjacent strokes is greater than the first threshold, the stroke is marked to the character closest to it.

[0132] Note 9. The method according to Note 8, wherein the first threshold is 8 pixels.

[0133] Note 10. A method according to any one of Notes 1 to 6, wherein the recognized text lines are obtained through a convolutional recurrent neural network, and the strokes of the characters are obtained through an over-segmentation algorithm.

[0134] Note 11. The method according to Note 10 also includes correcting the end position of each character in the recognized text line obtained by the convolutional recurrent neural network before marking the core strokes.

[0135] Note 12. The method according to Note 11, wherein the modification comprises:

[0136] In the case where the character in question is different from the character that follows it:

[0137] If it is determined that the character exists in the first P candidates of the recognition result of the next timestamp after the end position of the character and the corresponding confidence is greater than the second threshold, the next timestamp is re-divided as the end position of the character,

[0138] Iterating the re-dividing step until moving to a timestamp before the end position of the next character of the character; and

[0139] In the case where the character in question is the same as the next character:

[0140] If the character exists in the first Q candidates for each timestamp between the two identical character end positions, the end position of the character is not changed, and

[0141] If the character does not exist in the first Q candidates of each timestamp between the end positions of the two identical characters, the same processing as the case where the character is different from its next character is performed.

[0142] Wherein, P and Q are positive integers greater than 1 and P is less than Q.

[0143] Note 13. The method according to Note 12, wherein the second threshold is 0.01.

[0144] Note 14. The method according to any one of Notes 1 to 6 also includes using the final positioning result to fuse text lines recognized by different recognition models to obtain accurate recognition results.

[0145] Note 15. A device for locating each character in a recognized text line, comprising:

[0146] a first marking device configured to mark a core stroke for each character in the text line, the mark indicating to which character in the text line the stroke belongs;

[0147] A second marking device is configured to mark unmarked contiguous strokes and unmarked isolated strokes contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and

[0148] A merging device configured to merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100,

[0149] After being processed by the second marking device and the merging device, all strokes are marked as characters in the text line.

[0150] Note 16. The device according to Note 15, wherein the first marking device is further configured to:

[0151] Marking the strokes whose stroke ranges include or are included in the recognition range of the character in the text line and overlap with the core range of the character as the core strokes of the character,

[0152] For a character in the text line whose core strokes are still not marked, marking the unmarked strokes overlapping the core range of the character as the core strokes of the character, and

[0153] For the characters in the text line whose core strokes are still not marked, the unmarked strokes having the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character.

[0154] Note 17. The device according to Note 15 or 16, wherein the second marking device is further configured to:

[0155] If the unmarked sticky stroke is stuck to the marked stroke, merging the unmarked sticky stroke into the marked stroke; and

[0156] Merging the unmarked isolated stroke into the marked stroke that is closest to it; and

[0157] The above process is repeated until all the connected strokes and isolated strokes are marked.

[0158] Note 18. The device according to Note 17, wherein the merging device is further configured to:

[0159] Merge all strokes with the same mark together;

[0160] Calculate the distance between two adjacent strokes; and

[0161] Merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other.

[0162] Note 19. The device according to Note 15 further includes a correction device, which is configured to correct the end position of each character in the recognized text line obtained by the convolutional recurrent neural network before marking the core strokes.

[0163] Note 20. A computer-readable storage medium storing a program that can be executed by a processor to perform the following operations:

[0164] Operation S1: marking a core stroke for each character in the text line, wherein the mark indicates to which character in the text line the stroke belongs;

[0165] Operation S2: based on the marked strokes, marking unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and

[0166] Operation S3: merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100,

[0167] Operations S2 and S3 are repeated until all strokes are marked to characters in the text line.

[0168] Finally, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In addition, in the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or device that includes the elements.

[0169] Although the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only configured to illustrate the present invention and do not constitute a limitation of the present invention. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present invention. Therefore, the scope of the present invention is limited only by the attached claims and their equivalent meanings.

Claims

1. A method for locating each character in a recognized text line, comprising: Step S1: marking a core stroke for each character in the text line, wherein the mark indicates to which character in the text line the stroke belongs; Step S2: based on the marked strokes, marking the unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and Step S3: Merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, Repeat steps S2 and S3 until all strokes are marked to the characters in the text line. The marking core strokes include: Marking the strokes whose stroke ranges include or are included in the recognition range of the character in the text line and overlap with the core range of the character as the core strokes of the character, For a character in the text line whose core strokes are still not marked, marking the unmarked strokes overlapping the core range of the character as the core strokes of the character, and For the characters in the text line whose core strokes are still not marked, the unmarked strokes having the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character.

2. The method according to claim 1, in, If a stroke overlapping with the core range of the character is marked as a core stroke of multiple characters at the same time, the stroke is not marked as a core stroke of any character. And wherein, if a stroke range includes or is included in the recognition range of the character and overlaps with the core range of the character, or a stroke that has the largest overlap ratio with the recognition range of the character is marked as the core stroke of multiple characters at the same time, then the stroke is marked as the core stroke of the leftmost character.

3. The method according to claim 1, wherein: Based on the marked strokes, marking the unmarked connected strokes and the unmarked isolated strokes comprises: If the unmarked sticky stroke is stuck to the marked stroke, merging the unmarked sticky stroke into the marked stroke; and Merging the unmarked isolated stroke into the marked stroke that is closest to it; and Repeat the above steps until all the glued strokes and isolated strokes are marked.

4. The method according to claim 1, wherein: The step S3 further comprises: Merge all strokes with the same mark obtained in step S1 and step S2; Calculate the distance between two adjacent strokes; and Merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other.

5. The method according to claim 3 or 4, wherein: The distance is based on a difference between a maximum value among the leftmost position of one stroke and the leftmost position of the other stroke of two adjacent strokes and a minimum value among the rightmost position of the one stroke and the rightmost position of the other stroke.

6. The method according to any one of claims 1 to 4, wherein: If the absolute value of the difference between a stroke and its two adjacent strokes is less than a first threshold, the stroke is marked as the character whose recognition range overlaps the stroke's range the most; as well as If the absolute value of the difference between a stroke and its two adjacent strokes is greater than the first threshold, the stroke is marked as the character closest to it.

7. The method according to claim 6, further comprising modifying the end position of each character in the recognized text line obtained by the recognition engine before marking the core strokes, the modification comprising: In the case where the character in question is different from the character that follows it: If it is determined that the character exists in the first P candidates of the recognition result of the next timestamp after the end position of the character and the corresponding confidence is greater than the second threshold, the next timestamp is re-divided as the end position of the character, Iterating the re-dividing step until moving to a timestamp before the end position of the next character of the character; and In the case where the character in question is the same as the next character: If the character exists in the first Q candidates for each timestamp between the two identical character end positions, the end position of the character is not changed, and If the character does not exist in the first Q candidates of each timestamp between the end positions of the two identical characters, the same processing as the case where the character is different from its next character is performed. Wherein, P and Q are positive integers greater than 1 and P is less than Q.

8. An apparatus for locating each character in a recognized line of text, comprising: a first marking device configured to mark a core stroke for each character in the text line, the mark indicating to which character in the text line the stroke belongs; A second marking device is configured to mark unmarked contiguous strokes and unmarked isolated strokes contiguous with the marked strokes based on the marked strokes, wherein the isolated stroke refers to the only unmarked stroke between two marked strokes; and A merging device configured to merge the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, wherein, after being processed by the second marking device and the merging device, all strokes are marked as characters in the text line, and Wherein, the first marking device is further configured to: Marking the strokes whose stroke ranges include or are included in the recognition range of the character in the text line and overlap with the core range of the character as the core strokes of the character, For a character in the text line whose core strokes are still not marked, marking the unmarked strokes overlapping the core range of the character as the core strokes of the character, and For the characters in the text line whose core strokes are still not marked, the unmarked strokes having the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character.

9. A computer-readable storage medium storing a program that can be executed by a processor to perform the following operations: Operation S1: marking a core stroke for each character in a text line, wherein the mark indicates to which character in the text line the stroke belongs; Operation S2: Based on the marked strokes, unmarked contiguous strokes and unmarked isolated strokes that are contiguous with the marked strokes are marked, wherein: The isolated stroke refers to the only unmarked stroke between two marked strokes; and Operation S3: merging the first N pairs or the first M% pairs of two adjacent strokes that are closest to each other, wherein the two adjacent strokes are not marked to different characters, and wherein N is an integer greater than or equal to 1, and M is any value between 0 and 100, Repeat operations S2 and S3 until all strokes are marked to characters in the text line. The marking core strokes include: Marking the strokes whose stroke ranges include or are included in the recognition range of the character in the text line and overlap with the core range of the character as the core strokes of the character, For a character in the text line whose core strokes are still not marked, marking the unmarked strokes overlapping the core range of the character as the core strokes of the character, and For the characters in the text line whose core strokes are still not marked, the unmarked strokes having the largest overlap ratio with the recognition range of the character are marked as the core strokes of the character.

Images