Character coordinate extraction method and device, equipment and storage medium

Abstract

This invention discloses a method for extracting character coordinates. The method includes: inputting a target text image into a feature extraction backbone network; obtaining character segmentation features and text line segmentation features through feature fusion of different layers in the backbone network; inputting the character segmentation features and text line segmentation features into a text line segmentation module and a character segmentation module, respectively, to obtain a character segmentation heatmap and a text segmentation heatmap of the target image; and calculating the coordinates of a single character based on the character segmentation heatmap and the text segmentation heatmap. Through this method, this invention avoids redundant feature extraction; exhibits high robustness for character segmentation; accelerates network convergence and improves network segmentation efficiency; and improves the accuracy of single-character coordinate extraction by back-deriving coordinates based on CTC recognition results and combining it with character segmentation methods.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically to a method, apparatus, device, and storage medium for extracting character coordinates. Background Technology

[0002] Current known methods for extracting single-character coordinates in text mainly include: segmenting the target image to obtain independent connected components, then determining whether each connected component contains connected characters, detecting the contours of connected characters to obtain the center positions of closed regions within the characters, and then segmenting the connected characters to obtain the position of individual characters. This paper designs a text line recognition network based on an attention mechanism and trains the recognition model. The text line image to be segmented is input into the recognition model, and the single-character segmentation result is calculated through the weight probability distribution of the attention mechanism, ultimately obtaining the position information and recognition result of each character.

[0003] The above method first segments the target image to obtain independent connected components. Then, it determines whether each connected component contains connected characters based on the width and height of the character region occupied by each character in the target image. When a connected component containing connected characters exists, the center position of the closed region within the connected characters is determined. The center position of the connected characters is obtained based on the center position of the closed region, and the connected characters are segmented to obtain individual character and position information. This method determines the presence of connected characters by judging the width and height of characters in the connected component. However, for mixed Chinese and English text, the width of English characters differs from that of Chinese characters, making it impossible to determine whether characters are connected by width alone. Furthermore, segmenting connected characters requires using the center position of the closed region of the connected characters, but most common characters do not contain closed regions, thus presenting significant limitations.

[0004] Currently known methods for extracting single-character coordinates in text include: collecting text line training data; normalizing the image size; augmenting the training image; creating an attention-based text line recognition model; training the recognition model with a large amount of training data; inputting the text line image to be segmented into the recognition model; and calculating the single-character segmentation result through the weight probability distribution of the attention mechanism. This attention-based method suffers from attention drift, which affects the recognition result. While primarily used for training the recognition model, its accuracy in single-character segmentation is significantly influenced by the recognition model. When missed characters occur during recognition, it negatively impacts the accuracy of single-character segmentation, resulting in poor robustness. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention provide a method, apparatus, device and storage medium for extracting character coordinates with a wider range of applicability and higher robustness.

[0006] A first aspect of this invention provides a method for extracting the coordinates of a character, the method comprising the steps of:

[0007] The target text image is input into the feature extraction backbone network, and character segmentation features and text line segmentation features are obtained by feature fusion of different layers in the backbone network;

[0008] The character segmentation features and text line segmentation features are input into the text line segmentation module and the character segmentation module, respectively, to obtain the character segmentation heatmap and text segmentation heatmap of the target text image;

[0009] Based on the character segmentation heatmap and the text segmentation heatmap, the coordinates of a single character in the target text image are calculated.

[0010] In one alternative approach, the target text image is input into a feature extraction backbone network, and character segmentation features and text line segmentation features are obtained through feature fusion of different layers in the backbone network. The specific steps include:

[0011] The target text image is input into the feature extraction backbone network;

[0012] Feature maps of the target text image are extracted from the feature extraction backbone network;

[0013] The extracted feature maps are fused using FPN to obtain the character segmentation features and text line segmentation features.

[0014] In one optional approach, the step of inputting the character segmentation features and text line segmentation features into a segmentation network model to obtain a character segmentation heatmap and a text segmentation heatmap of the target text image, wherein the segmentation network model includes a single character segmentation network and a text line region segmentation network, specifically including the following steps:

[0015] The character segmentation features are input into the single character segmentation network to obtain a character segmentation probability map and a character segmentation threshold map.

[0016] The character segmentation heat map is calculated based on the difference between the character segmentation probability map and the character segmentation threshold map;

[0017] The text line segmentation features are input into the text line region segmentation network to obtain a text line segmentation probability map and a text line segmentation threshold map.

[0018] The text line segmentation heatmap is calculated based on the difference between the text line segmentation probability map and the text line segmentation threshold map.

[0019] In an alternative approach, calculating the coordinates of a single character based on the character segmentation heatmap and the text segmentation heatmap further includes the step of:

[0020] The detection box position information of the text line is obtained by using the text line segmentation heatmap;

[0021] The character segmentation heatmap is cropped according to the detection box position information of the text line to obtain the text line image;

[0022] The text line image is segmented using the watershed algorithm to form segmentation images, and the number of segmentation images is obtained.

[0023] The number of characters in the text line image is identified using CTC;

[0024] The number of segmented images obtained by the watershed algorithm is compared with the number of characters identified by CTC;

[0025] When the number of segmentation images is the same as the number of characters, the position information of each character is obtained through the watershed algorithm;

[0026] The position information of each character is restored to the target text image to obtain the coordinates of each character;

[0027] When the number of segmentation images is not the same as the number of characters, extract the coordinates of single characters from CTC.

[0028] In an alternative approach, when the number of segmentation images is not the same as the number of characters, extracting single-character coordinates from CTC specifically includes the following steps:

[0029] Based on CTC, the text line image is uniformly segmented to form several segmented image blocks.

[0030] The segmented image blocks are identified to obtain the character corresponding to each segmented image block. Segments that cannot be identified are marked as special characters.

[0031] The segmented image blocks corresponding to the same character are merged to form a merged image block;

[0032] The image block is divided at half its length to obtain the segmentation result for each character;

[0033] The character segmentation results are mapped to the text line image to obtain text boxes, and finally, the single-character coordinate information based on CTC is obtained.

[0034] In an alternative approach, the step of training a segmentation network model is further included: training the segmentation network model, wherein training the segmentation network model further includes:

[0035] Prepare training data, which needs to be labeled with the position information of each character and the position information of the entire text line. The position information of each character is used to train a single character segmentation network, and the position information of the entire text line is used to train a text line region segmentation network.

[0036] In an alternative approach, the trained segmentation network model further includes:

[0037] Design a joint training loss function, and train the segmentation network model using the joint training loss function;

[0038] The formula for calculating the joint training loss function is as follows:

[0039] Loss = αloss char +βloss textline ;

[0040] Where α and β are constant coefficients;

[0041] loss char and loss textline Segmentation map loss L, which includes characters and text lines respectively. S Threshold map loss L t :

[0042] loss char =α1L S1 +β1L t1 ;

[0043] loss textline =α2L S2 +β2L t2 ;

[0044] Where α1, α2, β1, and β2 are constant coefficients;

[0045] The segmentation probability map in the joint training loss function adopts the binary classification cross-entropy loss function, and the loss function L... S1 L S2 The inputs are the sample predicted probability map and the sample true label map:

[0046]

[0047] Among them, S l For the sample set, x i Let y be the probability value of a pixel in the sample prediction image. i This represents the true value of a pixel in the sample's true label.

[0048] Loss function L t1 L t2The input consists of a threshold map of the predicted text lines and a sample ground truth map. The threshold map uses the L1 distance loss function.

[0049]

[0050] Among them, R d This is the set of pixel indices in the threshold map. For label values, These are predicted values.

[0051] According to another aspect of the present invention, a character coordinate extraction apparatus is provided, comprising:

[0052] The target text image input module is used to input the target text image into the feature extraction backbone network;

[0053] The segmentation feature acquisition module is used to acquire character segmentation features and text line segmentation features;

[0054] The segmentation feature input module is used to input the character segmentation features and text line segmentation features into the text line segmentation module and the character segmentation module, respectively.

[0055] The character segmentation heatmap module is used to obtain the character segmentation heatmap of the target text image;

[0056] The text segmentation heatmap module is used to obtain the text segmentation heatmap of the target text image;

[0057] The coordinate calculation module is used to calculate the coordinates of a single character based on the character segmentation heatmap and the text segmentation heatmap.

[0058] A second aspect of the present invention provides a coordinate extraction device for a single character, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0059] The memory is used to store at least one executable instruction that causes the processor to perform the operation of any of the above-described character coordinate extraction methods.

[0060] A third aspect of the present invention provides a computer-readable storage medium storing at least one executable instruction, which, when executed on a single character coordinate extraction device / apparatus, causes the single character coordinate extraction device / apparatus to perform the operation of any of the above-described character coordinate extraction methods.

[0061] This invention integrates a single character segmentation network and a text line region segmentation network, sharing a common feature extraction backbone network, into a single neural network, thus avoiding redundant feature extraction. A parallel network model simultaneously segments text lines and character regions. The segmentation network simultaneously generates text line probability maps, text line threshold maps, character region probability maps, and character region threshold maps. Image interpolation ultimately yields heatmaps of character regions and text line regions, demonstrating high robustness for character segmentation. During the training of the character segmentation and text line segmentation networks, a joint training loss function for characters and text lines is employed, accelerating network convergence and improving segmentation efficiency. In the extraction of single character coordinates, the character segmentation network obtains most of the character coordinate information. When some text segmentation images are contiguous, a method based on CTC (Characteristic Character Recognition) to deduce coordinates is proposed. This method, combined with the character segmentation method, improves the accuracy of single-character coordinate extraction.

[0062] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0063] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0064] Figure 1 A flowchart illustrating an embodiment of a character coordinate extraction method according to the present invention is shown;

[0065] Figures 2 to 4 A flowchart of another embodiment of a character coordinate extraction method according to the present invention is shown;

[0066] Figure 5 A flowchart of another embodiment of the character coordinate extraction method of the present invention is shown;

[0067] Figure 6 A flowchart of yet another embodiment of the character coordinate extraction method of the present invention is shown;

[0068] Figure 7 A flowchart of one embodiment of a character coordinate extraction method according to the present invention is shown;

[0069] Figure 8 The diagram illustrates the network architecture of a character coordinate extraction method according to the present invention.

[0070] Figure 9A schematic diagram of image annotation in a character coordinate extraction method of the present invention is shown;

[0071] Figure 10 A schematic diagram of the segmentation network model in a character coordinate extraction method of the present invention is shown;

[0072] Figure 11 A schematic diagram of the detection box position information in a character coordinate extraction method of the present invention is shown;

[0073] Figure 12 The flowchart of coordinate extraction based on single-character segmentation in a character coordinate extraction method of the present invention is shown;

[0074] Figure 13 The flowchart illustrates the extraction of single-character coordinates using the watershed algorithm in a character coordinate extraction method of the present invention.

[0075] Figure 14 The text line heatmap shows the failure of the watershed algorithm to segment due to blurred boundaries;

[0076] Figure 15 The flowchart of CTC-based text recognition in a character coordinate extraction method of the present invention is shown.

[0077] Figure 16 The flowchart illustrates the reverse extraction of coordinates based on CTC recognition results in a character coordinate extraction method according to the present invention.

[0078] Figures 17 to 21 A schematic diagram of a character coordinate extraction device provided in an embodiment of the present invention is shown;

[0079] Figure 22 The diagram shows a structural schematic of a single character coordinate extraction device provided in an embodiment of the present invention. Detailed Implementation

[0080] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0081] Figure 1 A flowchart illustrating an embodiment of a character coordinate extraction method according to the present invention is shown. This method is performed by a single character coordinate extraction device. Figure 1 As shown, the method includes the following steps:

[0082] S100: Input the target text image into the feature extraction backbone network, and obtain character segmentation features and text line segmentation features through feature fusion of different layers in the backbone network;

[0083] Among them, the feature extraction backbone network refers to the main network of the deep convolutional neural network used to extract image features. The feature extraction backbone network includes, but is not limited to, ResNet and SKNet.

[0084] S200: Input the character segmentation features and text line segmentation features into the text line segmentation module and the character segmentation module respectively to obtain the character segmentation heatmap and text segmentation heatmap of the target text image;

[0085] S300: Calculate the coordinates of a single character in the target text image based on the character segmentation heatmap and the text segmentation heatmap.

[0086] The coordinates of a single character refer to the coordinate position information of each character in the string.

[0087] In this embodiment, the single character segmentation network and the text line region segmentation network share a feature extraction backbone network and are integrated into a single neural network, thus avoiding the repeated extraction of features.

[0088] Figures 2 to 4 A flowchart of another embodiment of a character coordinate extraction method according to the present invention is shown, which is performed by a single character coordinate extraction device. Figure 2 As shown, the method includes the following steps:

[0089] S110: Input the target text image into the feature extraction backbone network;

[0090] S120: Extract the feature map of the target text image from the feature extraction backbone network;

[0091] S130: The extracted feature maps are fused using FPN to obtain the character segmentation features and text line segmentation features.

[0092] It is worth noting that, such as Figure 7 and Figure 9 As shown, in convolutional neural networks, low-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and they contain more noise. High-level features have stronger semantic information, but their resolution is very low and their ability to perceive details is poor. Fusing high- and low-level features can improve the robustness of the network.

[0093] Specifically, the target text image is input into the feature extraction backbone network, such as... Figure 8As shown, five feature maps (stride4, stride8, stride16, stride32, and stride64) are extracted from the feature extraction backbone network and fused through FPN. The four feature maps (F2, F3, F4, and F5) after FPN are concatenated using concat and used as character segmentation features. The five feature maps (F2, F3, F4, F5, and F6) after FPN are concatenated using concat and used as text line segmentation features.

[0094] Furthermore, the FPN fusion method is used to fuse five low-level features with five high-level features to obtain F2 (size is 1 / 4 of the original image), F3 (1 / 8), F4 (1 / 16), F5 (1 / 32), and F6 (1 / 64). F3 is upsampled by 2 times, F4 by 4 times, F5 by 8 times, and F6 by 16 times, respectively. The sampled feature maps are all 1 / 4 of the original image. Then, the five feature maps F2, F3, F4, F5, and F6 are concatenated to obtain feature Fchar = C(F2, F3, F4, F5, F6) for character segmentation. The four feature maps F2, F3, F4, and F5 are concatenated to obtain feature map Fline = C(F2, F3, F4, F5) for text line segmentation.

[0095] like Figure 3 As shown, the method includes the following steps:

[0096] S210: Input the character segmentation features into the single character segmentation network to obtain a character segmentation probability map and a character segmentation threshold map.

[0097] The segmentation module can use a DBNet network structure to obtain a threshold map;

[0098] S220: Calculate the character segmentation heat map based on the difference between the character segmentation probability map and the character segmentation threshold map;

[0099] S230: Input the text line segmentation features into the text line region segmentation network to obtain a text line segmentation probability map and a text line segmentation threshold map;

[0100] S240: Calculate the text line segmentation heatmap based on the difference between the text line segmentation probability map and the text line segmentation threshold map.

[0101] Specifically, the fused features F = C(F2, F3, F4, F5, F6) are input into two segments of the network. The first segment is used to predict the probability map and threshold map of the entire text line region to obtain the text line location information for CTC-based text recognition. The other segment is used to predict the probability map and threshold map of each character region to the character image to obtain the location information of the character region.

[0102] Specifically, the predicted sample outputs four segmentation maps through model prediction. In this proposal, the heatmap is obtained by the difference between the probability map and the threshold segmentation map. After the input image goes through two segmentation branches, one branch obtains the text line segmentation probability map P of the image. textline Text line segmentation threshold map T textline The other branch yields the character segmentation probability map P. char Character segmentation threshold graph T char The difference between the corresponding probability map and the threshold map is used to obtain R. textline and R char The calculation formula is as follows:

[0103] R char =P char -T char ;

[0104] R textline =P textline -T textline ;

[0105] The difference image R textline and R char By displaying this using a heatmap, we obtain a heatmap of character and text line segmentation.

[0106] like Figure 4 As shown, the method includes the following steps:

[0107] S310: Obtain the detection box position information of the text line through the text line segmentation heatmap;

[0108] Among them, the location information of the detection box for each text line can be obtained through the text line segmentation heatmap, such as... Figure 11 As shown.

[0109] S320: The character segmentation heatmap is cropped according to the detection box position information of the text line to obtain the text line image.

[0110] Specifically, the character heatmap is cropped according to the position information of the text lines to obtain images segmented into text lines, such as... Figure 12 As shown.

[0111] S330: The text line image is segmented using the watershed algorithm to form a segmentation map, and the number of the segmentation maps is obtained;

[0112] S340: Identify the number of characters in the text line image using CTC;

[0113] S350: Compare the number of segmented images obtained by the watershed algorithm with the number of characters identified by CTC;

[0114] S360: When the number of segmentation images is the same as the number of characters, the position information of each character is obtained through the watershed algorithm;

[0115] S370: Restore the position information of each character to the target text image to obtain the coordinates of each character;

[0116] S380: When the number of segmentation images is not the same as the number of characters, extract the coordinates of single characters from CTC.

[0117] Among them, the watershed algorithm is a commonly used segmentation method for image regions. During the segmentation process, it takes the similarity between neighboring pixels as an important reference, thereby connecting pixels that are spatially close and have similar gray values ​​to form a closed contour.

[0118] Specifically, segmentation is performed using the commonly used watershed algorithm. If segmentation is successful, the position information of each character can be directly obtained, and restoring the position information to the original image yields the coordinates of the single character. The process for determining whether characters are connected based on the watershed algorithm is as follows: Figure 13 As shown.

[0119] If watershed segmentation fails, it indicates that the segmentation images may be stuck together. In this case, the coordinates of individual characters can be extracted using the recognition results based on CTC.

[0120] The design of the text line segmentation and character segmentation network model specifically includes: obtaining feature maps for text line segmentation through the segmentation network model, and inputting the fused features into two segmentation network branches respectively. The first branch is used to predict the probability map and threshold map of the entire text line region to obtain the text line location information for CTC-based text recognition; the other branch is used to predict the probability map and threshold map of each character region to the character image to obtain the character region location information.

[0121] like Figure 10As shown, the predicted sample outputs four segmentation images through model prediction. Heatmaps for character and text line segmentation are calculated. The text line segmentation heatmap provides the location information of the detection box for each text line. The character heatmap is then cropped according to the text line location information to obtain segmented text line images, which are then segmented using the commonly used watershed algorithm. If segmentation is successful, the location information of each character can be directly obtained. If the watershed algorithm fails to segment, it indicates that the segmentation images may be stuck together. In this case, the coordinates of individual characters can be extracted using the recognition results based on CTC (Characteristic Character Coding).

[0122] This embodiment employs two parallel methods in the process of extracting character coordinates, which has high robustness for character segmentation. Branch 1 combines the segmented text line information with CTC to obtain the text content and the number of characters. Branch 2 obtains the segmented image and the position information of individual characters through the single-character segmentation method. When there is no sticking in the segmented image, the result is directly output. This method has high robustness and can solve the problem of sticking character segmentation in the segmentation network.

[0123] Figure 5 A flowchart of another embodiment of a character coordinate extraction method according to the present invention is shown, which is performed by a single character coordinate extraction device. Figure 5 As shown, the method includes the following steps:

[0124] S381: The text line image is uniformly segmented based on CTC to form several segmented image blocks;

[0125] S382: Identify several segmented image blocks to obtain the character corresponding to each segmented image block, and mark unrecognizable segmented image blocks as special characters;

[0126] S383: Merge the segmented image blocks corresponding to the same character to form a merged image block;

[0127] S384: Segment the image block at the 1 / 2 position to obtain the segmentation result of each character;

[0128] S385: The character segmentation result is mapped to the text line image to obtain a text box, and finally the single character coordinate information based on CTC is obtained.

[0129] like Figure 14 For text lines that fail to be segmented by the watershed algorithm, the coordinates of individual characters are extracted from CTC.

[0130] CTC is a loss calculation method that does not require alignment. CTC is commonly used in character content recognition processes, and the steps are as follows: Figure 13As shown, the image is first uniformly divided, and the probability of each block belonging to a certain character is calculated. Unrecognizable image blocks are marked with the special character "-". For example... Figure 13 As shown, after the text image is processed by CTC, the recognition result "-st-aatte" is obtained. Then, the final recognition result "state" is obtained by deduplication.

[0131] like Figure 15 As shown, in this embodiment, image blocks corresponding to the same characters in the CTC intermediate results are merged. The merged characters are then segmented. Unrecognizable characters, such as "-", are divided equally left and right, meaning they are segmented from the 1 / 2 position of the character during the segmentation process. The segmentation results of each character are then mapped to text line images to obtain text boxes, ultimately yielding the CTC-based single-character coordinate information. The CTC-based single-character coordinate extraction method flow is as follows: Figure 16 As shown.

[0132] This embodiment employs two parallel methods in the character coordinate extraction process, exhibiting high robustness for character segmentation. Branch 1 combines the segmented text line information with CTC to obtain the text content and character count. When individual characters segmented by Branch 2 exhibit contiguous segmentation, the coordinates are verified using a CTC-based single-character coordinate verification method to obtain the single-character coordinate information. Branch 2 acquires segmented images and obtains single-character position information through single-character segmentation. When no contiguous segmentation occurs, the result is directly output. This method demonstrates high robustness, effectively resolving the problem of contiguous character segmentation in the segmentation network, while the shared backbone network avoids redundant feature extraction.

[0133] This embodiment uses a parallel network model to simultaneously segment text lines and character regions. Two single-character coordinate extraction methods are used for the two segmentation branches respectively. The combination of the two methods can solve the coordinate extraction process of connected characters.

[0134] Figure 6 A flowchart of yet another embodiment of a character coordinate extraction method according to the present invention is shown, which is performed by a single character coordinate extraction device. Figure 6 As shown, the method includes the following steps:

[0135] The S400 training segmentation network model further includes:

[0136] S410 prepares training data, which requires labeling the position information of each character and the position information of the entire text line. The position information of each character is used to train a single character segmentation network, and the position information of the entire text line is used to train a text line region segmentation network.

[0137] S420 designs a joint training loss function, and trains the segmentation network model using the joint training loss function;

[0138] The formula for calculating the joint training loss function is as follows:

[0139] Loss = αloss char +βloss textline ;

[0140] Where α and β are constant coefficients;

[0141] loss char and loss textline Segmentation map loss L, which includes characters and text lines respectively. S Threshold map loss L t :

[0142] loss char =α1L S1 +β1L t1 ;

[0143] loss textline =α2L S2 +β2L t2 ;

[0144] Where α1, α2, β1, and β2 are constant coefficients;

[0145] The segmentation probability map in the joint training loss function adopts the binary classification cross-entropy loss function, and the loss function L... S1 L S2 The inputs are the sample predicted probability map and the sample true label map:

[0146]

[0147] Among them, S l For the sample set, x i Let y be the probability value of a pixel in the sample prediction image. i This represents the true value of a pixel in the sample's true label.

[0148] Loss function L t1 L t2 The input consists of a threshold map of the predicted text lines and a sample ground truth map. The threshold map uses the L1 distance loss function.

[0149]

[0150] Among them, R d This is the set of pixel indices in the threshold map. For label values, These are predicted values.

[0151] It's worth noting that the loss function, also called the loss function, is the difference between the predicted value and the true value of a single sample. The smaller the loss, the better the model. In this proposal, since the training process segments both characters and text lines simultaneously, there are two segmentation loss functions: character segmentation loss and text segmentation loss. char and text box segmentation loss textline To improve the accuracy of the segmentation network, this scheme designs the following joint training loss function, where the segmentation network loss function consists of the character segmentation loss. char and text box segmentation loss textline The summation is used to construct the equation, where α and β are constant coefficients that can be adjusted based on experience.

[0152] Loss = αloss char +βloss textline ;

[0153] Where loss char and loss textline Segmentation map loss L, which includes characters and text lines respectively. S Threshold map loss L t In the formula, α1, α2, β1, and β2 are constant coefficients, which are adjusted based on experience.

[0154] loss char =α1L S1 +β1L t1 ;

[0155] loss textline =α2L S2 +β2L t2 ;

[0156] The segmentation probability map in the loss function uses the binary classification cross-entropy loss function, and the loss function L... S1 L S2 The input is Figure 4 The sample predicted probability map and the sample true label map are shown in the formula, where S l For the sample set, x i Let y be the probability value of a pixel in the sample prediction image. i This is the true value of a pixel in the sample's real label.

[0157]

[0158] Loss function L t1 L t2 The input consists of a threshold map of the predicted text lines and a sample ground truth map. The threshold map uses the L1 distance loss function. R d This is the set of pixel indices in the threshold map. For label values, For predicted values:

[0159]

[0160] This embodiment segments both character regions and text line regions simultaneously. By jointly training the loss function through character segmentation branches and text line segmentation branches, the network convergence is accelerated, achieving better segmentation results.

[0161] Figure 17 A schematic diagram of an embodiment of a character coordinate extraction device according to the present invention is shown. Figure 17 As shown, the device includes:

[0162] The target text image input module 100 is used to input the target text image into the feature extraction backbone network;

[0163] The segmentation feature acquisition module 101 is used to acquire character segmentation features and text line segmentation features;

[0164] The segmentation feature input module 102 is used to input the character segmentation features and text line segmentation features to the text line segmentation module and the character segmentation module, respectively.

[0165] The character segmentation heatmap module 103 is used to obtain the character segmentation heatmap of the target text image;

[0166] The text segmentation heatmap module 104 is used to obtain the text segmentation heatmap of the target text image;

[0167] The coordinate calculation module 105 is used to calculate the coordinates of a single character based on the character segmentation heatmap and the text segmentation heatmap.

[0168] In one alternative approach, the following is included:

[0169] The first input module 110 is used to input the target text image into the feature extraction backbone network;

[0170] Feature map extraction module 120 is used to extract feature maps of the target text image from the feature extraction backbone network;

[0171] The fusion module 130 is used to fuse the extracted feature maps through FPN to obtain the character segmentation features and text line segmentation features;

[0172] The first acquisition module 210 is used to input the character segmentation features into the single character segmentation network to obtain a character segmentation probability map and a character segmentation threshold map;

[0173] The first calculation module 220 is used to calculate the character segmentation heat map based on the difference between the character segmentation probability map and the character segmentation threshold map;

[0174] The second acquisition module 230 is used to input the text line segmentation features into the text line region segmentation network to obtain a text line segmentation probability map and a text line segmentation threshold map.

[0175] The second calculation module 240 calculates the text line segmentation heat map based on the difference between the text line segmentation probability map and the text line segmentation threshold map.

[0176] In one alternative approach, such as Figure 18-21 As shown, it includes:

[0177] The detection box position information acquisition module 310 is used to acquire the detection box position information of the text line through the text line segmentation heatmap;

[0178] The cropping module 320 is used to crop the character segmentation heatmap according to the detection box position information of the text line to obtain a text line image;

[0179] The segmentation module 330 is used to segment the text line image using a watershed algorithm to form a segmentation map and to obtain the number of the segmentation maps;

[0180] The first recognition module 340 is used to recognize the number of characters in the text line image via CTC;

[0181] The second recognition module 350 is used to compare the number of segmented images obtained by the watershed algorithm with the number of characters recognized by CTC.

[0182] The location information acquisition module 360 ​​is used to acquire the location information of each character through the watershed algorithm when the number of segmentation images is the same as the number of characters.

[0183] The restoration module 370 is used to restore the position information of each character to the target text image to obtain the coordinates of each character;

[0184] Extraction module 380 is used to extract single character coordinates from CTC when the number of segmentation images is not the same as the number of characters.

[0185] The image block segmentation module 381 is used to uniformly segment the text line image based on CTC to form a plurality of segmented image blocks.

[0186] The marking module 382 is used to identify several of the segmented image blocks, obtain the character corresponding to each segmented image block, and mark the segmented image blocks that cannot be identified as special characters;

[0187] The image block merging module 383 is used to merge the segmented image blocks corresponding to the same characters to form a merged image block;

[0188] The merged image block segmentation module 384 is used to segment the merged image block from the 1 / 2 position to obtain the segmentation result of each character;

[0189] The single-character coordinate information acquisition module 385 is used to map the character segmentation result to the text line image to obtain a text box, and finally obtain the single-character coordinate information based on CTC.

[0190] In one alternative approach, the following is included:

[0191] Training module 400 is used to train the segmentation network model; training module 400 includes:

[0192] The data preparation module 410 is used to prepare training data. The training data needs to be labeled with the position information of each character and the position information of the entire text line. The position information of each character is used to train a single character segmentation network, and the position information of the entire text line is used to train a text line region segmentation network.

[0193] Design module 420 is used to design a joint training loss function and train the segmentation network model using the joint training loss function.

[0194] Figure 22 The diagram shows a structural schematic of an embodiment of a single character coordinate extraction device according to the present invention. The specific implementation of the present invention does not limit the specific implementation of the single character coordinate extraction device.

[0195] like Figure 22 As shown, the device for extracting the coordinates of a single character may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0196] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps described above in the XXXX method embodiment.

[0197] Specifically, program 510 may include program code, which includes computer-executable instructions.

[0198] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the XXXXXX device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0199] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0200] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a single character coordinate extraction device / apparatus, causes the single character coordinate extraction device / apparatus to perform the character coordinate extraction method in any of the above method embodiments.

[0201] This invention provides a computer program that can be called by a processor to cause a single character coordinate extraction device to execute the character coordinate extraction method in any of the above method embodiments.

[0202] This invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed on a computer, cause the computer to perform the character coordinate extraction method in any of the above method embodiments.

[0203] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0204] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0205] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0206] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0207] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0208] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for extracting character coordinates, characterized in that, The method includes the following steps: The target text image is input into the feature extraction backbone network, and character segmentation features and text line segmentation features are obtained by feature fusion of different layers in the backbone network; The character segmentation features and text line segmentation features are input into the text line segmentation module and the character segmentation module, respectively, to obtain the character segmentation heatmap and text line segmentation heatmap of the target text image; wherein, the character segmentation features are input into a single character segmentation network to obtain a character segmentation probability map and a character segmentation threshold map; the character segmentation heatmap is calculated based on the difference between the character segmentation probability map and the character segmentation threshold map; the text line segmentation features are input into a text line region segmentation network to obtain a text line segmentation probability map and a text line segmentation threshold map; the text line segmentation heatmap is calculated based on the difference between the text line segmentation probability map and the text line segmentation threshold map; Based on the character segmentation heatmap and the text line segmentation heatmap, the coordinates of a single character in the target text image are calculated. Specifically, the detection box position information of the text line is obtained from the text line segmentation heatmap; the character segmentation heatmap is cropped according to the detection box position information of the text line to obtain a text line image; the text line image is segmented using a watershed algorithm to form segmentation maps, and the number of segmentation maps is obtained; the number of characters in the text line image is identified using CTC; the number of segmentation maps obtained by the watershed algorithm is compared with the number of characters identified by CTC; when the number of segmentation maps is the same as the number of characters, the position information of each character is obtained using the watershed algorithm; the position information of each character is restored to the target text image to obtain the coordinates of each character; when the number of segmentation maps is different from the number of characters, the coordinates of a single character are extracted from CTC.

2. The character coordinate extraction method according to claim 1, characterized in that, The target text image is input into a feature extraction backbone network. Character segmentation features and text line segmentation features are obtained through feature fusion of different layers in the backbone network. The specific steps include: The target text image is input into the feature extraction backbone network; Feature maps of the target text image are extracted from the feature extraction backbone network; The extracted feature maps are fused using FPN to obtain the character segmentation features and text line segmentation features.

3. The method for extracting character coordinates according to claim 1, characterized in that, When the number of segmentation images is not the same as the number of characters, the specific steps for extracting single-character coordinates from CTC include: Based on CTC, the text line image is uniformly segmented to form several segmented image blocks. The segmented image blocks are identified to obtain the character corresponding to each segmented image block. Segments that cannot be identified are marked as special characters. The segmented image blocks corresponding to the same character are merged to form a merged image block; The image block is divided at half its length to obtain the segmentation result for each character; The character segmentation results are mapped to the text line image to obtain text boxes, and finally, the single-character coordinate information based on CTC is obtained.

4. The method for extracting character coordinates according to claim 1, characterized in that, The method further includes: training a segmentation network model, wherein the training segmentation network model further includes: Prepare training data, which needs to be labeled with the position information of each character and the position information of the entire text line. The position information of each character is used to train a single character segmentation network, and the position information of the entire text line is used to train a text line region segmentation network.

5. The character coordinate extraction method according to claim 4, characterized in that, The training segmentation network model also includes: Design a joint training loss function, and train the segmentation network model using the joint training loss function; The formula for calculating the joint training loss function is as follows: ； in, , The coefficient is constant. and Segmentation map losses, including characters and text lines respectively. Threshold map loss : ； ； in, , , , The coefficient is constant. The segmentation probability map in the joint training loss function adopts the binary classification cross-entropy loss function. The inputs are the sample predicted probability map and the sample true label map: ； in, For the sample set, This represents the probability value of a specific pixel in the sample prediction image. This represents the true value of a pixel in the sample's true label. loss function The input consists of a threshold map of the predicted text lines and a sample ground truth map. The threshold map uses the L1 distance loss function. ； in, This is the set of pixel indices in the threshold map. For label values, These are predicted values.

6. A device for extracting the coordinates of a character, characterized in that, The device includes: The target text image input module is used to input the target text image into the feature extraction backbone network; The segmentation feature acquisition module is used to acquire character segmentation features and text line segmentation features; The segmentation feature input module is used to input the character segmentation features and text line segmentation features into the text line segmentation module and the character segmentation module, respectively. The character segmentation heatmap module is used to obtain the character segmentation heatmap of the target text image; wherein, the character segmentation features are input into a single character segmentation network to obtain a character segmentation probability map and a character segmentation threshold map; the character segmentation heatmap is calculated based on the difference between the character segmentation probability map and the character segmentation threshold map; The text line segmentation heatmap module is used to obtain the text line segmentation heatmap of the target text image; wherein, the text line segmentation features are input into the text line region segmentation network to obtain the text line segmentation probability map and the text line segmentation threshold map; the text line segmentation heatmap is calculated based on the difference between the text line segmentation probability map and the text line segmentation threshold map; The coordinate calculation module is used to calculate the coordinates of a single character based on the character segmentation heatmap and the text line segmentation heatmap. Specifically, it obtains the detection box position information of the text line using the text line segmentation heatmap; crops the character segmentation heatmap according to the detection box position information of the text line to obtain a text line image; segments the text line image using a watershed algorithm to form segmentation maps and obtains the number of segmentation maps; identifies the number of characters in the text line image using CTC; compares the number of segmentation maps obtained by the watershed algorithm with the number of characters identified by CTC; when the number of segmentation maps is the same as the number of characters, it obtains the position information of each character using the watershed algorithm; restores the position information of each character to the target text image to obtain the coordinates of each character; when the number of segmentation maps is different from the number of characters, it extracts the coordinates of a single character from CTC.

7. A device for extracting the coordinates of a single character, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the character coordinate extraction method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on a single character coordinate extraction device, causes the single character coordinate extraction device to perform the operation of the character coordinate extraction method as described in any one of claims 1-5.

Images