Text image processing method, device, electronic scanning equipment and storage medium
Through the combination of adaptive enhancement algorithm and clustering algorithm, accurate positioning and line grouping processing of text images are achieved, and the problem of inaccurate recognition under the influence of noise and jitter in the prior art is solved, and the speed and accuracy of text positioning are improved. , suitable for fast operation of mobile devices.
Patent Information
- Application Number
- CN202010682019.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-07-15
AI Technical Summary
The existing technology has noise influence, offset and jitter in text positioning and line grouping processing, which leads to inaccurate recognition, especially in complex scenarios, which cannot effectively distinguish multiple lines of text, resulting in large calculations and long-term calculations, which cannot meet the real-time scanning recognition. need.
The text detection model using adaptive enhancement algorithm combines image pyramids and clustering algorithms, and performs stitching and combining through preprocessing and text position information acquisition to achieve accurate positioning and line grouping processing of text images.
It improves the accuracy and speed of text positioning, can effectively process complex arranged text images, reduces the rate of error judgment, and meets the fast operation needs of mobile devices.
Smart Images

Figure CN111832551B_ABST
Abstract
Description
Background Art
[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.
[0003] Handheld micro scanners such as scanning pens can use scanning technology to scan images or printed fonts into the pen for storage or transfer to a computer for storage for reading, editing, translation, etc. Common scanning pens include optical sensors, LED lights, and touch sticks. When scanning, the scanning pen can obtain multiple scanned images by scanning, for example, multiple lines of text. When processing, the scanned image sequence is processed by text positioning, text grouping, etc. for subsequent OCR processing.
[0004] Text positioning of a scanned image refers to determining the location information of the text area in the scanned image. Text positioning is a necessary process for identifying text in an image. For example, the image can be processed by morphological operations to obtain candidate text domains, and the text domains can be merged by combining the stroke width features of the image to achieve text positioning. However, the morphological operation processing method is easily affected by image noise, ink dots, and watermarks, resulting in inaccurate detection of text areas. In addition, occasional offsets and jitters during scanning can cause distortion of the spliced image, resulting in bending and deformation of the text. The morphological operation processing method cannot distinguish multiple lines of text in the above complex scenarios.
[0005] At present, there are text positioning solutions based on convolutional neural networks. The general process is to pre-train a text line recognition model based on convolutional neural networks, input the collected text line image into the text line recognition model, and obtain the text line position. However, since this text positioning processing based on convolutional neural networks includes feature extraction, image line segmentation, and line text positioning, the algorithm has a large amount of calculation in the use scenario of an electronic scanning pen, and the text positioning takes a long time. If it is run on a mobile device, it takes a long time to operate, which cannot meet the needs of instant scanning, recognition and translation, and seriously affects the user experience.
[0006] In addition, for text images with complex arrangements, such as curved text images, multi-line dense text images, and noisy text images, existing text localization and text line grouping engines are still unable to accurately distinguish text lines, and often misjudge multi-line text as a single line of text, resulting in a low recognition accuracy rate for the text image recognition engine for that line of text. Summary of the invention
[0007] The present application hopes to provide a text image processing method, apparatus, electronic scanning device and storage medium to at least solve some of the above-mentioned technical problems.
[0008] In a first aspect, an embodiment of the present application provides a text image processing method, comprising:
[0009] Preprocessing the target text to obtain a target image corresponding to the target text;
[0010] Acquire text position information corresponding to at least part of the text in the target image;
[0011] Based on the text position information, text line processing is performed to obtain a single line of text or multiple lines of text.
[0012] According to the text image processing method of the embodiment of the present application, the preprocessing of the target text includes:
[0013] A target image corresponding to the target text is obtained by performing a splicing process on a scanned image sequence of the target text obtained by scanning the target object once or multiple times using a scanning device to obtain a target image.
[0014] According to the text image processing method of the embodiment of the present application, the image is a panoramic image.
[0015] According to the text image processing method of the embodiment of the present application, the step of obtaining text position information corresponding to at least part of the text in the target image includes:
[0016] The target image is processed using a text detection model based on an adaptive enhancement algorithm to obtain text position information corresponding to at least part of the text in the target image.
[0017] According to the text image processing method of the embodiment of the present application, the text detection model is generated by training a classifier using training data, and the training data includes positive sample data and negative sample data, wherein:
[0018] The positive sample data includes a plurality of text units intercepted from a sample image, and the label data corresponding to each text unit is text position information of each text unit in the sample image;
[0019] The negative sample data includes a plurality of non-text images.
[0020] According to the text image processing method of the embodiment of the present application, the text position information includes the position and size information of the bounding box bbox of the text unit in the sample image.
[0021] According to the text image processing method of the embodiment of the present application, the text in the text unit meets at least one of the following conditions: there is no incomplete text in the text unit, the number of text in the text unit is one, and the text unit contains only one complete text.
[0022] According to the text image processing method of the embodiment of the present application, the step of obtaining text position information corresponding to at least part of the text in the target image includes:
[0023] Building an image pyramid based on the target image, and using each layer of the image in the image pyramid as an image to be recognized;
[0024] The images of each layer of the image pyramid are input into the text detection model to obtain a plurality of text position information corresponding to a plurality of text units output by the text detection model.
[0025] According to the text image processing method of the embodiment of the present application, the text line grouping processing is performed based on the text position information to obtain a single line of text or multiple lines of text, including:
[0026] Based on a clustering algorithm, setting n0 initial clustering centers for the plurality of text position information;
[0027] According to the positional relationship between the multiple text position information and the n0 initial clustering centers, the multiple text position information are clustered to obtain a single line of text or multiple lines of text.
[0028] According to the text image processing method of the embodiment of the present application, clustering the multiple text position information according to the positional relationship between the multiple text position information and the n initial clustering centers respectively includes:
[0029] The midpoints of two adjacent initial cluster centers among the n0 initial cluster centers are used as classification division points to obtain n0 classes;
[0030] Divide each text position information into the class with the closest distance;
[0031] Based on the text position information in the n0 classes after division, the cluster centers are recalculated to obtain n1 cluster centers;
[0032] The above process is repeated until the number and position of cluster centers no longer change, and n cluster centers are finally obtained, wherein multiple text position information corresponding to the same cluster center are in the same row.
[0033] According to the text image processing method of the embodiment of the present application, the method further includes:
[0034] Combining multiple bounding boxes corresponding to multiple text position information in the same row into one row in horizontal order;
[0035] Adjusting the heights of the plurality of bounding boxes combined into a row so that the heights of the plurality of bounding boxes are consistent;
[0036] Acquire corresponding multiple text pixel regions from the target image according to multiple text position information corresponding to the multiple bounding boxes;
[0037] The multiple text pixel regions are combined into a single line of text.
[0038] According to the text image processing method of an embodiment of the present application, optical character recognition (OCR) processing is performed on the single-line text or multiple-line text.
[0039] In a second aspect, an embodiment of the present application further provides a text image processing device, comprising:
[0040] A preprocessing module, used to preprocess the target text to obtain a target image corresponding to the target text;
[0041] A text position information acquisition module, used to acquire text position information corresponding to at least part of the text in the target image;
[0042] The text line grouping processing module is used to perform text line grouping processing based on the text position information to obtain a single line of text or multiple lines of text.
[0043] In a third aspect, an embodiment of the present application further provides a computing device, including:
[0044] one or more processors;
[0045] A storage device for storing one or more programs;
[0046] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the text image processing method as described above.
[0047] In a fourth aspect, an embodiment of the present application further provides a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the text image processing method as described above.
[0048] Based on the embodiments of the present application, a scanning device such as an electronic scanning pen is used to scan the target object once or multiple times to obtain an image corresponding to the target object. The text position information can be obtained from the image to achieve text positioning. Line grouping processing is performed based on the obtained text position information to obtain single-line or multi-line text corresponding to the target text. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:
[0050] Figure 1 The following schematically shows a flowchart of the text image processing method according to an embodiment of the present application;
[0051] Figure 2 A panoramic image in an embodiment of the present application is schematically shown;
[0052] Figure 3 The flowchart of processing an image pyramid according to an embodiment of the present application is schematically shown;
[0053] Figure 4 The effect diagram of the image pyramid of the embodiment of the present application is schematically shown;
[0054] Figure 5 A schematic diagram schematically shows a text unit obtained in an embodiment of the present application;
[0055] Figure 6 A flowchart of the text position information group of an embodiment of the present application is schematically shown;
[0056] Figure 7 The flowchart of text position information clustering in an embodiment of the present application is schematically shown;
[0057] Figure 8 The flowchart of grouping text pixels in an embodiment of the present application is schematically shown;
[0058] Fig. 9 and Fig.10 The schematic diagram schematically shows the effect of visually displaying multiple bboxes on the original image according to an embodiment of the present application;
[0059] Fig.11 and Fig.12 Schematically shows Fig. 9 and Fig.10 Schematic diagram of a combined single line of text;
[0060] Fig.13 The schematic diagram of the storage medium structure of the embodiment of the present application is shown schematically;
[0061] Fig.14 The structure diagram of the text image processing device according to the embodiment of the present application is schematically shown;
[0062] Fig.15 A schematic diagram of the computing device structure of an embodiment of the present application is shown schematically.
[0063] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0064] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.
[0065] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0066] According to the implementation manner of the present application, a text image processing method, apparatus, storage medium and computing device are proposed.
[0067] In this document, any number of elements in the drawings is used for illustration rather than limitation, and any naming is used only for distinction and does not have any limiting meaning.
[0068] Exemplary Methods
[0069] Reference below Figure 1 A text image processing method according to an exemplary embodiment of the present application is described.
[0070] Figure 1 The following schematically shows a flowchart of a text image processing method according to an embodiment of the present invention. The text image processing method comprises the following steps:
[0071] S101: preprocessing the target text to obtain a target image corresponding to the target text;
[0072] S102: Obtaining text position information corresponding to at least part of the text in the target image;
[0073] S103: Perform text line grouping processing based on the text position information to obtain a single-line text or multiple-line text.
[0074] Utilizing the embodiments of the present application, a scanning device such as an electronic scanning pen is used to scan a target object once or multiple times to obtain an image corresponding to the target object, from which text position information can be acquired, and line grouping processing can be performed based on the acquired text position information to obtain single-line or multi-line text corresponding to the target text.
[0075] In an embodiment of the present application, the preprocessing of the target text may include: splicing a scanned image sequence of the target text obtained by scanning the target object once or multiple times using a scanning device to obtain a target image corresponding to the target text.
[0076] In an embodiment of the present application, the target image is a panoramic image. Figure 2 A panoramic image is schematically shown, and the panoramic image contains text content scanned by the user. For example, the user uses an electronic scanning pen to scan one or more lines of printed text, and an image sequence of the printed text can be obtained. Since the user holds the scanning pen to a certain extent when scanning, a series of images in the image sequence may overlap and deform. Therefore, it is necessary to perform necessary processing on the image sequence, such as image registration and fusion splicing, to obtain a panoramic image containing text with a large field of view, and then locate the text in the panoramic image.
[0077] The embodiment of the present application uses a text detection model based on an adaptive boosting algorithm (Adaptive Boosting Algorithm, referred to as Adaboost algorithm) to process a target image, obtain text position information corresponding to at least part of the text in the target image, and realize text positioning.
[0078] The text detection model based on the Adaboost algorithm in the embodiment of the present application is generated by training a classifier using training data, wherein the training data includes positive sample data and negative sample data, wherein the positive sample data includes multiple text units captured from a sample image, and the label data corresponding to each text unit is the text position information of each text unit in the sample image; the negative sample data includes multiple non-text images.
[0079] The sample image as a positive sample is an image containing text, and the non-text image as a negative sample is an image without text, such as a background image.
[0080] Optionally, the characters in the character unit captured from the sample image meet at least one of the following conditions: there is no incomplete character in the character unit, the number of characters in the character unit is one, and the character unit contains only one complete character.
[0081] For example, the text unit intercepted from the sample image can be a single-word image unit, that is, only one word is intercepted as the positive sample data, and there are neither other complete words nor other incomplete words in the text unit. The advantage of this treatment is that the trained text detection model will not detect the position of text units of multiple words, avoiding the situation of text duplication or ghosting in subsequent line grouping.
[0082] Here, "single word" can be a single word in various languages, for example, a single Chinese character, a single foreign word, a single foreign letter, etc.
[0083] Optionally, the text position information of the text unit in the sample image includes: the position and size information of the bounding box (bbox) of the text unit in the sample image, for example, which can be expressed as (x, y, w, h), where x and y are the coordinates of the upper left corner vertex of the bounding box bbox, and w and h are the width and height of the bounding box bbox.
[0084] The embodiment of the present application uses the above training data to train a classifier and optimizes the parameters of the classifier until the training stop condition is reached. The trained classifier is used as a text detection model.
[0085] The embodiment of the present application uses labeled training data (or sample data) to train the classifier, so that the classifier continuously adjusts the model parameters autonomously according to the classification labels of the positive sample "text image" and the negative sample "non-text image" in the training data, until, for example, the difference between the adjusted output result and the labeled value is small enough, or the number of training times reaches a predetermined number, etc., then the training is stopped. At this time, it can be considered that the trained classifier has the ability to distinguish between text images and non-text images; and, since the label content of the positive sample annotation includes text position information, the trained model can output the text position information of the text in the original image, so the text detection model has the function of text positioning.
[0086] As an example, the embodiment of the present application can build a Boosted cascade classifier through the opencv_traincascade command of Opencv (i.e., a cross-platform computer vision library based on BSD license / open source release), set the initialized general parameters, cascade parameters and feature parameters, read sample data for training, and the generated text detection model Opencv can distinguish between "text images" and "non-text images" and output the text position information in the text image.
[0087] The embodiment of the present application uses a text detection model based on the Adaboost algorithm, which is particularly suitable for a relatively single application scenario of scanning paper documents (using an electronic scanning pen). The reason is that the model based on the Adaboost algorithm requires less training data, the feature screening process takes a short time, the training speed is fast, and the calculation cost is low. Therefore, training results that meet the usage requirements can be obtained with less investment.
[0088] In the text detection model based on the Adaboost algorithm in the embodiment of the present application, the positive samples in the training data can be simple text images, such as single-word images, and the negative samples can be any background images without text. The annotation cost is low, and the obtained model has high calculation accuracy and fast detection speed. It is particularly suitable for satisfying the application characteristics of fast operation and fast output of results of mobile devices.
[0089] Through the above embodiments, a text detection model can be constructed. Before using the text detection model to process a target image, in order to ensure that all or most of the text in the target image is detected, the embodiment of the present application constructs an image pyramid based on the target image. Specifically, refer to Figure 3 :
[0090] S201: constructing an image pyramid based on the target image, taking each layer of the image pyramid as an image to be recognized, wherein the image to be recognized includes a text line to be recognized;
[0091] S202: Inputting each layer image of the image pyramid into the aforementioned text detection model, a plurality of text position information corresponding to a plurality of text units output by the text detection model can be obtained, for example, (x, y, w, h) parameter information of a plurality of bboxes of the plurality of text units.
[0092] For example, the "Cascade Classifier" function in the trained text detection model Opencv can be used as a detection function. The detection function uses a preset first sliding window to move on each layer of the image pyramid to capture a large number of text units. Preferably, single-word text units are retained, and text units containing multiple words or half a word are discarded. Finally, the position information of the single-word text units is output.
[0093] As an example, see Figure 4 , a series of images of different sizes can be obtained by scaling the panoramic image with a fixed step length. These images can be constructed into an image pyramid. Each layer of the image pyramid is a panoramic image of different sizes to be recognized. The number of layers of the image pyramid can be set according to the situation, such as 10 layers, 45 layers, 120 layers, 200 layers, etc. The more layers of the image pyramid, the higher the processing accuracy of the text.
[0094] After inputting the panoramic image into the text detection model, the first sliding window of the model is used to move layer by layer along the width direction of the text line to be recognized in the image pyramid according to a preset step size. For example, the first sliding window traverses the image pyramid in the order from top to bottom and from left to right, so that each character in the panoramic image can be intercepted. Among them, the width and height of the first sliding window are model parameters that can be set, and the values of its width and height can be set according to actual needs. For example, it can be taken as the size of the bbox, or taken as the size of a general font (such as the size of any font size from No. 2 to No. 6 font) and so on. When the number of layers of the image pyramid is large enough, all valid text units can be obtained by traversing each layer of the image.
[0095] For example, referring to Figure 5 , where the left side is a set of text units intercepted by the first sliding window on different layers of the image pyramid. It can be seen that the text sizes in the text units are different. Some intercept multiple characters (such as "these" and "do" in the first and second screenshots), some intercepted parts contain incomplete characters (such as half a character in the first, second, third, fourth, and sixth screenshots), and some intercept a single complete character (such as "go" in the fifth screenshot). Preferably, Figure 6 only the single complete character "go" is retained in the set of text units of
[0096] , and the text position information of the character "go" in the original panoramic image is detected and output, such as the (x, y, w, h) parameters of the bbox of the character "go".
[0097] Through the above processing, the text position information of some or all characters in the panoramic image can be obtained, which prepares for the subsequent text line grouping process. Generally, text line grouping is a process of merging text images belonging to the same line in the line direction to obtain a complete text line.
[0097] In the embodiment of the present application, the output of the text detection model is the multiple text position information of multiple text units in the target image. In this case, first, the multiple text position information is grouped into lines, and then the pixel regions corresponding to each text position information in the target image are grouped into lines.
[0098] The following describes the process of grouping the text position information into lines in detail.
[0099] Referring to Figure 6 , the process of grouping the text into lines based on the text position information in step S103 to obtain single-line text or multi-line text may include the following steps:
[0100] S1031: Based on the clustering algorithm, set n0 initial clustering centers for the multiple text position information;
[0101] S1032: Clustering the multiple text position information according to the positional relationships between the multiple text position information and the n0 initial cluster centers to obtain a single line of text or multiple lines of text.
[0102] It can be seen that the embodiment of the present application uses a clustering algorithm to group text position information. First, n0 (for example, 2, 3 or 4, etc.) initial clustering centers are assumed, and clustering processing is performed based on the distances between multiple text position information and the clustering centers to ultimately determine the number of rows.
[0103] Further, in step S1032, clustering processing is performed on the plurality of text position information according to the positional relationship between the plurality of text position information and the n0 initial clustering centers, referring to Figure 7 , which may include the following processing:
[0104] S301: taking the midpoints of two adjacent initial cluster centers among the n0 initial cluster centers as classification division points to obtain n0 classes;
[0105] S302: Classify each text position information into the class with the closest distance;
[0106] S303: recalculating cluster centers based on the text position information in the n0 divided classes to obtain n1 cluster centers;
[0107] S304, repeat the above process until the number and position of cluster centers no longer change, and finally obtain n cluster centers, wherein multiple text position information corresponding to the same cluster center are in the same row.
[0108] In some embodiments of the present application, a k-means clustering algorithm is used, and the text position information corresponding to the text unit is represented by the (x, y, w, h) of the bbox of the text unit. Classification can be performed based on the vertical distance between the y value of each bbox and the cluster center. If the y value of the first bbox falls into the class of the first cluster center, then the first bbox belongs to the class of the first cluster center. After classification, the bboxes in the classes of each cluster center belong to the same row.
[0109] Furthermore, in order to correctly group the rows, multiple bboxes belonging to the same row are horizontally sorted according to the x value of each bbox, and the final grouping of the corresponding text pixels in the original image is realized based on this. Figure 8 , can be processed by the following steps:
[0110] S401: combining multiple bounding boxes corresponding to multiple text position information in the same row into one row in a horizontal order;
[0111] S402: adjusting the heights of the plurality of bounding boxes combined into a row so that the heights of the plurality of bounding boxes are consistent;
[0112] S403: Acquire corresponding multiple text pixel regions from the target image according to multiple text position information corresponding to the multiple bounding boxes;
[0113] S404: Combining the multiple text pixel regions into a single line of text.
[0114] In the embodiment of the present application, the bboxes of the text position information belonging to the same row after clustering are combined or arranged into a row in a horizontal order, referring to Fig. 9 and 10 , which shows a schematic diagram of the effect of visually displaying multiple bboxes in the same row on the original image. Fig. 9 is a panoramic image of Chinese text, Fig.10 This is a panoramic image of English text, where you can see that the heights of the various bboxes are different. To avoid offset or jitter after subsequent grouping, it is preferred that the heights of the bboxes in the same row be adjusted to be consistent. Optionally, a suitable height value can be selected as a reference for adjustment. For example, all the bboxes (K in total) in the same row are arranged from small to large according to h, and the h value of the 0.85×Kth bbox is taken as the h value of the entire row. That is, assuming that the h value of the 0.85×Kth bbox is 1.2 cm, the heights of the K bboxes in the row are all adjusted to 1.2 cm.
[0115] Preferably, after adjusting the height of the bbox itself, it is also necessary to adjust the vertical position of the bbox, and adjust each bbox to be at the same height by translating the bbox vertically.
[0116] Finally, according to the multiple text position information corresponding to each bbox, the corresponding multiple text pixel areas are obtained from the original image, that is, the corresponding characters in the original image are grouped into lines to form a single line of text. Fig.11 and Fig.12 They are shown respectively Fig. 9 and Fig.10 Schematic diagram of the combined single-line text, the grouping result is accurate.
[0117] Based on the above description, the embodiment of the present application uses a text detection model based on the Adaboost algorithm to detect images and obtain multiple text position information in the image. When the text is grouped, clustering processing is performed based on the multiple text position information to obtain the number of text lines. Then, through the grouping of the text position information, the corresponding text pixel area can be finally grouped. Using the embodiment of the present application, accurate grouping can be achieved for complexly arranged text images to be identified, which is beneficial to improving the accuracy of text recognition.
[0118] In the embodiments of the present application, optical character recognition (OCR) processing can be performed on the obtained single-line text or multiple-line text, and the accuracy of OCR recognition is high. The display module of the electronic scanning pen in some embodiments of the present application includes an LCD display screen for displaying OCR recognition results.
[0119] Exemplary Media
[0120] After introducing the method of the exemplary embodiment of the present invention, next, refer to Fig.13 A medium according to an exemplary embodiment of the present invention is described.
[0121] In some possible embodiments, various aspects of the present invention may also be implemented as a computer-readable medium having a program stored thereon, which, when executed by a processor, is used to implement the steps of the text image processing method according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification.
[0122] When the processor executes the program, for example, it is used to implement the following steps:
[0123] Preprocessing the target text to obtain a target image corresponding to the target text;
[0124] Acquire text position information corresponding to at least part of the text in the target image;
[0125] Based on the text position information, text line processing is performed to obtain a single line of text or multiple lines of text.
[0126] It should be noted that the above-mentioned medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
[0127] like Fig.13 As shown, a medium 60 according to an embodiment of the present invention is described, which can be a portable compact disk read-only memory (CD-ROM) and include a program and can be run on a device. However, the present invention is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device.
[0128] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0129] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).
[0130] Exemplary Devices
[0131] After introducing the medium of the exemplary embodiment of the present application, next, reference is made to Fig.14 The device according to the exemplary embodiment of the present application is described.
[0132] The present application also provides a text image processing device. Fig.14 As shown, the text image processing device may include:
[0133] A preprocessing module 110 is used to preprocess the target text to obtain a target image corresponding to the target text;
[0134] A text position information acquisition module 120, used to acquire text position information corresponding to at least part of the text in the target image;
[0135] The text line grouping processing module 130 is used to perform text line grouping processing based on the text position information to obtain a single line of text or multiple lines of text.
[0136] In a possible implementation, the preprocessing module includes:
[0137] The splicing processing submodule is used to splice the scanned image sequence of the target text obtained by scanning the target object once or multiple times using a scanning device to obtain a target image corresponding to the target text.
[0138] In a possible implementation manner, the image is a panoramic image.
[0139] In a possible implementation, the text position information acquisition module processes the target image using a text detection model based on an adaptive enhancement algorithm to acquire text position information corresponding to at least part of the text in the target image.
[0140] In a possible implementation, the text detection model is generated by training a classifier using training data, wherein the training data includes positive sample data and negative sample data, wherein:
[0141] The positive sample data includes a plurality of text units intercepted from a sample image, and the label data corresponding to each text unit is text position information of each text unit in the sample image;
[0142] The negative sample data includes a plurality of non-text images.
[0143] In a possible implementation, the text position information includes position and size information of a bounding box bbox of the text unit in the sample image.
[0144] In a possible implementation, the characters in the character unit meet at least one of the following conditions: there is no incomplete character in the character unit, the number of characters in the character unit is one, and the character unit contains only one complete character.
[0145] In a possible implementation, the text position information acquisition module includes:
[0146] An image pyramid construction submodule, used to construct an image pyramid based on the target image, and use each layer of images in the image pyramid as an image to be recognized;
[0147] The image pyramid processing submodule is used to input the images of each layer of the image pyramid into the text detection model to obtain a plurality of text position information corresponding to a plurality of text units output by the text detection model.
[0148] In a possible implementation, the text group line processing module includes:
[0149] A cluster center setting submodule, used for setting n0 cluster centers for the plurality of text position information based on a clustering algorithm;
[0150] The clustering processing submodule is used to perform clustering processing on the multiple text position information according to the positional relationship between the multiple text position information and the n0 initial clustering centers, so as to obtain a single line of text or multiple lines of text.
[0151] In a possible implementation, the clustering processing submodule includes:
[0152] A classification unit, used for taking the midpoints of two adjacent initial cluster centers among the n0 initial cluster centers as classification points to obtain n0 classes;
[0153] A text position information classification component is used to classify each text position information into the class with the closest distance;
[0154] The cluster center calculation component is used to recalculate the cluster centers based on the text position information in the n0 classes after the division, and obtain n1 cluster centers;
[0155] Among them, the classification unit, the text position information division component and the cluster center calculation component repeat processing until the number and position of cluster centers no longer change, and finally obtain n cluster centers, among which multiple text position information corresponding to the same cluster center are in the same row.
[0156] In a possible implementation manner, the text group line processing module further includes:
[0157] A bounding box row grouping module is used to group multiple bounding boxes corresponding to multiple text position information in the same row into one row in horizontal order;
[0158] A bounding box adjustment module, configured to adjust the heights of the plurality of bounding boxes combined into a row so that the heights of the plurality of bounding boxes are consistent;
[0159] A text pixel acquisition module, used for acquiring corresponding multiple text pixel areas from the target image according to multiple text position information corresponding to the multiple bounding boxes;
[0160] The text pixel grouping module is used to group the multiple text pixel areas into a single line of text.
[0161] In a possible implementation, the device further includes a text recognition module, configured to perform optical character recognition (OCR) processing on the single-line text or multiple-line text.
[0162] Exemplary Computing Devices
[0163] After introducing the method, medium and device of the exemplary embodiment of the present application, next, reference is made to Fig.15 A computing device according to an exemplary embodiment of the present application is described.
[0164] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0165] In some possible implementations, a computing device according to an implementation of the present application may include at least one processing unit and at least one storage unit. The storage unit stores a program code, and when the program code is executed by the processing unit, the processing unit executes the steps of the text image processing method according to various exemplary implementations of the present application described in the above “Exemplary Method” section of this specification.
[0166] Refer to the following Fig.15 hereinafter, a computing device 80 according to this embodiment of the present application is described. Fig.15 The computing device 80 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0167] like Fig.15 As shown, the computing device 80 is in the form of a general computing device. The components of the computing device 80 may include but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802).
[0168] The bus 803 includes a data bus, a control bus, and an address bus.
[0169] The storage unit 802 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8021 and / or a cache memory 8022 , and may further include a readable medium in the form of a non-volatile memory, such as a read-only memory (ROM) 8023 .
[0170] The storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0171] In a possible implementation, the computing device 80 includes an electronic scanning device, such as an electronic scanning pen.
[0172] The computing device 80 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 805. Furthermore, the computing device 80 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. The network adapter 806 communicates with other modules of the computing device 80 via the bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0173] It should be noted that, although several units / modules or sub-units / sub-modules of the text image processing device are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above may be embodied in one unit / module. Conversely, the features and functions of one unit / module described above may be further divided into multiple units / modules for embodiment.
[0174] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.
[0175] Although the spirit and principle of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.
Claims
1. A text image processing method, It is characterized in that The method comprises: Preprocessing the target text to obtain a target image corresponding to the target text; The target image is processed using a text detection model based on an adaptive enhancement algorithm to obtain text position information corresponding to at least part of the text in the target image; the text detection model is generated by training a classifier using training data, the training data includes positive sample data and negative sample data, wherein the positive sample data includes a plurality of text units intercepted from a sample image, the label data corresponding to each text unit is the text position information of each text unit in the sample image, and the negative sample data includes a plurality of non-text images; the text position information corresponding to at least part of the text includes a plurality of text position information corresponding to a plurality of text units, each of the text units contains only one complete text; the text position information includes the position and size information of a bounding box bbox of the text unit in the sample image; the position information of the bounding box bbox includes the coordinates of the upper left corner vertex of the bounding box bbox; Perform text grouping processing based on the text position information to obtain a single-line text or multiple-line text; The step of performing text line grouping processing based on the text position information to obtain a single-line text or multiple-line text includes: Based on a clustering algorithm, setting n0 initial clustering centers for the plurality of text position information; According to the positional relationship between the multiple text position information and the n0 initial clustering centers, the multiple text position information are clustered to obtain a single line of text or multiple lines of text.
2. The method according to claim 1, It is characterized in that The preprocessing of the target text includes: A target image corresponding to the target text is obtained by performing a splicing process on a scanned image sequence of the target text obtained by scanning the target object once or multiple times using a scanning device to obtain a target image.
3. The method according to claim 2, It is characterized in that The target image is a panoramic image.
4. The method according to claim 1, It is characterized in that The obtaining of text position information corresponding to at least part of the text in the target image includes: Building an image pyramid based on the target image, and using each layer of the image in the image pyramid as an image to be recognized; The images of each layer of the image pyramid are input into the text detection model to obtain a plurality of text position information corresponding to a plurality of text units output by the text detection model.
5. The method according to claim 1, It is characterized in that The clustering of the plurality of text position information according to the positional relationships between the plurality of text position information and the n0 initial clustering centers includes: The midpoints of two adjacent initial cluster centers among the n0 initial cluster centers are used as classification division points to obtain n0 classes; Divide each text position information into the class with the closest distance; Based on the text position information in the n0 divided classes, the cluster centers are recalculated to obtain n1 cluster centers; The above process is repeated until the number and position of cluster centers no longer change, and n cluster centers are finally obtained, where multiple text position information corresponding to the same cluster center are in the same row.
6. The method according to claim 5, It is characterized in that The method further comprises: Combining multiple bounding boxes corresponding to multiple text position information in the same row into one row in horizontal order; Adjusting the heights of the plurality of bounding boxes combined into a row so that the heights of the plurality of bounding boxes are consistent; Acquire corresponding multiple text pixel regions from the target image according to multiple text position information corresponding to the multiple bounding boxes; The multiple text pixel regions are combined into a single line of text.
7. The method according to claim 1, It is characterized in that The method further comprises, Perform optical character recognition (OCR) processing on the single-line text or multiple-line text.
8. A text image processing device, It is characterized in that include: A preprocessing module, used to preprocess the target text to obtain a target image corresponding to the target text; A text position information acquisition module, used to process the target image using a text detection model based on an adaptive enhancement algorithm, and obtain text position information corresponding to at least part of the text in the target image, wherein the text detection model is generated by training a classifier using training data, and the training data includes positive sample data and negative sample data, wherein the positive sample data includes multiple text units intercepted from a sample image, and the label data corresponding to each text unit is the text position information of each text unit in the sample image, and the negative sample data includes multiple non-text images; the text position information corresponding to at least part of the text includes multiple text position information corresponding to multiple text units, and each of the text units contains only one complete text; the text position information includes the position and size information of a bounding box bbox of the text unit in the sample image; the position information of the bounding box bbox includes the coordinates of the upper left corner vertex of the bounding box bbox; A text line processing module, used for performing text line processing based on the text position information to obtain a single line of text or multiple lines of text; The text group line processing module includes: A cluster center setting submodule, used for setting n0 cluster centers for the plurality of text position information based on a clustering algorithm; The clustering processing submodule is used to perform clustering processing on the multiple text position information according to the positional relationship between the multiple text position information and the n0 initial clustering centers, so as to obtain a single line of text or multiple lines of text.
9. The device according to claim 8, It is characterized in that The preprocessing module comprises: The splicing processing submodule is used to splice the scanned image sequence of the target text obtained by scanning the target object once or multiple times using a scanning device to obtain a target image corresponding to the target text.
10. The device according to claim 9, It is characterized in that The target image is a panoramic image.
11. The device according to claim 8, It is characterized in that The text position information acquisition module includes: An image pyramid construction submodule, used to construct an image pyramid based on the target image, and use each layer of images in the image pyramid as an image to be recognized; The image pyramid processing submodule is used to input the images of each layer of the image pyramid into the text detection model to obtain a plurality of text position information corresponding to a plurality of text units output by the text detection model.
12. The device according to claim 8, It is characterized in that The cluster processing submodule includes: A classification unit, used for taking the midpoints of two adjacent initial cluster centers among the n0 initial cluster centers as classification points to obtain n0 classes; A text position information classification component is used to classify each text position information into the class with the closest distance; The cluster center calculation component is used to recalculate the cluster centers based on the text position information in the n0 classes after the division, and obtain n1 cluster centers; Among them, the classification unit, the text position information division component and the cluster center calculation component repeat processing until the number and position of cluster centers no longer change, and finally obtain n cluster centers, among which multiple text position information corresponding to the same cluster center are in the same row.
13. The device according to claim 12, It is characterized in that The text group line processing module also includes: A bounding box row grouping module is used to group multiple bounding boxes corresponding to multiple text position information in the same row into one row in horizontal order; A bounding box adjustment module, configured to adjust the heights of the plurality of bounding boxes combined into a row so that the heights of the plurality of bounding boxes are consistent; A text pixel acquisition module, used for acquiring corresponding multiple text pixel areas from the target image according to multiple text position information corresponding to the multiple bounding boxes; The text pixel grouping module is used to group the multiple text pixel areas into a single line of text.
14. The device according to claim 8, It is characterized in that The device also includes: The text recognition module is used to perform optical character recognition (OCR) processing on the single-line text or multiple-line text.
15. A computing device, It is characterized in that include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the text image processing method according to any one of claims 1 to 7.
16. The computing device according to claim 15, It is characterized in that The computing device includes an electronic scanning device.
17. A non-volatile computer-readable storage medium storing a computer program, It is characterized in that When the program is executed by a processor, the text image processing method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Video subtitle information extracting method
CN101453575A
Domain-based self-adaptive English scene character recognition method
CN104537362A
Method for recognizing character contents included in image and device thereof
CN104899586A
Target recognition method and target recognition device
CN105095905A
Image processing method and apparatus
CN105404846A
