Text content extraction and recognition method based on line text frame word segmentation algorithm

Abstract

The present invention discloses a text content extraction and recognition method based on a line text box word segmentation algorithm. According to the position information, semantic information, and finger coordinate information of the line text box, a simple and efficient algorithm is used to extract the words closest to the finger. The algorithm is simple and efficient, and solves the problem of insufficient server resources. At the same time, the present invention uses target detection, hand detection and recognition, text detection, and text recognition to find the coordinates of the key points of the finger, detect the position information of the text box in the picture, and recognize the text content in the target text box in the picture. After using the existing model and post-training, the accuracy is improved and the speed is faster, which can better cooperate with the finger word selection algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and text recognition, and in particular relates to a text content extraction and recognition method based on a line text frame word segmentation algorithm. Background Art

[0002] Nowadays, the times are developing rapidly. In the past, the post-80s and post-90s held heavy Oxford and Xinhua dictionaries and racked their brains to explore ways to quickly look up dictionaries. Now, the post-00s and post-10s students are already using electronic dictionary pens to look up new words. A Sensor (infrared light) + MCU + OID algorithm + special coating printing that reflects infrared light. This is the most basic hardware and software architecture of a point-reading pen, which is a signal reading device + storage card + processing chip + point-reading material with secret codes + pronunciation device. The key to point-reading technology is OID (Optical Identify Digital) coding and a point-reading pen that can recognize OID coding. OID is a two-dimensional optical identification code that identifies different contents through different combinations of codes; the point-reading pen method is not only expensive, but also inconvenient to carry and has high operational complexity. To improve the complexity, time consumption, accuracy and other problems of the above operations, it is necessary to involve related technologies such as OCR technology, text content extraction and merging in computer vision. OCR refers to the process in which an electronic device (such as a scanner or digital camera) checks the characters on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into a computer text description. Text box filtering is to filter out the corresponding text boxes based on coordinates.

[0003] The Chinese patent with publication number CN110705282 A proposes a keyword extraction method, specifically: obtain a first text to be subjected to keyword extraction; segment the first text to obtain multiple segmentations; input the multiple segmentations into a word graph model to obtain a weight value corresponding to each segmentation; perform keyword extraction on the first text according to the weight value corresponding to each segmentation; wherein the word graph model is used to determine the weight value of each segmentation in the following manner: obtain a target word graph; in the target word graph, determine the first edge weight between the node corresponding to the first segmentation and the node corresponding to the second segmentation; determine the second edge weight between the node corresponding to the first segmentation and the node corresponding to the second segmentation in the preset word graph; determine the weight value of the node corresponding to the first segmentation in the target word graph according to the first edge weight and the second edge weight. Although this patented technology belongs to keyword extraction, it is obtained by putting the obtained multiple keywords into a neural network for calculation, and there is still a certain degree of error.

[0004] The Chinese patent with publication number CN111694946A proposes a method for visualizing and displaying text keywords, specifically: receiving text data to be audited to obtain corresponding plain text data; segmenting the plain text data to obtain corresponding segmentation results; filtering the segmentation results for noise words to obtain segmentation filtering results, which include non-noise word groups and noise word groups; extracting keywords from each segmentation in the non-noise word group to obtain keywords that do not exceed the preset ranking threshold to form a keyword set; converting the keywords included in the keyword set into word clouds, and sending the word clouds corresponding to each keyword to the user end for display. This patented technology is to perform text detection, recognition and segmentation on the text, sort it according to the frequency of word occurrence, and then filter out the high-frequency words and send them to the user end. In this way, the keywords found are inaccurate, and generally multiple keywords will be displayed. Summary of the invention

[0005] In view of the above, the present invention provides a text content extraction and recognition method based on a line text box word segmentation algorithm. The method uses a finger to point out the words you want to view under the device camera and uploads them to a server in the form of a picture. The word can be extracted through a simple algorithm, thereby extracting the word in the line text box pointed by the finger and presenting the word in the line text box.

[0006] A text content extraction and recognition method based on a line text frame word segmentation algorithm comprises the following steps:

[0007] (1) For text images of books and test papers, the images are preprocessed first, and then the location information of all text boxes in the images is extracted using the existing text detection algorithm;

[0008] (2) Using the existing target detection model, the hand area pointing to the question is selected in the form of a rectangular frame, and then the position information of each key node of the finger is detected in the hand area;

[0009] (3) using the position information of the line text box and the key node of the finger, extracting the line text box where the word pointed by the finger is located as the target text box, and using the existing text recognition method to recognize the character content in the target text box;

[0010] (4) Using the existing word segmentation algorithm to perform word segmentation on the text content in the target text box;

[0011] (5) calculating the length of each word and punctuation mark in the target text box on the x-axis relative to the initial position of the target text box and the length of the fingertip node on the x-axis relative to the initial position of the target text box;

[0012] (6) Using the information calculated in step (5) for comparison, the word pointed to by the finger is identified from the target text box based on the comparison result.

[0013] Furthermore, the image preprocessing in step (1) includes image perspective transformation and mean filtering denoising, wherein the transformation matrix used in the image perspective transformation is automatically adjusted by the height and angle of the captured image; using this method can improve the accuracy of subsequent text detection and text recognition.

[0014] Furthermore, in the step (1), a text detection algorithm based on PaddleOCR is used to identify and extract the text boxes in the image. On the basis of using the Paddle pre-trained model, training is performed using a text image dataset of books and test papers, which can effectively extract text boxes including text, punctuation marks, and underlines, effectively compensating for the occurrence of Paddle detection and recognition errors and recognition failures, and improving the accuracy of model detection and recognition.

[0015] Furthermore, in the step (2), the YOLOv5 model is used to select the hand area pointing to the question in the form of a rectangular frame, and the position information of each key node of the finger is detected by using skeleton detection.

[0016] Furthermore, the target text box extracted in step (3) is the line text box closest to the fingertip node, and then the RNN model is used to recognize the character content in the target text box. On the basis of using the RNN pre-trained model, training is performed using a text image dataset about books and test papers, which can effectively recognize various character contents including text, punctuation marks, and underlines.

[0017] Furthermore, in step (4), the Jieba word segmentation algorithm is used to perform word segmentation processing on the text content in the target text box.

[0018] Furthermore, the specific implementation process of step (5) is as follows:

[0019] First, calculate the width x_width of the target text box;

[0020] x_width=x_left-x_right

[0021] Among them: x_left and x_right are the x-axis coordinate values ​​of the upper left and upper right vertices of the target text box respectively;

[0022] Then, calculate the average width of the characters in the target text box, sing_word_len;

[0023] sing_word_len=x_width / sing_word_sums

[0024] Where: sing_word_sums is the number of characters in the text box;

[0025] Then, the length of each word and punctuation mark in the target text box on the x-axis relative to the initial position in the target text box is calculated and saved. Specifically:

[0026] If the current word or punctuation mark is the first word or punctuation mark from left to right in the target text box, then:

[0027] words_len=term_nums*sing_word_len

[0028] In other cases,

[0029] words_len=(term_nums*sing_word_len)+pred_words_len

[0030] Where: words_len is the length of the current word or punctuation mark on the x-axis relative to the initial position in the target text box, term_nums is the number of characters in the current word, and pred_words_len is the length of the previous word or punctuation mark relative to the initial position in the target text box;

[0031] Finally, calculate the length relative_x1 of the finger tip node on the x-axis relative to the initial position of the target text box;

[0032] relative_x1=x1-x_left

[0033] Where: x1 is the x-axis coordinate value of the fingertip node.

[0034] Furthermore, the specific implementation method of step (6) is as follows: in the target text box, from left to right, the length of each word and punctuation mark on the x-axis relative to the initial position of the target text box is compared with the length relative_x1 of the fingertip node on the x-axis relative to the initial position of the target text box; if the length of the current word or punctuation mark on the x-axis relative to the initial position of the target text box is words_len>relative_x1, the current word or punctuation mark is excluded and the next one is judged until the word or punctuation mark corresponding to words_len≤relative_x1 is found; if a word is found, the word is the word pointed to by the finger; if a punctuation mark is found and the punctuation mark is the last character from left to right in the target text box, the word before the punctuation mark is the word pointed to by the finger; if the punctuation mark is not the last character from left to right in the target text box, the word after the punctuation mark is the word pointed to by the finger.

[0035] The present invention uses a simple and efficient algorithm to extract the words closest to the finger based on the position information, semantic information, and finger coordinate information of the line text box. The algorithm is simple and efficient, and solves the problem of insufficient server resources. At the same time, the present invention uses target detection, hand detection and recognition, text detection, and text recognition to find the coordinates of the key points of the finger, detect the position information of the text box in the picture, and recognize the text content in the target text box in the picture. After using the existing model and post-training, the accuracy is improved and the speed is faster, which can better cooperate with the finger word selection algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The figure is a flow chart of the text content extraction and recognition method of the present invention.

[0037] Figure 2 This is the effect picture after image preprocessing.

[0038] Figure 3 This is the image result obtained after text detection.

[0039] Figure 4 This is the image result obtained after hand area detection.

[0040] Figure 5 This is the image result obtained after hand key point detection.

[0041] Figure 6 This is an example of a text box after filtering.

[0042] Figure 7 This is an example of the results obtained by text recognition.

[0043] Figure 8 This is an example of the result obtained after the word segmentation algorithm.

[0044] Fig. 9 This is an example of the final result obtained by the word selection module. DETAILED DESCRIPTION

[0045] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0046] The text content extraction and recognition method based on the line text frame word segmentation algorithm of the present invention selects the line text frame closest to the finger, and extracts the words pointed by the finger from the line text frame obtained by the frame screening according to the position information of the key node of the finger, and discards other words in the line text frame. The specific implementation process is as follows Figure 1 As shown, the following steps are included:

[0047] (1) Image preprocessing.

[0048] This embodiment uses image perspective transformation, brightness enhancement, and mean filter denoising algorithm to process the image, wherein the perspective transformation matrix M is automatically adjusted by the height and angle of the captured image. This method can improve the accuracy of subsequent text detection and text recognition.

[0049] The image effect after preprocessing is as follows Figure 2 shown.

[0050] (2) Text detection.

[0051] Text detection is performed on the preprocessed image. This implementation uses a text detection algorithm based on PaddleOCR to select the text box in the image. On the basis of using the Paddle pre-trained model, it uses a data set suitable for this scenario for training and fine-tunes the text detection model parameters. It can effectively extract text boxes including text, punctuation marks, and underlines, effectively compensating for the occurrence of Paddle detection and recognition errors and recognition failures, and improving the accuracy of model detection and recognition.

[0052] The trained model can identify all text boxes in the image and return the position information of the four vertices of the text box. The image result obtained after text detection is as follows: Figure 3 shown.

[0053] (3) Hand area detection.

[0054] The target detection algorithm is used in the original image to detect the position of the hand. In this implementation, the yolov5 model is used to select the hand area with a rectangular frame. Based on the target detection pre-trained model, pictures with hand areas are used for training, and the parameters of the target detection model are fine-tuned. This approach can effectively extract the hand area, which is convenient for the next step of finger key point detection.

[0055] The image results obtained after hand area detection are as follows Figure 4 shown.

[0056] (4) Finger key point detection.

[0057] In the hand area obtained in step (3), the finger skeleton key point detection can be used to obtain the position information of each key point of the hand, including the coordinate information (x, y) of the finger tip point. Then, the (x, y) coordinate is transformed into (x1, y1) according to the perspective transformation matrix M. This operation can accurately find the finger position information so as to locate the text box position below. The image result obtained after the hand key point detection is as follows: Figure 5 shown.

[0058] The formula for transforming (x,y) coordinates to (x1,y1) is as follows:

[0059] x1=(M[0][0]*p[0]+M[0][1]*p[1]+M[0][2]) / ((M[2][0]*p[0]+M[2][1]*p[1]+M[2][2])) y1=(M[1][0]*p[0]+M[1][1]*p[1]+M[1][2]) / ((M[2][0]*p[0]+M[2][1]*p[1]+M[2][2]))

[0060] (5)Text box filtering module.

[0061] The present invention uses the position information of the text box and the finger coordinates to filter the text box. The algorithm is simple and can effectively extract the text box closest to the finger according to the position information of the text box. The specific implementation process is as follows:

[0062] According to the finger positioning, filter the row text boxes that meet the conditions. According to the finger coordinate x1 obtained by the four finger key point detection, find the text boxes whose upper left corner x value is less than x1 and whose upper right corner x value is greater than x1; according to the finger coordinate y1, find the text boxes whose upper left corner y value is less than y1 among all the text boxes that meet the above x range, perform preliminary screening on the text boxes, and store the text box information obtained by the preliminary screening in the order of screening in the list.

[0063] The judgment criterion is the line text box where x_left <= x1 <= x_right and y_left < y1. Here, x_left is the x value of the upper left corner of the text box, x_right is the x value of the upper right corner of the text box, and y_left is the y value of the upper left corner of the text box.

[0064] Select the information of the last text box in the list obtained by the preliminary screening. The information of this text box is the position information of the text box closest to the key node of the finger.

[0065] An example of the text box after screening is Figure 6 as shown.

[0066] (6) Text recognition.

[0067] Use the text recognition algorithm with fine-tuned parameters for the text box screening result obtained in step (5) to obtain the content in the text box. This content is all the text descriptions of the text box pointed by the finger. Using the text recognition algorithm with fine-tuned parameters can effectively improve the accuracy of text recognition in this scenario, including the recognition of underlines and punctuation marks.

[0068] An example of the final result of text recognition is Figure 7 as shown.

[0069] (7) Word segmentation algorithm.

[0070] Use the jieba word segmentation algorithm for the text recognition result obtained in step (6) to combine the words in the text box into words, and store the punctuation marks and the combined words in order.

[0071] An example of the result after the word segmentation algorithm is Figure 8 as shown.

[0072] (8) Word selection module.

[0073] Use the word segmentation result obtained in step (7) and the position information of the text box obtained in step (2) to calculate the length of each word on the x-axis, i.e., the horizontal axis, relative to the initial position of the text box, and calculate the length of the finger key node relative to the initial position of the line text box. Compare the length of each word and punctuation mark in the text box relative to the initial position of the text box with the length of the finger key node relative to the initial position of the line text box, and then process the words or punctuation marks at special positions of the text box. Finally, obtain the word pointed by the finger. The specific implementation process is as follows:

[0074] Step 1: Calculate the length of each word on the x-axis, i.e., the horizontal axis, relative to the initial position of the text box.

[0075] First, calculate the target text box width, x_width = x_left-x_right, where x_left and x_right are the x values ​​of the upper left and upper right corners of the text box respectively, and x_width is the text box width; then calculate the average width of each character in the text box, sing_word_len = x_width / sing_word_sums, where sing_word_sums is the number of characters in the text box, and sing_word_len is the average width of each character in the text box.

[0076] Then calculate the length of each word or punctuation mark relative to the initial position of the text box and save it in a list. The subscript of the list represents the sequence number of the word or punctuation mark in the text box. The length of the word or punctuation mark relative to the initial position of the text box is the number of characters in the word multiplied by the average width of the text box occupied by each character sing_word_len plus the length of the previous word or punctuation mark relative to the initial position of the text box. If it is the first word in the text box, the length of the word or punctuation mark relative to the initial position of the text box is the number of characters in the word * the average width of the text box occupied by each character sing_word_len.

[0077] If the word is the first word in the text box, then:

[0078] words_len=term_nums*sing_word_len

[0079] If the word is not the first word in the text box, then:

[0080] words_len=(term_nums*sing_word_len)+pred_words_len

[0081] Where: term_nums is the number of characters of the word or punctuation mark, and pred_words_len is the length of the previous word or punctuation mark relative to the initial position of the text box.

[0082] Step 2: Calculate the length of the finger key node relative to the initial position of the text box in this line.

[0083] Calculate the length of the x value of the fingertip key point relative to the initial position of the text box:

[0084] relative_x1=x1-x_left

[0085] Among them: x1 is the x value of the key point of the fingertip, and relative_x1 is the length of the x value of the key point of the fingertip relative to the initial position of the text box.

[0086] Step 3: Use the length relative_x1 of the x - value of the fingertip key point of the finger relative to the initial position of the text box to compare with the length words_len of each word or punctuation mark relative to the initial position of the text box from left to right in sequence. If relative_x1 < words_len, it means that the word is not the word pointed by the finger, and continue to compare with the length words_len of the next word or punctuation mark relative to the initial position of the text box to the right until a word with relative_x1 ≥ words_len is found. This word is the word pointed by the finger. If the position where relative_x1 ≥ words_len is found is a punctuation mark instead of a word, then proceed to Step 4.

[0087] The judgment criterion is to start from the first word or punctuation mark of the text box and search in order to the right. When a word that meets relative_x1 ≥ words_len is found, retain this word and stop searching; where words_len is the length of the word or punctuation mark relative to the initial position of the text box.

[0088] Step 4: If the extracted word is the punctuation mark obtained by segmenting the text box, then extract the word at the position after this punctuation mark as the word pointed by the finger; if the extracted word is at the last position of the text box and the character at this position is a punctuation mark, then extract the word at the position before this punctuation mark as the word pointed by the finger.

[0089] An example of the final result obtained by the word - selection module is as Fig. 9 shown.

[0090] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and apply the present invention. It is obvious that those who are familiar with the technology in this field can easily make various modifications to the above - mentioned embodiments and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above - mentioned embodiments, and all improvements and modifications made by those skilled in the art according to the disclosure of the present invention should fall within the protection scope of the present invention.

Claims

1. A text content extraction and recognition method based on a line text frame word segmentation algorithm comprises the following steps: (1) For text images of books and test papers, the images are preprocessed first, and then the location information of all text boxes in the images is extracted using the existing text detection algorithm; (2) Using the existing target detection model, the hand area pointing to the question is selected in the form of a rectangular frame, and then the position information of each key node of the finger is detected in the hand area; (3) using the position information of the line text box and the key node of the finger, extracting the line text box where the word pointed by the finger is located as the target text box, and using the existing text recognition method to recognize the character content in the target text box; (4) Using the existing word segmentation algorithm to perform word segmentation on the text content in the target text box; (5) Calculate the length of each word and punctuation mark in the target text box on the x-axis relative to the initial position of the target text box and the length of the fingertip node on the x-axis relative to the initial position of the target text box. The specific implementation process is as follows: First, calculate the width x_width of the target text box; x_width=x_left-x_right in: x_left and x_right are the x-axis coordinate values ​​of the upper left and upper right vertices of the target text box respectively; Then, calculate the average width of the characters in the target text box, sing_word_len; sing_word_len=x_width / sing_word_sums Where: sing_word_sums is the number of characters in the text box; Then, the length of each word and punctuation mark in the target text box on the x-axis relative to the initial position in the target text box is calculated and saved. Specifically: If the current word or punctuation mark is the first word or punctuation mark from left to right in the target text box, then: words_len=term_nums*sing_word_len In other cases, words_len=(term_nums*sing_word_len)+pred_words_len Where: words_len is the length of the current word or punctuation mark on the x-axis relative to the initial position in the target text box, term_nums is the number of characters in the current word, and pred_words_len is the length of the previous word or punctuation mark relative to the initial position in the target text box; Finally, calculate the length relative_x1 of the finger tip node on the x-axis relative to the initial position of the target text box; relative_x1=x1-x_left Where: x1 is the x-axis coordinate value of the fingertip node; (6) Using the information calculated in step (5) for comparison, the word pointed to by the finger is identified from the target text box according to the comparison result. Specifically, from left to right in the target text box, the length of each word and punctuation mark on the x-axis relative to the initial position of the target text box is compared with the length relative_x1 of the fingertip node on the x-axis relative to the initial position of the target text box. If the length of the current word or punctuation mark on the x-axis relative to the initial position of the target text box is words_len>relative_x1, the current word or punctuation mark is excluded and the next one is determined until a word or punctuation mark corresponding to words_len≤relative_x1 is found. If a word is found, the word is the word pointed to by the finger. If a punctuation mark is found and the punctuation mark is the last character from left to right in the target text box, the word before the punctuation mark is the word pointed to by the finger. If the punctuation mark is not the last character from left to right in the target text box, the word after the punctuation mark is the word pointed to by the finger.

2. The text content extraction and recognition method according to claim 1, Features: The image preprocessing in step (1) includes image perspective transformation and mean filter denoising, wherein the transformation matrix used in the image perspective transformation is automatically adjusted according to the height and angle of the captured image.

3. The text content extraction and recognition method according to claim 1, Features: In the step (1), a text detection algorithm based on PaddleOCR is used to identify and extract text boxes in the image. Based on the use of the Paddle pre-trained model, training is performed using a text image dataset of books and test papers, which can effectively extract text boxes including text, punctuation marks, and underlines.

4. The text content extraction and recognition method according to claim 1, Features: In the step (2), the YOLOv5 model is used to select the hand area pointing to the question in the form of a rectangular frame, and the position information of each key node of the finger is detected by using skeleton detection.

5. The text content extraction and recognition method according to claim 1, Features: The target text box extracted in step (3) is the line text box closest to the fingertip node, and then the RNN model is used to recognize the character content in the target text box. On the basis of using the RNN pre-trained model, training is performed using a text image dataset about books and test papers, which can effectively recognize various character contents including text, punctuation marks, and underlines.

6. The text content extraction and recognition method according to claim 1, Features: In the step (4), the Jieba word segmentation algorithm is used to perform word segmentation processing on the text content in the target text box.

Images