Character recognition method and apparatus

Abstract

The present disclosure relates to the technical field of image processing, and provides a character recognition method and apparatus. The character recognition method comprises: a screenshot acquisition step: acquiring a screenshot of an area where a fingertip point of a user is located; an area textbox detection step: performing text detection on the acquired screenshot to acquire an area textbox; a matching step: acquiring a target area textbox matching the fingertip point of the user; and a detection and recognition step: performing individual character detection on the target area textbox, and on the basis of the individual character detection result or on the basis of the individual character detection result and character positioning information, determining a target character pointed to by the fingertip point of the user. The technical solution of the present disclosure can improve the accuracy of fingertip point character positioning and recognition.

Description

Character recognition method and device Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a character recognition method and apparatus. Background Technology

[0002] Reading assistance technologies include finger-reading technology, where users point to specific locations on a text with their finger, and the technology reads aloud the text at those locations or provides related information. This is particularly helpful in children's learning scenarios, as it aids in reading comprehension and improves reading efficiency.

[0003] Summary of the Invention

[0004] The technical problem to be solved by this disclosure is to provide a character recognition method and apparatus that can improve the accuracy of fingertip character positioning and recognition.

[0005] To address the aforementioned technical problems, the embodiments of this disclosure provide the following technical solutions:

[0006] On the one hand, a character recognition method is provided, including:

[0007] The steps to capture a screenshot are as follows: capture a screenshot of the area where the user's fingertip is located;

[0008] The region text box detection step involves performing text detection on the acquired screenshot to obtain region text boxes, which include row text boxes and / or column text boxes.

[0009] The matching step involves obtaining a text box representing the target area that matches the user's fingertip.

[0010] The detection and recognition step involves performing single-character detection on the target area text box, and determining the target character pointed to by the user's fingertip based on the single-character detection result or based on the single-character detection result and character positioning information.

[0011] In some embodiments, the screenshot acquisition step includes:

[0012] Get an image of the user's finger;

[0013] Determine the fingertip and nodal point of the user's finger from the image;

[0014] The fingertip direction and fingertip angle are calculated based on the fingertip and finger joint of the user's finger. The fingertip angle is the angle between the user's fingertip direction and the opposite direction of the x-axis of the image.

[0015] A screenshot is taken from the image based on the angle and location of the fingertip.

[0016] In some embodiments, the relationship between the screenshot and the angle and location of the fingertip satisfies any one of the following:

[0017] When the angle of the fingertip is greater than or equal to 0° and less than 75°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is two-thirds of the screenshot size;

[0018] When the angle of the fingertip is greater than or equal to 75° and less than 100°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-half of the screenshot size;

[0019] When the angle of the fingertip is greater than or equal to 100° and less than 180°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-third of the screenshot size.

[0020] In some embodiments, after the screenshot acquisition step, the method further includes:

[0021] The trapezoidal correction step involves performing trapezoidal correction on the screenshot.

[0022] In some embodiments, the method further includes:

[0023] When the number of detected area text boxes is 0, perform the following steps:

[0024] Step a: After enlarging the screenshot size, repeat the screenshot acquisition step;

[0025] Step b: Determine if the number of detected text boxes in the region is 0. If yes, proceed to step c; otherwise, do not execute the screenshot acquisition step.

[0026] Step c: Determine whether the number of times the screenshot size has been enlarged has reached the preset number. If not, proceed to step a; if so, do not execute the screenshot acquisition step again.

[0027] In some embodiments, the region text box is a line text box, and the matching step includes:

[0028] When the user's fingertip falls into the text box, a target text box matching the user's fingertip is determined based on the vertical distance between the user's fingertip and the four edges of the text box.

[0029] When the user's fingertip does not fall within the text box, a target text box matching the user's fingertip is determined based on the text box intersecting the user's fingertip direction and the vertical distance between the user's fingertip and the bottom edge of the text box.

[0030] In some embodiments, determining the target text box matching the user's fingertip based on the vertical distance between the user's fingertip and the four edges of the text box includes:

[0031] When the text box is a horizontal text box, if the vertical distance between the user's fingertip and the four edges of the text box satisfies: db < (dt + db) / 3 and min(dr, dl) > box_h / 4, then the text box is determined to match the user's fingertip. Here, db is the vertical distance between the user's fingertip and the bottom edge of the text box, dt is the vertical distance between the user's fingertip and the top edge of the text box, dl is the vertical distance between the user's fingertip and the left edge of the text box, dr is the vertical distance between the user's fingertip and the right edge of the text box, and box_h is the height of the text box.

[0032] When the text box is a curved text box, if the vertical distances between the user's fingertip and the four edges of the text box satisfy: db1 < (dt1 + db1) * 2 / 3 and min(dr1, dl1) > box_h 1 / 4, and abs(c_db) < (dt1 + db1) / 3, then the text box is determined to match the user's fingertip. Here, c_db is the minimum distance from the user's fingertip to the text polygon outline of the curved text box, db1 is the vertical distance between the user's fingertip and the bottom edge of the text box, dt1 is the vertical distance between the user's fingertip and the top edge of the text box, dl1 is the vertical distance between the user's fingertip and the left edge of the text box, and dr1 is the vertical distance between the user's fingertip and the right edge of the text box. The curved text box satisfies the following condition: the ratio of the area of ​​the text box to the area of ​​the text polygon outline is greater than a preset ratio.

[0033] In some embodiments, the method further includes:

[0034] When the height of the text box is greater than a preset first threshold, and the vertical distance between the user's fingertip and the bottom edge of the text box is less than the product of the height of the text box and a preset first ratio, it is determined that the user's fingertip falls into the text box.

[0035] In some embodiments, determining the target text box matching the user's fingertip based on the text box intersecting the user's fingertip direction and the vertical distance between the user's fingertip and the bottom edge of the text box includes:

[0036] If there exists a text box that intersects with the user's fingertip direction, and the vertical distance between the user's fingertip and the lower edge of the text box is less than a preset second threshold, the text box is determined to be a target text box that matches the user's fingertip. The second threshold is determined by the average height of the text boxes in the screenshot.

[0037] In some embodiments, if there is no text box intersecting the direction of the user's fingertip, the method further includes:

[0038] Increase the size of all text boxes in the screenshot;

[0039] Determine whether there is a text box that intersects with the user's fingertip direction. If not, identify the text box above the user's fingertip whose vertical distance between its lower edge and the user's fingertip is less than the second threshold as the target text box that matches the user's fingertip.

[0040] In some embodiments, the region text box is a line text box, and after the matching step, the method further includes:

[0041] The judgment step is to determine whether the preset map expansion conditions are met;

[0042] In the image enlargement step, if the image enlargement conditions are met, the screenshot size is enlarged, and the screenshot acquisition step is executed again.

[0043] The map expansion conditions include any one of the following:

[0044] When no target text box matching the user's fingertip is found, the ratio of the maximum height of all text boxes in the screenshot to the screenshot size is greater than a preset second ratio;

[0045] When a target line text box matching the user's fingertip is obtained, the ratio of the minimum vertical distance between the target line text box and the edge of the screenshot to the screenshot size is less than a preset third ratio, and the ratio of the height of the target line text box to the screenshot size is greater than a preset second ratio.

[0046] In some embodiments, the region text box is a line text box, and the detection and recognition steps include:

[0047] Based on the position information of the target line text box, obtain the image of the line text to be recognized corresponding to the target line text box from the screenshot;

[0048] The character positioning points are obtained from the text image to be recognized based on the direction of the user's fingertip.

[0049] The text image to be identified is subjected to single-character detection to obtain multiple single-character detection boxes. Single-character detection boxes that meet preset requirements are removed from the multiple single-character detection boxes. The preset requirements are: the width of the single-character detection box is less than the product of the average width of the single-character detection box and a preset multiple, and the minimum distance between the single-character detection box and the edge of the text image to be identified is less than a preset third threshold.

[0050] The text image to be recognized is cropped based on the removed single-character detection boxes, and the cropped text image to be recognized is used for character recognition to obtain recognized text including multiple characters.

[0051] If the number of characters in the recognized text is equal to the number of single-character detection boxes, determine the target single-character detection box corresponding to the character positioning point, and determine the target character among the multiple characters according to the sequence number of the target single-character detection box in the multiple single-character detection boxes;

[0052] If the number of characters in the identified text is not equal to the number of single-character detection boxes, the connected text classification (CTC) information in the convolutional recurrent neural network (CRNN) is used to estimate the character position and output the target character corresponding to the character positioning point.

[0053] In some embodiments, obtaining character positioning points from the text image to be recognized based on the user's fingertip direction includes:

[0054] When the target text box is a horizontal text box, if the acute angle between the fingertip direction and the lower edge of the target text box is greater than 45°, the intersection point of the first straight line and the fingertip direction is determined. The first straight line intersects the target text box, is parallel to the lower edge, and the distance between the first straight line and the lower edge is one-third of the height of the target text box. If the acute angle between the fingertip direction and the lower edge of the target text box is less than or equal to 45°, the intersection point of the lower edge and the fingertip direction is determined.

[0055] When the target text box is a curved text box, the fingertip is determined as the intersection point when it is located within the text polygon outline of the target text box; when the fingertip is located within the target text box but outside the text polygon outline, the intersection point of the second line and the fingertip direction is determined, wherein the second line is parallel to the lower edge of the target text box and the distance between the second line and the lower edge and the upper edge of the target text box is half the height of the target text box; when the fingertip is located outside the target text box, the intersection point of the second line and the fingertip direction is determined.

[0056] The coordinates of the intersection point are mapped to the text image to be identified to obtain the character positioning point.

[0057] In some embodiments, after obtaining multiple single-character detection boxes, the method further includes:

[0058] When the distance between two adjacent single-character detection boxes is greater than a preset fourth threshold, a virtual single-character detection box is added between the two adjacent single-character detection boxes. The fourth threshold is determined by the average width of the single-character detection boxes.

[0059] In some embodiments, after obtaining multiple single-character detection boxes, the method further includes:

[0060] When the target line text box is a curved text box, if the preset conditions are met, the S single-character detection boxes closest to the character positioning point are retained, and the line text image to be identified is shrunk according to the retained S single-character detection boxes, where S is a positive integer;

[0061] The preset conditions include:

[0062] The ratio of the aspect ratio of the text image to be identified to the number of single-character detection boxes is between 0.9 and 1.8, and the number of single-character detection boxes is greater than S.

[0063] In some embodiments, the method further includes:

[0064] If the target character is a punctuation mark or a space, replace the target character with the character that is the nearest non-punctuation mark or space to the target character.

[0065] In some embodiments, the method further includes:

[0066] If the target character is English, the recognized text is searched forward and backward from the target character until a punctuation mark or space is encountered, resulting in candidate English strings. The candidate English strings are then segmented, and the target character is replaced with the segmented result.

[0067] Embodiments of this disclosure also provide a character recognition device, including:

[0068] The screenshot acquisition module is used to capture screenshots of the area where the user's fingertip is located;

[0069] The region text box detection module is used to perform text detection on the acquired screenshot and obtain region text boxes, which include row text boxes and / or column text boxes.

[0070] The matching module is used to obtain the target area text box that matches the user's fingertip point;

[0071] The detection and recognition module is used to perform single-character detection on the target area text box, and determine the target character pointed to by the user's fingertip based on the single-character detection result or based on the single-character detection result and character positioning information.

[0072] Embodiments of this disclosure also provide a character recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the character recognition method as described above.

[0073] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the character recognition method described above.

[0074] The embodiments disclosed herein have the following beneficial effects:

[0075] In the above scheme, after performing text detection on the screenshot of the area where the user's fingertip is located, a text box for the area is obtained. Then, single-character detection is performed on the text box for the area. Based on the single-character detection results, the target character pointed to by the user's fingertip is determined. This embodiment adopts a scheme that combines single-character detection and text recognition results, which can improve the accuracy of fingertip character positioning and recognition. Attached Figure Description

[0076] Figure 1 is a flowchart illustrating the character recognition method according to an embodiment of this disclosure;

[0077] Figure 2 is a flowchart illustrating the process of determining the target text box that matches the user's fingertip when the user's fingertip falls into the text box according to an embodiment of this disclosure.

[0078] Figure 3 is a flowchart illustrating the process of determining the target text box that matches the user's fingertip when the user's fingertip falls outside the text box in an embodiment of this disclosure.

[0079] Figure 4 is a schematic diagram showing the effect of an enlarged screenshot of an embodiment of this disclosure;

[0080] Figure 5 is a flowchart illustrating the process of determining the target character pointed to by the user's fingertip in an embodiment of this disclosure;

[0081] Figure 6 is a schematic diagram of the original multiple single-character detection boxes in an embodiment of this disclosure;

[0082] Figure 7 is a schematic diagram of the removal of the single-character detection boxes with truncated beginning and end in an embodiment of this disclosure;

[0083] Figure 8 is a schematic diagram of multiple single-word detection boxes before shrinkage in an embodiment of this disclosure;

[0084] Figure 9 is a schematic diagram of multiple single-character detection boxes after shrinking according to an embodiment of this disclosure;

[0085] Figure 10 is a schematic diagram showing that the number of characters in the identified text is equal to the number of single-character detection boxes in an embodiment of this disclosure;

[0086] Figure 11 is a schematic diagram showing that the number of characters included in the identified text in an embodiment of this disclosure is not equal to the number of single-character detection boxes;

[0087] Figure 12 is a schematic diagram of the character recognition device according to an embodiment of this disclosure. Detailed Implementation

[0088] To make the technical problems, technical solutions and advantages to be solved by the embodiments of this disclosure clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0089] In reading assistance technology, users point to specific locations on reading materials with their fingers. The camera of the reading assistance device can then take a picture of the area where the user's finger is pointing, analyze the image to identify the character pointed to by the fingertip, and read the character aloud or provide relevant information. However, due to factors such as text curvature and incomplete truncation of large characters, the accuracy of fingertip character positioning and recognition can be relatively low.

[0090] This disclosure provides a character recognition method and apparatus that can improve the accuracy of fingertip character positioning and recognition.

[0091] Embodiments of this disclosure provide a character recognition method, as shown in FIG1, including:

[0092] Step 101 of screenshot acquisition: Obtain a screenshot of the area where the user's fingertip is located;

[0093] The purpose of finger reading is to identify a single Chinese character, a single English word, or other type of character pointed to by the user's fingertip. Therefore, after capturing an image that includes the reading material and the user's finger, full-image recognition is not required. A screenshot can be taken from the captured image, and text detection and recognition can be performed on the screenshot. This reduces the amount of computation and improves the computation speed.

[0094] When taking a screenshot for the first time, you can use the preset size img_size, that is, the length and width of the screenshot are img_size.

[0095] Specifically, the fingertip point p1 and finger node p2 of the user's finger are determined from the captured image; the fingertip direction vector P and the fingertip angle angle are calculated based on the fingertip point p1 and finger node p2, wherein a planar coordinate system is established in the image, the x-axis is the line direction of the text in the reading material, the y-axis is perpendicular to the x-axis, and the fingertip angle is the angle between the user's fingertip direction and the x-axis; and a screenshot is taken from the image based on the fingertip angle and the location of the fingertip point.

[0096] Specifically, the relationship between the screenshot and the angle and location of the fingertip satisfies any one of the following:

[0097] When the angle of the fingertip is greater than or equal to 0° and less than 75°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is two-thirds of the screenshot size;

[0098] When the angle of the fingertip is greater than or equal to 75° and less than 100°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-half of the screenshot size;

[0099] When the angle of the fingertip is greater than or equal to 100° and less than 180°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-third of the screenshot size.

[0100] That is, if angle∈[0,75), starting from the fingertip point p1, extend the screenshot to the left by two-thirds of the screenshot size img_size, to the right by one-third of the screenshot size img_size, downwards by one-quarter of the screenshot size img_size, and upwards by three-quarters of the screenshot size img_size; if angle∈[100,180), starting from the fingertip point p1, extend the screenshot to the left by one-third of the screenshot size img_size, to the right by one-third of the screenshot size img_size, and upwards by one-quarter of the screenshot size img_size. The screenshot is taken by extending two-thirds of g_size downwards by one-quarter of the screenshot size img_size, and then extending three-quarters of the screenshot size img_size upwards. If angle∈[75,100), starting from the fingertip point p1, the screenshot is taken by extending one-half of the screenshot size img_size to the left, one-half of the screenshot size img_size to the right, one-quarter of the screenshot size img_size downwards, and then three-quarters of the screenshot size img_size upwards. This ensures that the captured screenshot includes the character pointed to by the user's fingertip.

[0101] Generally, cameras use a top-down view to capture images, which result in trapezoidal distortion. Compared to vertically captured images, text detection is more difficult. Therefore, after acquiring the screenshot, the method further includes a trapezoidal correction step to correct the screenshot, which can improve the accuracy of character recognition based on the screenshot.

[0102] Screenshots are typically rectangular. When performing trapezoidal correction, the perspective transformation matrix H of the entire image can be estimated based on camera intrinsic parameters and camera angle. Combined with the coordinates of the four points of the screenshot rectangle, the corresponding perspective transformation matrix M is calculated. This matrix M is then used to perform trapezoidal correction on the screenshot. Camera intrinsic parameters include focal length, lens blur radius, and image quality description parameters. Focal length represents the relationship between the image size and the focal length; lens blur radius refers to the distance from the optical center to the image edge; image quality description parameters include brightness range, noise level, and other indicators. The camera angle is the angle between the camera's shooting direction and the z-axis, which is perpendicular to the plane containing the x-axis and y-axis.

[0103] The region text box detection step 102 involves performing text detection on the acquired screenshot to obtain region text boxes, which include row text boxes and / or column text boxes.

[0104] The characters in the reading material can be arranged in rows or columns. When the characters are arranged in rows, the captured screenshot needs to be processed to detect the row text and obtain the row text boxes; when the characters are arranged in columns, the captured screenshot needs to be processed to detect the column text and obtain the column text boxes.

[0105] In this embodiment, the DBNet model can be used to perform text detection on the acquired screenshot to obtain the text box region. If no text box region is detected, it may be because the screenshot size is too small, the text exceeds the screenshot area, there is no text in the screenshot, or the text in the screenshot is too small to be detected. The screenshot size can be enlarged and the text detection can be performed again. If no text box region is detected, the screenshot size can be enlarged again until the preset enlargement number EXPAND_TIMES is reached. If no text box region is detected, it is assumed that the user's fingertip is not pointing to any character.

[0106] Specifically, when the number of detected area text boxes is 0, the following steps are performed:

[0107] Step a: After enlarging the screenshot size, repeat the screenshot acquisition step;

[0108] The screenshot size can be expanded according to the preset expansion ratio EXPAND_RATIO, that is, the size of each screenshot is EXPAND_RATIO times the size of the previous screenshot. EXPAND_RATIO is greater than 1, and can be 1.5, 2, 2.5 or 3, etc.

[0109] Step b: Determine if the number of detected text boxes in the region is 0. If yes, proceed to step c; otherwise, do not execute the screenshot acquisition step.

[0110] Step c: Determine whether the number of times the screenshot size has been enlarged has reached the preset number. If not, proceed to step a; if so, do not execute the screenshot acquisition step again.

[0111] Matching step 103: Obtain the target area text box that matches the user's fingertip point;

[0112] Taking a region text box as an example, we can first determine whether the user's fingertip is located within a certain text box. If the user's fingertip p1 falls within multiple text boxes det_box, we select a text box det_box from among them. The bottom edge (i.e., the bottom edge) of this text box is closest to the user's fingertip p1, and this text box det_box is taken as the text box containing the user's fingertip p1.

[0113] When the user's fingertip falls into the text box, as shown in Figure 2, the target text box matching the user's fingertip can be determined based on the vertical distance between the user's fingertip and the four edges of the text box. The text box where the user's fingertip p1 is located is called rec_box. The height of rec_box needs to be greater than a preset first threshold. This is because if the height of rec_box is too small, such as less than 8 pixels, it can be ignored. If character recognition is performed on rec_box, the recognized characters will be blurry due to their small size, and the recognition accuracy cannot be guaranteed.

[0114] Specifically, when the height of the text box (box_h) is greater than a preset first threshold (BIG_CHAR_THRE), and the vertical distance between the user's fingertip and the bottom edge of the text box is less than the product of the text box's height (box_h) and a preset first ratio (INBOX_RATIO), it is determined that the user's fingertip falls within the text box. That is, when the text box is a large text box and the user's fingertip (p1) is close to the bottom edge of rec_box, it is determined that the user's fingertip falls within the text box. The value of the first threshold (BIG_CHAR_THRE) can be set as needed, for example, it can be 32 pixels. The first ratio (INBOX_RATIO) can also be set as needed, for example, it can be 0.3.

[0115] After determining that the user's fingertip falls within the text box, the target text box matching the user's fingertip can be further determined based on the vertical distances between the user's fingertip and the four edges of the text box. When the text box is a horizontal text box, if the vertical distances between the user's fingertip and the four edges of the text box satisfy the first condition: db < (dt + db) / 3 and min(dr, dl) > box_h / 4, then the text box is determined to match the user's fingertip. Here, db is the vertical distance between the user's fingertip and the bottom edge of the text box, dt is the vertical distance between the user's fingertip and the top edge of the text box, dl is the vertical distance between the user's fingertip and the left edge of the text box, dr is the vertical distance between the user's fingertip and the right edge of the text box, and box_h is the height of the text box. That is, the user's fingertip needs to be slightly below the text box and maintain a certain distance from the left and right edges of the text box. If this condition is met, it is confirmed that the user's fingertip is pointing to the current rec_box.

[0116] When the text box is a curved text box, to avoid text box matching failure and line skipping recognition, the judgment condition needs to be changed. A curved text box is defined as one where the text is not aligned in a straight line due to text curvature. First, based on the text box, the polygonal outline of the text within the text box can be extracted. If the ratio of the area Area(rec_box) of the text box to the area Area(rec_poly) of the polygonal outline of the text is greater than a preset ratio CURVED_RATIO_THRE, the text box is determined to be a curved text box. After determining that the text box is a curved text box, if the vertical distance between the user's fingertip and the four edges of the text box satisfies the second condition: db1 < (dt1 + db1) * 2 / 3 and min(dr1, dl1) > box_h 1 / 4, and abs(c_db) < (dt1 + db1) / 3, then the text box is determined to match the user's fingertip. Here, c_db is the minimum distance from the user's fingertip to the text polygon outline of the curved text box, db1 is the vertical distance between the user's fingertip and the lower edge of the text box, dt1 is the vertical distance between the user's fingertip and the upper edge of the text box, dl1 is the vertical distance between the user's fingertip and the left edge of the text box, and dr1 is the vertical distance between the user's fingertip and the right edge of the text box.

[0117] When the user's fingertip does not fall within the text box, as shown in Figure 3, a target text box matching the user's fingertip can be determined based on the text boxes intersecting the user's fingertip direction and the vertical distance between the user's fingertip and the bottom edge of the text box. Specifically, if a text box rec_box intersects the user's fingertip direction, and the vertical distance between the user's fingertip and the bottom edge of the rec_box is less than a preset second threshold p2l_thre, the text box is determined to be the target text box matching the user's fingertip. The second threshold p2l_thre is determined by the average height of the text boxes in the screenshot. Specifically, the second threshold p2l_thre = min(max(mean_h,20),30)*OUT_BOX_THRE, where OUT_BOX_THRE is the distance multiplier threshold for determining the distance to rec_box, and its value can be 1.5. If no text box intersects with the user's fingertip direction, the size of all text boxes in the screenshot can be increased. For example, all `det_box` text boxes can be increased by half a character height (`box_h`) on both sides. Then, it can be checked again to see if a text box intersects with the user's fingertip direction. If not, among the text boxes above the user's fingertip, the text box whose lower edge has a vertical distance less than the second threshold `p2l_thre` is identified as the target text box matching the user's fingertip. If multiple text boxes have a vertical distance less than the second threshold `p2l_thre` between their lower edges and the user's fingertip, the text box closest to the user's fingertip is selected.

[0118] In this embodiment, text recognition is performed on the crop_img screenshot. For text with a large pixel area, truncation is likely to occur, or text may be missed or the text box may be incomplete. Therefore, it is necessary to determine whether the text in the current reading material is large based on the height of the text box, that is, to determine whether the preset image expansion condition is met. If it is met, image expansion and re-detection are performed. Specifically, after the matching step 103 and before the detection and recognition step 104, the following steps may also be included:

[0119] The judgment step is to determine whether the preset map expansion conditions are met;

[0120] In the image enlargement step, if the image enlargement conditions are met, the screenshot size is enlarged, and the screenshot acquisition step is executed again.

[0121] The map expansion conditions include any one of the following:

[0122] When no target text box matching the user's fingertip is found, the ratio of the maximum height of all text boxes in the screenshot to the screenshot size is greater than a preset second ratio.

[0123] When a target line text box matching the user's fingertip is obtained, the ratio of the minimum vertical distance between the target line text box and the edge of the screenshot to the screenshot size is less than a preset third ratio, and the ratio of the height of the target line text box to the screenshot size is greater than a preset second ratio.

[0124] If step 103 does not find a rec_box that matches the user's fingertip, a large character judgment is performed. The maximum height of all text boxes in the screenshot, max_box_h, is calculated. If the ratio of max_box_h to the screenshot height crop_img_h is greater than the preset second ratio BIG_CHAR_RATIO_THRE, it is considered that a large character exists. The image expansion ratio is set to min(max(EXPAND_RATIO*max_box_h / BIG_CHAR_THRE,2),MAX_EXPAND_RATIO), where BIG_CHAR_THRE is the pixel threshold for large character judgment, the initial value of EXPAND_RATIO is 1, and MAX_EXPAND_RATIO is the maximum image expansion ratio.

[0125] If step 103 finds a rec_box that matches the user's fingertip, to avoid unnecessary image expansion, only cases where the rec_box is close to the edge of the screenshot are considered large text boxes. If the ratio of the minimum vertical distance between the target text box rec_box and the edge of the screenshot to the screenshot size is less than a preset third ratio, then the target text box rec_box is determined to be located at the edge of the screenshot. The current rec_box is then determined to be a large text box based on whether the ratio of the height box_h of the rec_box to the height crop_img_h of the screenshot is greater than a preset second ratio BIG_CHAR_RATIO_THRE.

[0126] After enlarging the screenshot area, repeat steps 101-103. Figure 4 shows the effect of enlarging the screenshot.

[0127] When the area text box is a column text box, we can first determine whether the user's fingertip is located within a certain column text box. If the user's fingertip p1 falls within multiple column text boxes det_box, we determine a column text box det_box whose left or right edge is closest to the user's fingertip p1, and this column text box det_box is taken as the column text box containing the user's fingertip p1.

[0128] When the user's fingertip falls within the column text box, a target column text box matching the user's fingertip can be determined based on the vertical distances between the user's fingertip and the four edges of the column text box. The column text box where the user's fingertip p1 is located is called rec_box. The width of rec_box needs to be greater than a preset threshold. This is because if the width of rec_box is too small, such as less than 8 pixels, it can be ignored. If character recognition is performed on this rec_box, the recognized characters will be blurry due to their small size, and the recognition accuracy cannot be guaranteed.

[0129] Specifically, when the width of the column text box is greater than a preset threshold BIG_CHAR_THRE, and the vertical distance between the user's fingertip and the left or right edge of the column text box is less than the product of the width of the column text box and a preset first ratio INBOX_RATIO, it can be determined that the user's fingertip falls within the column text box. That is, when the column text box is a large column text box and the user's fingertip p1 is close to the left or right edge of the column text box, it is determined that the user's fingertip falls within the column text box. The value of the preset threshold BIG_CHAR_THRE can be set as needed, for example, it can be 32 pixels, and the first ratio INBOX_RATIO can be set as needed, for example, it can be 0.3.

[0130] After determining that the user's fingertip falls within the column text box, a target column text box matching the user's fingertip can be further determined based on the vertical distance between the user's fingertip and the four edges of the column text box.

[0131] In detection and recognition step 104, single-character detection is performed on the target area text box, and the target character pointed to by the user's fingertip is determined based on the single-character detection result or based on the single-character detection result and character positioning information.

[0132] Taking the area text box as an example, as shown in Figure 5, the detection and recognition step 104 includes the following steps:

[0133] (1) First, the image of the line text to be recognized corresponding to the target line text box can be obtained from the screenshot according to the position information of the target line text box. The coordinates of rec_box that matches the user's fingertip point determined in step 103 are mapped back to the original image, that is, the screenshot before the trapezoidal correction is performed. The image of the line text to be recognized, crop_line_img, is cropped from the screenshot. Cropping the image from the screenshot before the trapezoidal correction can avoid the blurring of the text image caused by the difference in the trapezoidal correction.

[0134] (2) Obtain character positioning points from the text image to be recognized based on the direction of the user's fingertip;

[0135] In this embodiment, different schemes are used to determine character positioning points for horizontal text boxes and curved text boxes. Curved text boxes are those where the text is curved, and the characters within the text box are not aligned in a straight line. Based on the text box, the polygonal outline of the text within the text box can be extracted. If the ratio of the area of ​​the text box (Area(rec_box)) to the area of ​​the polygonal outline (Area(rec_poly)) is greater than a preset ratio (CURVED_RATIO_THRE), the text box is determined to be a curved text box.

[0136] When the target text box is a horizontal text box, if the acute angle between the fingertip direction and the lower edge of the target text box is greater than 45°, the intersection point of the first straight line and the fingertip direction is determined. The first straight line intersects the target text box, is parallel to the lower edge, and the distance between the first straight line and the lower edge is one-third of the height of the target text box. If the acute angle between the fingertip direction and the lower edge of the target text box is less than or equal to 45°, the intersection point of the lower edge and the fingertip direction is determined.

[0137] When the target text box is a curved text box, the fingertip is determined as the intersection point when it is located within the text polygon outline of the target text box; when the fingertip is located within the target text box but outside the text polygon outline, the intersection point of the second line and the fingertip direction is determined, wherein the second line is parallel to the lower edge of the target text box and the distance between the second line and the lower edge and the upper edge of the target text box is half the height of the target text box; when the fingertip is located outside the target text box, the intersection point of the second line and the fingertip direction is determined.

[0138] The coordinates of the intersection point are mapped to the crop_line_img image of the text to be identified to obtain the character positioning point.

[0139] (3) Perform single-character detection on the text image to be identified to obtain multiple single-character detection boxes;

[0140] Specifically, a YOLOv5 deep learning network can be used to perform single-character detection on the crop_line_img image of the text to be identified. In this embodiment, single-character detection is performed on the basis of line text detection, which takes less time, reduces the detection difficulty, and does not require consideration of text direction.

[0141] Since punctuation marks are easily missed, after obtaining multiple single-character detection boxes, the distance between two adjacent single-character detection boxes is analyzed. When the distance between two adjacent single-character detection boxes is greater than a preset fourth threshold, a virtual single-character detection box is added between the two adjacent single-character detection boxes. The fourth threshold is determined by the average width of the single-character detection boxes. The fourth threshold can be equal to the product of the average width of the single-character detection boxes and a preset multiplier r.

[0142] In this embodiment, a screenshot is taken during the text detection stage. Therefore, the first and last characters of the text line are prone to truncation, leading to unstable character recognition. Depending on the degree of truncation, recognition errors or failure to recognize single-character detection boxes may occur. When single-character detection boxes cannot be recognized, the number of detection boxes does not match the number of characters included in the recognized text `rec_text`, affecting character positioning accuracy. Therefore, it is necessary to remove truncated single-character detection boxes, i.e., remove single-character detection boxes that meet preset requirements from the plurality of single-character detection boxes. The preset requirements are: the width of the single-character detection box is less than the product of the average width of the single-character detection boxes and a preset multiplier, and the minimum distance between the single-character detection box and the edge of the text image to be recognized is less than a preset third threshold.

[0143] Specifically, based on the coordinates of the character detection box inv_char_box mapped back to the original image, it can be determined whether the character detection box is close to the edge of the text image crop_line_img to be recognized. Only for character detection boxes that are close to the edge of the text image crop_line_img to be recognized, the beginning and end truncation judgment is performed, because only character detection boxes at the edge of the text image crop_line_img to be recognized will be truncated. For the character detection boxes near the left edge of the text image crop_line_img (i.e., at the beginning of the text), if the width of the first character detection box is less than the product of the average width of the character detection boxes (mean_w) and r_c, and the distance between it and the left edge of the text image crop_line_img is less than mean_w / 2, then the first character detection box is removed. Here, r_c is a preset multiplier, such as 0.8. For the character detection boxes near the right edge of the text image crop_line_img (i.e., at the end of the text), if the width of the last character detection box is less than mean_w*r_c, and the distance between it and the right edge of the text image crop_line_img is less than mean_w / 2, then the last character detection box is removed. Figure 6 shows a schematic diagram of the original multiple character detection boxes in a specific example, and Figure 7 shows a schematic diagram after removing the truncated character detection boxes.

[0144] Furthermore, in picture book reading scenarios, the natural curvature of the book can easily produce curved text. To improve the accuracy of curved text recognition, a specific character length can be selected based on the single-character detection box to avoid some curved text. As the text length decreases, the degree of curvature decreases, the background area decreases, and the accuracy of character recognition can be improved. Specifically, when the target line text box is a curved text box, if a preset condition is met, the S single-character detection boxes closest to the character positioning point are retained. The crop_line_img image of the line text to be recognized is shrunk based on the retained S single-character detection boxes to obtain the shrunk line text image shrink_img, where S is a positive integer.

[0145] The preset conditions include:

[0146] The ratio of the aspect ratio of the text image to be identified to the number of single-character detection boxes is between 0.9 and 1.8, and the number of single-character detection boxes is greater than S.

[0147] Determining the aspect ratio of the text image to be recognized is to avoid truncating English words, as English characters differ from Chinese characters, and the output is English words. Figure 8 shows a schematic diagram of multiple single-character detection boxes before shrinking in a specific example, and Figure 9 shows a schematic diagram of multiple single-character detection boxes after shrinking.

[0148] (4) The text image to be identified is cropped according to the removed single character detection box, and the cropped text image to be identified is subjected to character recognition to obtain the recognized text including multiple characters.

[0149] Specifically, the CRNN model can be used to obtain the recognition text rec_text. When the character recognition device in this embodiment is deployed in the cloud, a large model can be used for recognition. When the character recognition device in this embodiment is deployed on the terminal, a lightweight model can be used for recognition.

[0150] (5) If the number of characters in the recognized text is equal to the number of single-character detection boxes, determine the target single-character detection box corresponding to the character positioning point, and determine the target character among the multiple characters according to the sequence number of the target single-character detection box in the multiple single-character detection boxes; if the number of characters in the recognized text is not equal to the number of single-character detection boxes, estimate the character position using the connected text classification (CTC) information of the recognition network, and output the target character corresponding to the character positioning point.

[0151] Figure 10 shows a specific example where the number of characters in the identified text is equal to the number of single-character detection boxes. When the number of characters in the identified text equals the number of single-character detection boxes, the single-character detection boxes are sorted according to their x-axis coordinates. The distance between the center point of each single-character detection box and the character positioning point can be calculated. The single-character detection box whose center point is closest to the character positioning point is determined as the target single-character detection box. The target character among the multiple characters is determined based on the sequence number of the target single-character detection box among the multiple single-character detection boxes. For example, if the sequence number of the target single-character detection box is 5, then the 5th character among the multiple characters is determined as the target character; if the sequence number of the target single-character detection box is 3, then the 3rd character among the multiple characters is determined as the target character, and so on. This embodiment uses a scheme of matching single-character detection results with line text recognition results, which can improve the accuracy of fingertip character positioning and recognition.

[0152] Figure 11 shows a specific example where the number of characters in the recognized text is not equal to the number of single-character detection boxes due to missed punctuation detection. In this embodiment, the character positioning information can specifically be CTC information. CTC information can be used to determine the positioning information of characters. When the number of characters in the recognized text is not equal to the number of single-character detection boxes, the character position can be estimated based on the CTC information in the text recognition result. Specifically, the character positioning point is converted into an index cross_idx. The index cross_idx is determined by the x-axis coordinate C_X of the character positioning point in the screenshot and the downsampling number stride of the recognition network on the input image width. Specifically, cross_idx = C_X / / stride, / / means rounding down. The downsampling number stride of the input image width is the input image slice width corresponding to each bit in the vector Y obtained by softmax calculation of the CTC information output probability map.

[0153] According to the CTC decoding rules, 'blank' is a placeholder. The index is decoded. If Y[cross_idx] is decoded as 'blank', the difference dist = C_X - cross_idx * stride is calculated. If dist is less than 0, starting from the placeholder 'blank', the nearest non-blank character in the recognized text is searched in the order of searching one character to the left and then one character to the right to obtain the target character. If dist is greater than or equal to 0, the nearest non-blank character in the recognized text is searched in the order of searching one character to the right and then one character to the left to obtain the target character.

[0154] In this embodiment, when the single-character detection result does not match the line text recognition result, the CTC information is used to estimate the position of the target character, which can ensure the accurate output of the target character.

[0155] (6) If the target character is a punctuation mark or a space, since punctuation marks or spaces are meaningless characters, the target character is replaced with the non-punctuation space character that is closest to the target character.

[0156] (7) If the target character is English, the recognition text is searched forward and backward from the target character until a punctuation mark or space is encountered to obtain a candidate English string. The candidate English string is segmented and the target character is replaced with the segmentation result.

[0157] Alternatively, the execution order of the above steps can also be (1)(2)(4)(3)(5)(6)(7), or (1)(4)(2)(3)(5)(6)(7).

[0158] When the area text box is a column text box, the detection and recognition step 104 includes the following steps:

[0159] (1) First, the text image of the column to be identified corresponding to the target column text box can be obtained from the screenshot according to the position information of the target column text box. The coordinates of rec_box that matches the user's fingertip point determined in step 103 are mapped back to the original image, that is, the screenshot before the trapezoidal correction is performed. The text image of the column to be identified crop_line_img is cropped from the screenshot. Cropping the image from the screenshot before the trapezoidal correction can avoid the blurring of the text image caused by the difference in the trapezoidal correction.

[0160] (2) Obtain character positioning points from the text image to be identified based on the direction of the user's fingertip;

[0161] (3) Perform single-character detection on the text image to be identified to obtain multiple single-character detection boxes;

[0162] Specifically, a YOLOv5 deep learning network can be used to perform single-character detection on the crop_line_img image of the column text to be identified. In this embodiment, single-character detection is performed on the basis of column text detection, which takes less time, reduces the detection difficulty, and does not require consideration of text direction.

[0163] Since punctuation marks are easily missed, after obtaining multiple single-character detection boxes, the distance between two adjacent single-character detection boxes is analyzed. When the distance between two adjacent single-character detection boxes is greater than a preset fourth threshold, a virtual single-character detection box is added between the two adjacent single-character detection boxes. The fourth threshold is determined by the average height of the single-character detection boxes, and the fourth threshold can be equal to the product of the average height of the single-character detection boxes and a preset multiplier r.

[0164] In this embodiment, screenshots are taken during the text detection stage. Therefore, the first and last characters of the column text are prone to truncation. The recognition of truncated characters is unstable; depending on the degree of truncation, recognition errors or failure to recognize single-character detection boxes may occur. When single-character detection boxes cannot be recognized, the number of single-character detection boxes does not match the number of characters included in the recognized text rec_text, affecting character positioning accuracy. Therefore, it is necessary to remove truncated single-character detection boxes, that is, remove single-character detection boxes that meet preset requirements from the plurality of single-character detection boxes. The preset requirements are: the height of the single-character detection box is less than the product of the average height of the single-character detection boxes and a preset multiplier, and the minimum distance between the single-character detection box and the edge of the column text image to be recognized is less than a preset third threshold.

[0165] Specifically, based on the coordinates of the single-character detection box inv_char_box mapped back to the original image, it can be determined whether the single-character detection box is close to the edge of the text image crop_line_img to be recognized. Only single-character detection boxes that are close to the edge of the text image crop_line_img to be recognized are subject to truncation, because only single-character detection boxes that are close to the edge of the text image crop_line_img to be recognized will be truncated. For the single-character detection box near the top edge of the crop_line_img image to be recognized, i.e., at the beginning of the column text, if the height of the first single-character detection box is less than the product of the average height of the single-character detection boxes and r_c, and the distance between it and the top edge of the crop_line_img image to be recognized is less than half of the average height, then the first single-character detection box is removed. Here, r_c is a preset multiplier, such as 0.8. For the single-character detection box near the bottom edge of the crop_line_img image to be recognized, i.e., at the end of the column text, if the height of the last single-character detection box is less than the average height * r_c, and the distance between it and the bottom edge of the crop_line_img image to be recognized is less than the average height / 2, then the last single-character detection box is removed.

[0166] (4) The text image to be identified is cropped according to the removed single character detection box, and the cropped text image to be identified is subjected to character recognition to obtain the recognized text including multiple characters.

[0167] Specifically, the CRNN model can be used to obtain the recognition text rec_text. When the character recognition device in this embodiment is deployed in the cloud, a large model can be used for recognition. When the character recognition device in this embodiment is deployed on the terminal, a lightweight model can be used for recognition.

[0168] (5) If the number of characters in the recognized text is equal to the number of single-character detection boxes, determine the target single-character detection box corresponding to the character positioning point, and determine the target character among the multiple characters according to the sequence number of the target single-character detection box in the multiple single-character detection boxes; if the number of characters in the recognized text is not equal to the number of single-character detection boxes, estimate the character position using the connected text classification (CTC) information of the recognition network, and output the target character corresponding to the character positioning point.

[0169] When the number of characters in the identified text equals the number of single-character detection boxes, the single-character detection boxes are sorted according to their y-axis coordinates. The distance between the center point of each single-character detection box and the character positioning point can be calculated. The single-character detection box whose center point is closest to the character positioning point is determined as the target single-character detection box. The target character among the multiple characters is determined based on the sequence number of the target single-character detection box among the multiple single-character detection boxes. For example, if the sequence number of the target single-character detection box is 5, then the 5th character among the multiple characters is determined as the target character; if the sequence number of the target single-character detection box is 3, then the 3rd character among the multiple characters is determined as the target character, and so on. This embodiment uses a scheme of matching single-character detection results with column text recognition results, which can improve the accuracy of fingertip character positioning and recognition.

[0170] (6) If the target character is a punctuation mark or a space, since punctuation marks or spaces are meaningless characters, the target character is replaced with the non-punctuation space character that is closest to the target character.

[0171] The technical solution of this embodiment can be applied to the scenario of finger-pointing reading of picture books, and is suitable for devices such as picture book robots and reading tablets. The camera of the character recognition device can take a picture of the area where the user's finger is located, analyze the image to identify the target character pointed to by the user's fingertip, and then read the target character pointed to by the fingertip aloud, or provide relevant information. For example, if the target character pointed to by the user's fingertip is Chinese, the meaning of the Chinese can be explained, or the Chinese can be combined into words; if the target character pointed to by the user's fingertip is English, the corresponding Chinese can be provided.

[0172] In this embodiment, after performing text detection on the screenshot of the area where the user's fingertip is located, a text box for the area is obtained. Then, single-character detection is performed on the text box for the area. Based on the single-character detection results, the target character pointed to by the user's fingertip is determined. This embodiment adopts a scheme that combines single-character detection and text recognition results, which can improve the accuracy of fingertip character positioning and recognition.

[0173] Embodiments of this disclosure also provide a character recognition device, as shown in FIG12, comprising:

[0174] Screenshot acquisition module 21 is used to acquire a screenshot of the area where the user's fingertip is located;

[0175] The region text box detection module 22 is used to perform text detection on the acquired screenshot and obtain region text boxes, which include row text boxes and / or column text boxes.

[0176] Matching module 23 is used to obtain a target area text box that matches the user's fingertip point;

[0177] The detection and recognition module 24 is used to perform single-character detection on the target area text box, and determine the target character pointed to by the user's fingertip based on the single-character detection result or based on the single-character detection result and character positioning information.

[0178] In this embodiment, after performing text detection on the screenshot of the area where the user's fingertip is located, a text box for the area is obtained. Then, single-character detection is performed on the text box for the area. Based on the single-character detection results, the target character pointed to by the user's fingertip is determined. This embodiment adopts a scheme that combines single-character detection and text recognition results, which can improve the accuracy of fingertip character positioning and recognition.

[0179] In some embodiments, the screenshot acquisition module 21 is specifically used to acquire an image of a user's finger; determine the fingertip and nodal point of the user's finger from the image; calculate the fingertip direction and fingertip angle based on the fingertip and nodal point of the user's finger, wherein the fingertip angle is the angle between the user's fingertip direction and the opposite direction of the x-axis of the image; and take a screenshot from the image based on the fingertip angle and the location of the fingertip.

[0180] In some embodiments, the relationship between the screenshot and the angle and location of the fingertip satisfies any one of the following:

[0181] When the angle of the fingertip is greater than or equal to 0° and less than 75°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is two-thirds of the screenshot size;

[0182] When the angle of the fingertip is greater than or equal to 75° and less than 100°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-half of the screenshot size;

[0183] When the angle of the fingertip is greater than or equal to 100° and less than 180°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-third of the screenshot size.

[0184] In some embodiments, the apparatus further includes:

[0185] A trapezoidal correction module is used to perform trapezoidal correction on the screenshot.

[0186] In some embodiments, the apparatus further includes:

[0187] The processing module performs the following steps when the number of detected area text boxes is 0:

[0188] Step a: After enlarging the screenshot size, repeat the screenshot acquisition step;

[0189] Step b: Determine if the number of detected text boxes in the region is 0. If yes, proceed to step c; otherwise, do not execute the screenshot acquisition step.

[0190] Step c: Determine whether the number of times the screenshot size has been enlarged has reached the preset number. If not, proceed to step a; if so, do not execute the screenshot acquisition step again.

[0191] In some embodiments, the region text box is a line text box, and the matching module is specifically used to determine a target line text box that matches the user's fingertip when the user's fingertip falls into the line text box, based on the vertical distance between the user's fingertip and the four edges of the line text box; and to determine a target line text box that matches the user's fingertip when the user's fingertip does not fall into the line text box, based on the line text box that intersects with the user's fingertip direction and the vertical distance between the user's fingertip and the lower edge of the line text box.

[0192] In some embodiments, the matching module is specifically used for:

[0193] When the text box is a horizontal text box, if the vertical distance between the user's fingertip and the four edges of the text box satisfies: db < (dt + db) / 3 and min(dr, dl) > box_h / 4, then the text box is determined to match the user's fingertip. Here, db is the vertical distance between the user's fingertip and the bottom edge of the text box, dt is the vertical distance between the user's fingertip and the top edge of the text box, dl is the vertical distance between the user's fingertip and the left edge of the text box, dr is the vertical distance between the user's fingertip and the right edge of the text box, and box_h is the height of the text box.

[0194] When the text box is a curved text box, if the vertical distance between the user's fingertip and the four edges of the text box satisfies: db1 < (dt1 + db1) * 2 / 3 and

[0195] If min(dr1,dl1)>box_h1 / 4 and abs(c_db)<(dt1+db1) / 3, then the text box is determined to match the user's fingertip. Here, c_db is the minimum distance from the user's fingertip to the text polygon outline of the curved text box, db1 is the vertical distance between the user's fingertip and the bottom edge of the text box, dt1 is the vertical distance between the user's fingertip and the top edge of the text box, dl1 is the vertical distance between the user's fingertip and the left edge of the text box, and dr1 is the vertical distance between the user's fingertip and the right edge of the text box. The curved text box satisfies the following condition: the ratio of the area of ​​the text box to the area of ​​the text polygon outline is greater than a preset ratio.

[0196] In some embodiments, the matching module is further configured to determine that the user's fingertip falls within the text box when the height of the text box is greater than a preset first threshold and the vertical distance between the user's fingertip and the lower edge of the text box is less than the product of the height of the text box and a preset first ratio.

[0197] In some embodiments, the matching module is specifically used to determine that the text box is a target text box that matches the user's fingertip if there is a text box that intersects with the user's fingertip direction and the vertical distance between the user's fingertip and the lower edge of the text box is less than a preset second threshold. The second threshold is determined by the average height of the text boxes in the screenshot.

[0198] In some embodiments, if there is no text box intersecting the direction of the user's fingertip, the device further includes:

[0199] The expansion module is used to enlarge the size of all text boxes in the screenshot.

[0200] The first judgment module is used to determine whether there is a line text box that intersects with the direction of the user's fingertip. If there is no line text box, the line text box above the user's fingertip is identified as the target line text box that matches the user's fingertip if the vertical distance between the lower edge of the line text box and the user's fingertip is less than the second threshold.

[0201] In some embodiments, the area text box is a line text box, and the device further includes:

[0202] The second judgment module is used to determine whether the preset map expansion conditions are met;

[0203] The image enlargement module is used to enlarge the screenshot size and re-execute the screenshot acquisition step when the image enlargement conditions are met.

[0204] The map expansion conditions include any one of the following:

[0205] When no target text box matching the user's fingertip is found, the ratio of the maximum height of all text boxes in the screenshot to the screenshot size is greater than a preset second ratio;

[0206] When a target line text box matching the user's fingertip is obtained, the ratio of the minimum vertical distance between the target line text box and the edge of the screenshot to the screenshot size is less than a preset third ratio, and the ratio of the height of the target line text box to the screenshot size is greater than a preset second ratio.

[0207] In some embodiments, the region text box is a line text box, and the detection and recognition module is specifically used to obtain a line text image to be recognized corresponding to the target line text box from the screenshot based on the position information of the target line text box; obtain character positioning points from the line text image to be recognized based on the user's fingertip direction; perform single-character detection on the line text image to be recognized to obtain multiple single-character detection boxes, and remove single-character detection boxes that meet preset requirements from the multiple single-character detection boxes. The preset requirements are: the width of the single-character detection box is less than the product of the average width of the single-character detection boxes and a preset multiple, and the minimum distance between the single-character detection box and the edge of the line text image to be recognized is less than a preset third threshold. The text image to be recognized is cropped based on the removed single-character detection boxes. Character recognition is then performed on the cropped text image to obtain recognized text containing multiple characters. If the number of characters in the recognized text is equal to the number of single-character detection boxes, a target single-character detection box corresponding to the character location point is determined. The target character among the multiple characters is determined based on the sequence number of the target single-character detection box in the multiple single-character detection boxes. If the number of characters in the recognized text is not equal to the number of single-character detection boxes, CTC information in the Convolutional Recurrent Neural Network (CRNN) is used for character position estimation, and the target character corresponding to the character location point is output.

[0208] In some embodiments, the detection and recognition module is specifically used to determine the intersection point of a first straight line and the fingertip direction when the target line text box is a horizontal text box, and when the acute angle formed by the fingertip direction and the lower edge of the target line text box is greater than 45°, wherein the first straight line intersects the target line text box, is parallel to the lower edge, and the distance between the first straight line and the lower edge is one-third of the height of the target line text box; and to determine the intersection point of the lower edge and the fingertip direction when the acute angle formed by the fingertip direction and the lower edge of the target line text box is less than or equal to 45°.

[0209] When the target text box is a curved text box, the fingertip is determined as the intersection point when it is located within the text polygon outline of the target text box; when the fingertip is located within the target text box but outside the text polygon outline, the intersection point of the second line and the fingertip direction is determined, wherein the second line is parallel to the lower edge of the target text box and the distance between the second line and the lower edge and the upper edge of the target text box is half the height of the target text box; when the fingertip is located outside the target text box, the intersection point of the second line and the fingertip direction is determined.

[0210] The coordinates of the intersection point are mapped to the text image to be identified to obtain the character positioning point.

[0211] In some embodiments, the detection and recognition module is further configured to add a virtual single-character detection box between two adjacent single-character detection boxes when the distance between two adjacent single-character detection boxes is greater than a preset fourth threshold, wherein the fourth threshold is determined by the average width of the single-character detection boxes.

[0212] In some embodiments, the detection and recognition module is further configured to, when the target line text box is a curved text box, retain the S single-character detection boxes closest to the character positioning point if a preset condition is met, and shrink the line text image to be recognized according to the retained S single-character detection boxes, where S is a positive integer;

[0213] The preset conditions include:

[0214] The ratio of the aspect ratio of the text image to be identified to the number of single-character detection boxes is between 0.9 and 1.8, and the number of single-character detection boxes is greater than S.

[0215] In some embodiments, the detection and recognition module is further configured to convert the character positioning point into an index cross_idx, the index cross_idx being determined by the x-axis coordinate C_X of the character positioning point in the screenshot and the number of downsampled samples stride of the recognition network for the width of the input image;

[0216] The index is decoded. If it is decoded as a placeholder "blank", the difference value dist = C_X - cross_idx * stride is calculated. If dist is less than 0, the nearest non-blank character in the recognized text is searched, starting from the placeholder "blank" and following the order of searching one character to the left and then one character to the right, to obtain the target character. If dist is greater than or equal to 0, the nearest non-blank character in the recognized text is searched, following the order of searching one character to the right and then one character to the left, to obtain the target character.

[0217] In some embodiments, the detection and recognition module is further configured to replace the target character with the non-punctuation space character that is closest to the target character if the target character is a punctuation mark or a space.

[0218] In some embodiments, the detection and recognition module is further configured to, if the target character is English, perform a forward and backward search on the recognized text starting from the target character until a punctuation mark or space is encountered, obtain candidate English strings, perform word segmentation on the candidate English strings, and replace the target character with the word segmentation result.

[0219] This disclosure also provides a character recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the character recognition method as described above.

[0220] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the character recognition method described above.

[0221] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage, or any other non-transferable medium that can be used to store information accessible to the computer-readable terminal device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0222] In the various method embodiments of this disclosure, the sequence numbers of each step are not intended to limit the order of the steps. For those skilled in the art, any changes in the order of the steps are within the scope of protection of this disclosure without any creative effort.

[0223] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, since the embodiments are basically similar to the product embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the product embodiments.

[0224] The above description represents the preferred embodiments of this disclosure. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles described herein, and these improvements and modifications should also be considered within the scope of protection of this disclosure.

Claims

1. A character recognition method, characterized in that, include: The steps to capture a screenshot are as follows: capture a screenshot of the area where the user's fingertip is located; The region text box detection step involves performing text detection on the acquired screenshot to obtain region text boxes, which include row text boxes and / or column text boxes. The matching step involves obtaining a text box representing the target area that matches the user's fingertip. The detection and recognition step involves performing single-character detection on the target area text box, and determining the target character pointed to by the user's fingertip based on the single-character detection result or based on the single-character detection result and character positioning information.

2. The character recognition method according to claim 1, characterized in that, The screenshot acquisition steps include: Get an image of the user's finger; Determine the fingertip and nodal point of the user's finger from the image; The fingertip direction and fingertip angle are calculated based on the fingertip and finger joint of the user's finger. The fingertip angle is the angle between the user's fingertip direction and the opposite direction of the x-axis of the image. A screenshot is taken from the image based on the angle and location of the fingertip.

3. The character recognition method according to claim 2, characterized in that, The relationship between the screenshot and the angle and location of the fingertip satisfies any one of the following: When the angle of the fingertip is greater than or equal to 0° and less than 75°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is two-thirds of the screenshot size; When the angle of the fingertip is greater than or equal to 75° and less than 100°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-half of the screenshot size; When the angle of the fingertip is greater than or equal to 100° and less than 180°, the fingertip is located within the screenshot, and the vertical distance between the fingertip and the bottom edge of the screenshot is one-quarter of the screenshot size, and the vertical distance between the fingertip and the left edge of the screenshot is one-third of the screenshot size.

4. The character recognition method according to claim 1, characterized in that, After the screenshot acquisition step, the method further includes: The trapezoidal correction step involves performing trapezoidal correction on the screenshot.

5. The character recognition method according to claim 1, characterized in that, The method further includes: When the number of detected area text boxes is 0, perform the following steps: Step a: After enlarging the screenshot size, repeat the screenshot acquisition step; Step b: Determine if the number of detected text boxes in the region is 0. If yes, proceed to step c; otherwise, do not execute the screenshot acquisition step. Step c: Determine whether the number of times the screenshot size has been enlarged has reached the preset number. If not, proceed to step a; if so, do not execute the screenshot acquisition step again.

6. The character recognition method according to claim 1, characterized in that, The area text box is a line text box, and the matching step includes: When the user's fingertip falls into the text box, a target text box matching the user's fingertip is determined based on the vertical distance between the user's fingertip and the four edges of the text box. When the user's fingertip does not fall within the text box, a target text box matching the user's fingertip is determined based on the text box intersecting the user's fingertip direction and the vertical distance between the user's fingertip and the bottom edge of the text box.

7. The character recognition method according to claim 6, characterized in that, The step of determining the target text box that matches the user's fingertip based on the vertical distance between the user's fingertip and the four edges of the text box includes: When the text box is a horizontal text box, if the vertical distance between the user's fingertip and the four edges of the text box satisfies: db < (dt + db) / 3 and min(dr, dl) > box_h / 4, then the text box is determined to match the user's fingertip. Here, db is the vertical distance between the user's fingertip and the bottom edge of the text box, dt is the vertical distance between the user's fingertip and the top edge of the text box, dl is the vertical distance between the user's fingertip and the left edge of the text box, dr is the vertical distance between the user's fingertip and the right edge of the text box, and box_h is the height of the text box. When the text box is a curved text box, if the user's fingertip is aligned with the four sides of the text box... The perpendicular distance between the edges satisfies: db1 < (dt1 + db1) * 2 / 3 and If min(dr1,dl1)>box_h1 / 4 and abs(c_db)<(dt1+db1) / 3, then the text box is determined to match the user's fingertip. Here, c_db is the minimum distance from the user's fingertip to the text polygon outline of the curved text box, db1 is the vertical distance between the user's fingertip and the bottom edge of the text box, dt1 is the vertical distance between the user's fingertip and the top edge of the text box, dl1 is the vertical distance between the user's fingertip and the left edge of the text box, and dr1 is the vertical distance between the user's fingertip and the right edge of the text box. The curved text box satisfies the following condition: the ratio of the area of ​​the text box to the area of ​​the text polygon outline is greater than a preset ratio.

8. The character recognition method according to claim 6, characterized in that, The method further includes: When the height of the text box is greater than a preset first threshold, and the vertical distance between the user's fingertip and the bottom edge of the text box is less than the product of the height of the text box and a preset first ratio, it is determined that the user's fingertip falls into the text box.

9. The character recognition method according to claim 6, characterized in that, The step of determining the target text box matching the user's fingertip based on the text box intersecting the user's fingertip direction and the vertical distance between the user's fingertip and the bottom edge of the text box includes: If there exists a text box that intersects with the user's fingertip direction, and the vertical distance between the user's fingertip and the lower edge of the text box is less than a preset second threshold, the text box is determined to be a target text box that matches the user's fingertip. The second threshold is determined by the average height of the text boxes in the screenshot.

10. The character recognition method according to claim 9, characterized in that, If there is no text box that intersects with the direction of the user's fingertip, the method further includes: Increase the size of all text boxes in the screenshot; Determine whether there is a text box that intersects with the user's fingertip direction. If not, identify the text box above the user's fingertip whose vertical distance between its lower edge and the user's fingertip is less than the second threshold as the target text box that matches the user's fingertip.

11. The character recognition method according to claim 1, characterized in that, The area text box is a line text box. After the matching step, the method further includes: The judgment step is to determine whether the preset map expansion conditions are met; In the image enlargement step, if the image enlargement conditions are met, the screenshot size is enlarged, and the screenshot acquisition step is executed again. The map expansion conditions include any one of the following: When no target text box matching the user's fingertip is found, the ratio of the maximum height of all text boxes in the screenshot to the screenshot size is greater than a preset second ratio. When a target line text box matching the user's fingertip is obtained, the ratio of the minimum vertical distance between the target line text box and the edge of the screenshot to the screenshot size is less than a preset third ratio, and the ratio of the height of the target line text box to the screenshot size is greater than a preset second ratio.

12. The character recognition method according to claim 1, characterized in that, The region text box is a line text box, and the detection and recognition steps include: Based on the position information of the target line text box, obtain the image of the line text to be recognized corresponding to the target line text box from the screenshot; The character positioning points are obtained from the text image to be recognized based on the direction of the user's fingertip. The text image to be identified is subjected to single-character detection to obtain multiple single-character detection boxes. Single-character detection boxes that meet preset requirements are removed from the multiple single-character detection boxes. The preset requirements are: the width of the single-character detection box is less than the product of the average width of the single-character detection box and a preset multiple, and the minimum distance between the single-character detection box and the edge of the text image to be identified is less than a preset third threshold. The text image to be recognized is cropped based on the removed single-character detection boxes, and the cropped text image to be recognized is used for character recognition to obtain recognized text including multiple characters. If the number of characters in the recognized text is equal to the number of single-character detection boxes, determine the target single-character detection box corresponding to the character positioning point, and determine the target character among the multiple characters according to the sequence number of the target single-character detection box in the multiple single-character detection boxes; If the number of characters in the identified text is not equal to the number of single-character detection boxes, the connected text classification (CTC) information in the convolutional recurrent neural network (CRNN) is used to estimate the character position and output the target character corresponding to the character positioning point.

13. The character recognition method according to claim 12, characterized in that, The step of obtaining character positioning points from the text image to be recognized based on the user's fingertip direction includes: When the target text box is a horizontal text box, if the acute angle between the fingertip direction and the lower edge of the target text box is greater than 45°, the intersection point of the first straight line and the fingertip direction is determined. The first straight line intersects the target text box, is parallel to the lower edge, and the distance between the first straight line and the lower edge is one-third of the height of the target text box. If the acute angle between the fingertip direction and the lower edge of the target text box is less than or equal to 45°, the intersection point of the lower edge and the fingertip direction is determined. When the target text box is a curved text box, the fingertip is determined as the intersection point when it is located within the text polygon outline of the target text box; when the fingertip is located within the target text box but outside the text polygon outline, the intersection point of the second line and the fingertip direction is determined, wherein the second line is parallel to the lower edge of the target text box and the distance between the second line and the lower edge and the upper edge of the target text box is half the height of the target text box; when the fingertip is located outside the target text box, the intersection point of the second line and the fingertip direction is determined. The coordinates of the intersection point are mapped to the text image to be identified to obtain the character positioning point.

14. The character recognition method according to claim 12, characterized in that, After obtaining multiple single-character detection boxes, the method further includes: When the distance between two adjacent single-character detection boxes is greater than a preset fourth threshold, a virtual single-character detection box is added between the two adjacent single-character detection boxes. The fourth threshold is determined by the average width of the single-character detection boxes.

15. The character recognition method according to claim 12, characterized in that, After obtaining multiple single-character detection boxes, the method further includes: When the target line text box is a curved text box, if the preset conditions are met, the S single-character detection boxes closest to the character positioning point are retained, and the line text image to be identified is shrunk according to the retained S single-character detection boxes, where S is a positive integer; The preset conditions include: The ratio of the aspect ratio of the text image to be identified to the number of single-character detection boxes is between 0.9 and 1.8, and the number of single-character detection boxes is greater than S.

16. The character recognition method according to claim 12, characterized in that, The method further includes: If the target character is a punctuation mark or a space, replace the target character with the character that is the nearest non-punctuation mark or space to the target character.

17. The character recognition method according to claim 12, characterized in that, The method further includes: If the target character is English, the recognized text is searched forward and backward from the target character until a punctuation mark or space is encountered, resulting in candidate English strings. The candidate English strings are then segmented, and the target character is replaced with the segmented result.

18. A character recognition device, characterized in that, include: The screenshot acquisition module is used to capture screenshots of the area where the user's fingertip is located; The line text box detection module is used to perform text detection on the acquired screenshot and obtain line text boxes; The matching module is used to obtain the target line text box that matches the user's fingertip point; The detection and recognition module is used to perform single-character detection on the target line text box, and determine the target character pointed to by the user's fingertip based on the single-character detection result or based on the single-character detection result and character positioning information.

19. A character recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the character recognition method as described in any one of claims 1-17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the character recognition method as described in any one of claims 1-17.

Images