Image word segmentation recognition technology based on self-adaptive character box generation and adjustment

Abstract

The invention discloses an image word segmentation recognition technology based on self-adaptive character box generation and adjustment, belongs to the technical field of image processing and analysis, and provides an image word segmentation recognition technology based on self-adaptive character box generation and adjustment through a series of image processing and bounding box reduction operations. According to the method, dynamic separation and optimization of the character level bounding box in complex OCR output are achieved, effective adjustment of character boundaries is achieved through union search, topological sorting and R-Tree indexing technologies, the overlapping problem of characters is avoided, and therefore the recognition accuracy of an OCR system in a high-density or overlapping scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and analysis, and more specifically to an interactive image word marking and positioning method based on an adaptive dynamic frame. Background Art

[0002] The image wording function refers to the use of OCR (optical character recognition) technology to recognize the text in the image and highlight the text on the screen, while providing translation, explanation and other functions. This function is usually provided by some translation software, such as NetEase Youdao Dictionary.

[0003] At present, the existing image wording mainly uses OCR technology to focus on improving the accuracy of text recognition. However, when processing complex object backgrounds, since the OCR box detection algorithm is usually based on image features and edge detection, when the paper surface is irregular (such as wrinkles, bends), these boxes often overlap, which may cause the recognition result to be inconsistent with the actual text area. Especially when the user needs to manually select a certain area of ​​text, the precise area may not be selected due to coverage problems, or even deviate from the area expected by the user. Summary of the invention

[0004] 1. Technical problems to be solved:

[0005] In view of the problems existing in the prior art, the purpose of the present invention is to provide an image word recognition technology based on adaptive text box generation and adjustment, which realizes the dynamic separation and optimization of character-level bounding boxes in complex OCR output through a series of graph processing and bounding box reduction operations, and realizes effective adjustment of character boundaries through union-find, topological sorting and R-Tree indexing technology, avoiding the overlap problem between characters, thereby improving the recognition accuracy of the OCR system in high-density or overlapping scenarios.

[0006] 2. Technical solution:

[0007] To solve the above problems, the present invention adopts the following technical solutions.

[0008] An image word recognition technology based on adaptive text box generation and adjustment includes the following steps:

[0009] S1, input and preprocess the word-level bounding boxes in the image, and split the word-level bounding boxes into character-level bounding boxes;

[0010] S2, preliminarily grouping the extracted character-level bounding boxes and optimizing the efficiency of image processing;

[0011] S3, topologically sorting and reducing the bounding boxes to eliminate overlapping states of character bounding boxes;

[0012] S4, performing overlap detection on the bounding box and adjusting the algorithm to generate a new bounding box without overlap;

[0013] S5. Dynamically update the bounding box bbox of each character level and the spatial index to make them non-overlapping.

[0014] A further improvement is that the method for inputting and preprocessing the word-level bounding box in step S1 includes:

[0015] S11, input: word-level bounding box bbox and its corresponding text obtained from the OCR result;

[0016] S12. Objective: Subdivide word-level bounding boxes into character-level bounding boxes and eliminate overlaps, ultimately obtaining clear, non-overlapping character-level bounding boxes.

[0017] A further improvement is that the method for grouping and optimizing the bounding boxes in step S2 includes:

[0018] S21. Use union-find and STF grouping to preliminarily group character-level bounding boxes to optimize the efficiency of image processing. The time complexity is O(n\logn);

[0019] S22. Based on the bounding box and text information, the word-level bounding box is divided into character-level bounding boxes, and the average change rate is obtained through vector calculation.

[0020] A further improvement is that the topological sorting and bounding box reduction method in step S3 includes:

[0021] S31. Use the bidirectional edge structure (built by the build_bidirectional_edges function) to form an undirected graph and process the nodes with in-degree 1.

[0022] S32, the topological sorting operation sequentially processes the nodes with in-degree 1 and checks whether the adjacent nodes overlap. If there is overlap, call the shrink_bbox_to_avoid_overlap function to dynamically adjust the bounding box and gradually eliminate the overlap;

[0023] S33, checking whether the nodes overlap by using the check_overlap function;

[0024] S34, reducing the overlapping boundary boxes according to the overlapping conditions, so that the character boundary boxes reach a non-overlapping state.

[0025] A further improvement is that the overlap detection method in step S4 includes:

[0026] S41, coordinate separation: extract the coordinates of the four vertices of each bounding box;

[0027] S42, overlap detection: calculating the overlap range of the two bounding boxes according to the axial direction, and reducing the target according to the center position and the overlap distance;

[0028] S43, shrinking the bounding box: adjusting the vertices of the character's bounding box in the reverse axis direction to keep it away from the overlapping area, and shrinking the character in the left-right or up-down direction according to the center position, thereby finally generating a new bounding box without overlap.

[0029] A further improvement is that the dynamic update and spatial indexing method in step S5 includes:

[0030] S51, dynamically update the bounding box bbox of each character level to make them non-overlapping;

[0031] S52. Implement query operation based on spatial index R-Tree, search for corresponding character-level bounding box according to input coordinates (x, y), and return -1 if not found.

[0032] 3. Beneficial effects:

[0033] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0034] (1) Accurate character-level separation and positioning

[0035] Traditional OCR methods often have recognition errors in complex backgrounds, uneven character spacing or overlapping situations. This technical solution refines the bounding box at the word level to the character level and combines topological sorting and union-find algorithms to effectively distinguish the position of each character, significantly improving the separation and accuracy of the character level.

[0036] (2) Dynamically eliminate overlap to improve recognition reliability

[0037] The present invention provides a real-time dynamic adjustment mechanism for the overlap problem in character-level bounding box detection. The overlap detection algorithm is used to automatically reduce the bounding box to avoid interference between characters without manual intervention, which improves the applicability of images in OCR and is particularly suitable for text extraction in complex text scenes (such as wrinkled paper and pictures with complex backgrounds).

[0038] (3) Efficient image processing speed

[0039] The present invention improves processing efficiency through union-find and spatial index (R-Tree), optimizes the time complexity of overlap detection to O(n\log n), greatly reduces the amount of computation for image processing and bounding box adjustment, and is suitable for fast OCR parsing of large-scale document images.

[0040] (4) Generalization capability to adapt to various scenarios

[0041] The present invention designs strategies such as topological sorting and reverse axial reduction. The scheme is applicable to scenarios with various character arrangements and different spacings, avoids overlap between characters, and enables the scheme to maintain consistent recognition performance on different types of documents, scans, and photos.

[0042] (5) Efficient search based on spatial index

[0043] The present invention adds R-Tree spatial indexing to quickly query character-level bounding boxes, effectively improving the speed of finding characters at specific locations and making character positioning more efficient. This spatial indexing method is particularly beneficial for real-time OCR applications and helps to improve the response speed of the system.

[0044] In summary, this technical solution combines character separation, overlap elimination, bounding box adjustment and efficient index search to ensure the recognition accuracy and efficiency of OCR in complex image scenes. It can significantly improve the performance in pictures or documents with changing backgrounds and crowded characters, and is an important supplement and enhancement to existing OCR processing technology.

[0045] It should be noted that the structures not introduced in the present invention are the same as the prior art or can be implemented by using the prior art, and are not described in detail here because they do not involve the design points and improvement directions of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is the result of combining traditional image word segmentation with OCR;

[0047] Figure 2 For the general Figure 1 The effect diagram of grouping OCR recognition results;

[0048] Figure 3 For the general Figure 1 The effect diagram of character cutting based on the bbox;

[0049] Figure 4 Propose repeated word-level bboxes for union-find pruning;

[0050] Figure 5 for Figure 4 Character level bbox;

[0051] Figure 6 Repeat Algorithm 2 again to perform character level grouping.

[0052] Figure 7 The left picture is the effect picture before optimization, and the right picture is the effect picture after optimization;

[0053] Figure 8 For the final dynamic adjustment effect diagram;

[0054] Fig. 9 It is a system flow chart of the whole invention;

[0055] Fig.10 is the recognition data of the OCR of the present invention, i.e., the input data of the algorithm;

[0056] Fig.11 The output text data of the algorithm part of the present invention is removed from the overlapping bboxes and character-level bboxes recognized by OCR. DETAILED DESCRIPTION

[0057] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0058] Example:

[0059] See also Figure 1-11 , an image word recognition technology based on adaptive text box generation and adjustment, wherein; the bbox at the word level of the image and the character corresponding to the bbox are obtained according to OCR, such as Figure 1 As shown; the time complexity of the union-find and STF grouping process is O(nlogn), and the grouping is implemented as follows Figure 2 As shown in the figure, according to the bbox and its text information, the word-level bbox is segmented into character level.

[0060] In this embodiment:

[0061] 1. Bounding box input format

[0062] The input bounding box cur_bbox_numpy is an array containing the coordinates of eight vertices, representing the four vertices of the quadrilateral, given in the form [x1, y1, x2, y2, x3, y3, x4, y4].

[0063] 2. Definition of start and end points

[0064] Define the starting point P start and end point P end , respectively:

[0065] P start =[x1,y1],P end =[x2,y2]

[0066] These two points are usually the coordinates of the upper left corner and the upper right corner, which are used to determine the width direction of the character box.

[0067] 3. Calculation of character width vector

[0068] Character width vector V width Used to indicate the horizontal length of a character, defined as follows:

[0069]

[0070] Among them, num charsh Indicates the number of characters, so V width is the horizontal width of each character's bounding box.

[0071] 4. Calculation of character height vector

[0072] The character's height vector V height Represents the length of the character bounding box in the vertical direction, defined as:

[0073] v height =[x3,y3]-P start

[0074] in, Usually the coordinate point of the lower right corner of the bounding box, used to represent the vertical component of the character box.

[0075] 5. Generate bounding boxes for each character

[0076] The bounding box of each character is composed of four vertex coordinates, which are iteratively generated according to the following formula:

[0077] For each character i (i=0,1,...,num_chars-1), the character bounding box vertices are defined as:

[0078] Bi,1=P start +i·v width

[0079] Bi,2=P start +(i+1)·v width

[0080] Bi,3=P start +(i+1)·v width +v height

[0081] Bi,4=P start +i·v width +v height

[0082] The resulting character-level bounding box is a list of num_chars character boxes, each of which is composed of the four vertices mentioned above. Figure 3 Shown

[0083] 4. Topological sorting and bounding box reduction

[0084] Starting from the edge number 1, perform a topological sorting operation. The topological sorting operation is as follows:

[0085] #Input: edges represents the connection relationship of the undirected graph, and edge nodes represent nodes with an in-degree of 1.

[0086] #The format of edges is a dictionary, for example, {0:[1, 2, 3], 1:[0, 4], ...} means that 0 is connected to 1, 2, and 3.

[0087]

[0088]

[0089] 1. Build bidirectional edges: Use the function build_bidirectional_edges to convert the input undirected edge dictionary edges into a bidirectional edge structure bidir_edges, that is, each node and its neighbors are bidirectionally connected for subsequent processing.

[0090] 2. Initialize edge nodes: Store the nodes with in-degree 1 (edge ​​nodes) in the graph into the single_degree_nodes list. These nodes will serve as the starting point of topological sorting and gradually process the boundaries with in-degree 1.

[0091] 3. Topological sorting operation: In the main loop, process the nodes in single_degree_nodes one by one.

[0092] Each time, pop a node with an in-degree of 1 from single_degree_nodes and find its only connected neighbor neighbor.

[0093] Delete the edge in both directions to update the connectivity structure of the graph.

[0094] 4. Check overlap and adjust: After deleting the edge, use the check_overlap function to check whether the node and neighbor overlap. If so, call the shrink_bbox_to_avoid_overlap function to shrink the bounding box to eliminate the overlap.

[0095] 5. Update nodes with in-degree 1: If the in-degree of neighbor becomes 1, add it to single_degree_nodes so that it can be processed in subsequent iterations.

[0096] 6. Clean up processed nodes: If the connection set of node or neighbor is empty, that is, there are no neighbor nodes, delete it from bidir_edges to ensure the simplification of the graph structure.

[0097] 7. End of the loop: Repeat the above process until single_degree_nodes is empty, that is, there are no more nodes with in-degree 1 in the graph, or all nodes in bidir_edges have been processed.

[0098] Embodiment 1:

[0099] In this embodiment, each operation specifically updates the anti-axis shrinking direction of the bbox at the word level to which the edge belongs, and the calculation method of the adjustment algorithm is as follows:

[0100] 1. Coordinate separation

[0101] The two input bounding boxes bbox1 and bbox2 each contain four vertices in the format [x1, y1, x2, y 2, x3, y3, x4, y4]. First, separate the x and y coordinates into two lists of vertex coordinates:

[0102] x1_coords=[x1, x2, x3, x4], y1_coords=[y1, y2, y3, y4]

[0103] x2_coords=[x1,x2,x3,x4],y2_coords=[y1,y2,y3,y4]

[0104] 2. Determine the coordinate axis for overlap detection

[0105] According to the input parameter axis, select the coordinate axis for detecting overlap. If axis=1 indicates the y-axis, the y coordinates (y1_coords and y2_coords) are processed; otherwise, the x coordinates (x1_coords and x2_coords) are processed.

[0106] 3. Calculate the center position

[0107] Calculate the center position center1 and center2 of each bounding box respectively, and the calculation formula is the average value of the vertex coordinates:

[0108]

[0109] where coords1 and coords2 are the vertex coordinates of bbox1 and bbox2 on the selected coordinate axis respectively.

[0110] 4. Overlap Range Calculation

[0111] Calculate the overlap range of the two bounding boxes on the selected axis, and take the intersection of the minimum and maximum values of the two:

[0112] overlap_min = max(min(coords1), min(coords2))

[0113] overlap_max = min(max(coords1), max(coords2))

[0114] The target overlap distance target_distance is half of the length of this intersection:

[0115]

[0116] 5. Shrink the Vertex Positions of the Bounding Boxes

[0117] According to the overlap position and the center position, shrink the overlapping bounding box vertices inward by the amount of target_distance. The bounding box with the smaller center will be shrunk downward or to the left, and the bounding box with the larger center will be shrunk upward or to the right:

[0118] · When center1 < center2, execute:

[0119] new_coords1 = coords1 - target_distance,

[0120] new_coords2 = coords2 + target_distance

[0121] · Otherwise, execute:

[0122] new_coords1 = coords1 + target_distance,

[0123] new_coords2 = coords2 - target_distance

[0124] The adjustment algorithm result is as Figure 7 shown.

[0125] 6. Merge the New Vertices

[0126] Merge the new coordinates back into the vertex order of the bounding boxes according to the axis, and return the adjusted two bounding boxes new_bbox1 and new_bbox2.

[0127] Finally, the final result is obtained by combining overlap and all token_bboxes without overlap. Figure 8 .

[0128] Embodiment 2:

[0129] The detailed algorithm process of class topological sorting and bounding box reduction is as follows:

[0130] (1) Construction of bidirectional edges in undirected graphs

[0131] E={v i :[v i1 , v 12 ,...,v ik ]}

[0132] Among them, node V i and the neighbor node set [V i1 , V i2 ,.....V ik ,] connected,

[0133] Build a function by building a bidirectional edge b idirection al e dges, unify the direction of each edge to undirected, and obtain a bidirectional edge set:

[0134] If °v i →v j ° exists, then °v j →v i ° must also exist.

[0135] (2) Initialization of the node set with in-degree 1

[0136] The degree of a node is defined as deg(v), which is the number of neighbors connected to the node.

[0137] Initially, the set of all nodes with degree 1 is recorded as:

[0138] S = {v|deg(v) = 1}

[0139] (3) Quasi-topological sorting process

[0140] Starting from the edge number 1, perform a topological sorting operation. The topological sorting operation is as follows:

[0141] a. Take node v from set S and remove the bidirectional edges between the node and its neighbors.

[0142] b. Update the degree of the neighbor node. If its degree becomes 1, add the node to the set S.

[0143] c. When removing edges, determine whether the bounding boxes between nodes overlap. If so, reduce the bounding boxes.

[0144] (4) Bounding box overlap detection and reduction

[0145] In the process of topological sorting, for node v and its only neighbor node u, it is necessary to detect whether the bounding boxes overlap: the node bounding box can be expressed as:

[0146] Rectangle R V and R U , whose shape is [X min , Y min , X max , Y max ].

[0147] The judgment condition is whether the projections of two rectangles on a certain axis intersect:

[0148]

[0149] If the bounding boxes overlap, the shrinking function shrink_bbox is called to adjust the bounding boxes:

[0150] R v ′=srink(R v ,R u ),

[0151] R u ′=srink(R u , R v )

[0152] The specific implementation of the squeeze function is as follows:

[0153] a. Coordinate separation

[0154] The two input bounding boxes bbox1 and bbox2 each contain four vertices in the format [x1, y1, x2, y 2, x3, y3, x4, y4]. First, separate the x and y coordinates into two lists of vertex coordinates:

[0155] x1_coords=[x1,x2,x3,x4],y1_coords=[y1,y2,y3,y4]

[0156] x2_coords=[x1,x2,x3,x4],y2_coords=[y1,y2,y3,y4]

[0157] b. Determine the coordinate axis for overlap detection

[0158] Select the coordinate axis for detecting overlap according to the input parameter axis. If axis = 1 represents the y-axis, then process the y-coordinates (y1_coords and y2_coords); otherwise, process the x-coordinates (x1_coords and x2_coords).

[0159] c. Calculate the center position

[0160] Calculate the center positions center1 and center2 of each bounding box respectively. The calculation formula is the average of the vertex coordinates:

[0161]

[0162] where coords1 and coords2 are the vertex coordinates of bbox1 and bbox2 on the selected coordinate axis respectively.

[0163] d. Calculate the overlap range

[0164] Calculate the overlap range of the two bounding boxes on the selected axis, and take the intersection of the minimum and maximum values of the two:

[0165] overlap_min = max(min(coords1), min(coords2))

[0166] overlap_max = min(max(coords1), max(coords2))

[0167] The target overlap distance target_distance is half of the length of this intersection:

[0168]

[0169] e. Shrink the vertex positions of the bounding box

[0170] According to the overlap position and the center position, shrink the overlapping bounding box vertices inward by the amount of target_distance. The bounding box with the smaller center will shrink downward or to the left, and the bounding box with the larger center will shrink upward or to the right:

[0171] · When center1 < center2, execute:

[0172] new_coords1 = coords1 - target_distance,

[0173] new_coords2 = coords2 + target_distance

[0174] Otherwise, execute:

[0175] new_coords1=coords1+target_distance,

[0176] new_coords2=coords2-target_distance

[0177] Adjust the algorithm results as follows Figure 7 shown.

[0178] f. Merge new vertices

[0179] Merge the new coordinates back into the vertex order of the bounding box according to the axis, and return the adjusted two bounding boxes new_bbox1 and new_bbox2.

[0180] (5) Algorithm termination condition

[0181] The quasi-topological sort terminates when the node set S is empty, the final bidirectional edge set B is empty, or the degree of all remaining nodes is greater than 1.

[0182] The entire algorithm pseudo code is as follows:

[0183] Input: Initial undirected graph E = {V1: {V i1 , V i2 ,.....V ik}}.

[0184] Output: The adjusted bounding box set {R v}.

[0185] Algorithm steps:

[0186] 1. Construct bidirectional edges:

[0187] B=build\_bidirectional\_edges(E)

[0188] 2. Initialize the node set:

[0189] S = {v|deg(v) = 1}

[0190] 3. When S≠φ, repeat the following steps:

[0191] ①. Take node V from S and find its only neighbor U;

[0192] ②, delete the edge V→U and update deg(u);

[0193] ③、If Overlap(R V ,R U )=True, then the reduction function is called:

[0194] R v ,R u =srink(R v ,R u )

[0195] ④. If deg(u)=1, add u to s.

[0196] Finally, the final result is obtained by combining overlap and all token_bboxes without overlap. Figure 8

[0197] The above-described embodiments only express a certain implementation mode of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. An image word recognition technology based on adaptive text box generation and adjustment, characterized in that: The following steps are involved: S1, input and preprocess the word-level bounding boxes in the image, and split the word-level bounding boxes into character-level bounding boxes; S2, preliminarily grouping the extracted character-level bounding boxes and optimizing the efficiency of image processing; S3, topologically sorting and reducing the bounding boxes to eliminate overlapping states of character bounding boxes; S4, performing overlap detection on the bounding box and adjusting the algorithm to generate a new bounding box without overlap; S5. Dynamically update the bounding box bbox of each character level and the spatial index to make them non-overlapping.

2. The image word recognition technology based on adaptive text box generation and adjustment according to claim 1, characterized in that: The method for inputting and preprocessing the word-level bounding box in step S1 includes: S11, input: word-level bounding box bbox and its corresponding text obtained from the OCR result; S12. Objective: Subdivide word-level bounding boxes into character-level bounding boxes and eliminate overlaps, ultimately obtaining clear, non-overlapping character-level bounding boxes.

3. The image word recognition technology based on adaptive text box generation and adjustment according to claim 1, characterized in that: The method for grouping and optimizing the bounding boxes in step S2 includes: S21. Use union-find and STF grouping to preliminarily group character-level bounding boxes to optimize the efficiency of image processing. The time complexity is O(n\logn); S22. Based on the bounding box and text information, the word-level bounding box is divided into character-level bounding boxes, and the average change rate is obtained through vector calculation.

4. The image word recognition technology based on adaptive text box generation and adjustment according to claim 1, characterized in that: The topological sorting and bounding box reduction method in step S3 includes: S31. Use the bidirectional edge structure (built by the build_bidirectional_edges function) to form an undirected graph and process the nodes with in-degree 1. S32, the topological sorting operation sequentially processes the nodes with in-degree 1 and checks whether the adjacent nodes overlap. If there is overlap, call the shrink_bbox_to_avoid_overlap function to dynamically adjust the bounding box and gradually eliminate the overlap; S33, checking whether the nodes overlap by using the check_overlap function; S34, reducing the overlapping boundary boxes according to the overlapping conditions, so that the character boundary boxes reach a non-overlapping state.

5. The image word recognition technology based on adaptive text box generation and adjustment according to claim 1, characterized in that: The overlapping detection method in step S4 includes: S41, coordinate separation: extract the coordinates of the four vertices of each bounding box; S42, overlap detection: calculating the overlap range of the two bounding boxes according to the axial direction, and reducing the target according to the center position and the overlap distance; S43, shrinking the bounding box: adjusting the vertices of the character's bounding box in the reverse axis direction to keep it away from the overlapping area, and shrinking the character in the left-right or up-down direction according to the center position, thereby finally generating a new bounding box without overlap.

6. The image word recognition technology based on adaptive text box generation and adjustment according to claim 1, characterized in that: The dynamic update and spatial indexing method in step S5 includes: S51, dynamically update the bounding box bbox of each character level to make them non-overlapping; S52. Implement query operation based on spatial index R-Tree, search for corresponding character-level bounding box according to input coordinates (x, y), and return -1 if not found.

Images