A method, system and computing storage medium for extracting text content from images

Abstract

The present invention provides a method, system, and computing storage medium for extracting text content from an image, comprising the following steps: S1, obtaining the minimum bounding rectangle of each image region on an image; S2, intercepting each image region by perspective transformation; S3, calculating a third rotation angle of each image region; performing non-cropping rotation and straightening on each intercepted image region according to the third rotation angle; S4, performing a first splicing on the text content of each text box in each image region; S5, calculating a standard rotation angle of the image, and performing non-cropping rotation and straightening on the image; S6, performing a second splicing on the text content of each image region after the first splicing; and S7, outputting the text content after the second splicing. The present invention intelligently reads the text content of multiple image regions on an image, thereby improving the accuracy of post-processing of the image text content.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method, system and computing storage medium for extracting text content from an image. Background Art

[0002] Currently, more and more industries are promoting services such as "face-to-face processing", "online processing", and "mobile phone processing", which are particularly common in the government sector. Processing matters often involves the review of documents' images, and the review content mainly focuses on the text content on the images. In this case, the text content on the images needs to be read and sorted in a certain order for output. The existing technology, OCR (Optical Character Recognition), can only recognize the text content on the image, but cannot intelligently sort it, resulting in a disordered order of the read text content. This unreasonable situation is particularly serious in some complex documents containing multiple materials. Summary of the Invention

[0003] The present invention provides a method for extracting text content from an image, so as to solve the technical problem in the prior art that the order of text content read from a complex document image containing multiple materials is disordered.

[0004] One aspect of the present invention is to provide a method for extracting text content from an image, the method comprising the following steps:

[0005] S1. Performing layout segmentation and recognition on the image to obtain the minimum bounding rectangle of each image area on the image;

[0006] S2. Calculate the first rotation angle of each image region using the minimum circumscribed rectangle of each image region, and intercept each image region by perspective transformation;

[0007] S3, acquiring all text boxes of each intercepted image area, determining a second rotation angle of each text box, and calculating a third rotation angle of each image area according to the second rotation angle;

[0008] Performing non-cropping rotation and straightening on each intercepted image region according to a third rotation angle;

[0009] S4, performing OCR recognition on each rotated and straightened image area, and performing a first splicing of the text content of each text box in each image area;

[0010] S5. Calculate the standard rotation angle of the image and rotate and straighten the image without cropping it;

[0011] S6. Sort each image region of the rotated and straightened image, and perform a second splicing on the text content of each image region after the first splicing according to the sorting;

[0012] S7. Output the text content after the second splicing.

[0013] In a preferred embodiment, in step S1, a layout segmentation recognition model is trained to perform layout segmentation recognition on the image to obtain the minimum bounding rectangle of each image area on the image.

[0014] In a preferred embodiment, step S2 includes the following method steps:

[0015] S201, determining the order of the four vertices of the minimum circumscribed rectangle of each image area;

[0016] S202, establishing a rectangular coordinate system for the minimum circumscribed rectangle of each image area, and determining the position of a first vertex among the four vertices of the minimum circumscribed rectangle of each image area, as well as the positions of the four vertices of the minimum circumscribed rectangle of each image area;

[0017] S203, determining a first rotation angle of a minimum circumscribed rectangle of each image region;

[0018] S204, rotating each image region counterclockwise according to a first rotation angle with the first vertex of the minimum circumscribed rectangle of each image region as the center;

[0019] S205 , intercepting each rotated image region by perspective transformation.

[0020] In a preferred embodiment, in step S201, the order of the four vertices of the minimum bounding rectangle of each image region is determined as follows:

[0021] Select any one of the four vertices of the minimum circumscribed rectangle of each image area as a preset first vertex;

[0022] Calculate the distance between the preset first vertex and the other three vertices, take the preset first vertex as the starting point, and determine the sequence numbers of the other three vertices according to the clockwise rule or the left-hand rule.

[0023] In a preferred embodiment, in step S202, the position of the first vertex is determined among the four vertices of the minimum bounding rectangle of each image region according to the following method:

[0024] Determine the coordinates of the midpoints of the four sides of the minimum bounding rectangle of each image region;

[0025] Connect the midpoints of the two opposite sides of the minimum circumscribed rectangle of each image area;

[0026] In the two opposite sides of the minimum circumscribed rectangle of each image area, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two opposite sides of the minimum circumscribed rectangle of each image area, forming two directed lines;

[0027] Using the left-hand rule and two directed lines, the position of the first vertex is determined among the four vertices of the minimum circumscribed rectangle of each image region.

[0028] In a preferred embodiment, in step S203, the first rotation angle of the minimum circumscribed rectangle of each image region is determined by the following method:

[0029] With the first vertex as the center, rotate the line connecting the first and second vertices counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the counterclockwise rotation of the line connecting the first and second vertices is used as the first rotation angle of the minimum circumscribed rectangle of each image area.

[0030] In a preferred embodiment, in step S205, when the area of ​​the image region is smaller than 1 / 20 of the area of ​​the picture, the image region is not cropped.

[0031] In a preferred embodiment, step S3 includes the following method steps:

[0032] S301, obtaining all text boxes in each intercepted image area;

[0033] S302: For all text boxes in each intercepted image area, intercept each text box by perspective transformation, including:

[0034] S3021. Determine the order of the four text box points of each text box;

[0035] S3022: Establish a rectangular coordinate system for each text box, and determine the position of the first text box point among the four text box points of each text box, as well as the positions of the four text box points of each text box;

[0036] S3023, determining the second rotation angle of each text box;

[0037] S3024, rotating each text box counterclockwise according to a second rotation angle with the first text box point of each text box as the center;

[0038] S3025, intercepting each rotated text box by perspective transformation;

[0039] S303, using a binary classification image direction determiner to determine each intercepted text box, and determining the flip angle of each text box based on the determination result;

[0040] S304: Taking the sum of the second rotation angle and the flip angle as a third rotation angle, and performing non-cropping rotation and straightening on each intercepted image region according to the third rotation angle.

[0041] In a preferred embodiment, in step S3021, the order of the four text box points of each text box is determined according to the following method:

[0042] Select any one of the four text box points of each text box as the preset first text box point;

[0043] Calculate the distance between the preset text box point and the other three text box points, take the preset first text box point as the starting point, and determine the sequence numbers of the other three text box points according to the clockwise rule or the left-hand rule.

[0044] In a preferred embodiment, in step S3022, the position of the first text box point is determined among the four text box points of each text box according to the following method:

[0045] Determine the coordinates of the midpoints of the four sides of each text box;

[0046] Connect the midpoints of the two opposite sides of each text box;

[0047] In the two opposite sides of each text box, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two opposite sides of each text box, forming two directed lines;

[0048] Using the left-hand rule and two directed lines, the position of the first text box point is determined among the four text box points of each text box.

[0049] In a preferred embodiment, in step S3023, the second rotation angle of each text box is determined by the following method:

[0050] With the first text box point as the center, rotate the longer side of the two adjacent sides of the first text box point counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the longer side of the two adjacent sides of the first text box point counterclockwise rotation is used as the second rotation angle of each text box.

[0051] In a preferred embodiment, in step S4, all text boxes in each image area are sorted in order from top to bottom and from left to right.

[0052] Perform the first splicing of the text contents in the sorted text boxes.

[0053] In a preferred embodiment, in step S5, the standard rotation angle of the image is calculated by the following method:

[0054] S501, obtaining all image regions captured from the image and the third rotation angles of all image regions;

[0055] S502: taking the sum of the third rotation angle and the first rotation angle of each image region as the fourth rotation angle of each image region;

[0056] S503 , performing simple clustering on all image regions, and obtaining a standard rotation angle of the image through the fourth rotation angles of all image regions.

[0057] In a preferred embodiment, in step S503, simple clustering of all image regions includes:

[0058] Among all image regions, the image regions where the positive and negative difference of the fourth rotation angle is less than 5° are classified as one category;

[0059] From the class containing the largest image area, find the fourth rotation angle with the smallest difference between the fourth rotation angle and the mean of the fourth rotation angles, and use it as the standard rotation angle of the image.

[0060] In a preferred embodiment, when there are two classes containing the most image areas,

[0061] Then, the number of text boxes in each of the two classes containing the largest image area is counted; in the class with the largest number of text boxes, the fourth rotation angle with the smallest difference between the fourth rotation angle and the mean of the fourth rotation angle is found, and this is used as the standard rotation angle of the image.

[0062] In a preferred embodiment, when the number of text boxes of the two classes containing the largest image area is the same,

[0063] The standard rotation angle is selected based on the variance of the fourth rotation angles of all image areas in the two classes containing the largest image areas.

[0064] In a preferred embodiment, in step S6, each image region of the rotated and straightened picture is sorted according to the following method:

[0065] A rectangular coordinate system is established for the image, and each image area of ​​the rotated and straightened image is sorted in ascending order according to the coordinates of the first vertex of the minimum circumscribed rectangle of each image area.

[0066] Another aspect of the present invention is to provide a system for extracting text content from an image, comprising:

[0067] A layout segmentation and recognition module is used to perform layout segmentation and recognition on the image and obtain the minimum circumscribed rectangle of each image area on the image;

[0068] An image region interception module, configured to calculate a first rotation angle of each image region using a minimum circumscribed rectangle of each image region, and intercept each image region by perspective transformation;

[0069] An image region rotation and straightening module is used to obtain all text boxes of each intercepted image region, determine a second rotation angle of each text box, and calculate a third rotation angle of each image region based on the second rotation angle;

[0070] Performing non-cropping rotation and straightening on each intercepted image region according to a third rotation angle;

[0071] The first stitching module is used to perform OCR recognition on each rotated and straightened image area and perform the first stitching on the text content of each text box in each image area;

[0072] Image rotation and straightening module, used to calculate the standard rotation angle of the image and rotate and straighten the image without cropping;

[0073] The second stitching module is used to sort each image area of ​​the rotated and straightened picture, stitch the text content of each image area after the first stitching for the second time according to the sorting, and output the text content after the second stitching.

[0074] Another aspect of the present invention is to provide a computer storage medium for storing computer-executable instructions, wherein the computer-executable instructions are used to execute a method for extracting text content from an image.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] The present invention provides a method, system and computing storage medium for extracting text content from images. For complex document images containing multiple materials, layout segmentation is used to divide each image area of ​​the image, simulating the way in which auditors read text content, and intelligently reading the text content of multiple image areas on the image. This can greatly improve the work efficiency of auditors and at the same time improve the accuracy of post-processing of image text content. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0078] Figure 1 The present invention is a flowchart of a method for extracting text content from an image.

[0079] Figure 2 is a schematic diagram of multiple image areas on a picture according to an embodiment of the present invention.

[0080] Figure 3 Schematic diagram of obtaining the minimum bounding rectangle of an image area on a picture in one embodiment of the present invention.

[0081] Figure 4 1 is a schematic diagram of the ordering of the four vertices of the minimum circumscribed rectangle of an image area in one embodiment of the present invention.

[0082] Figure 5 FIG. 1 is a schematic diagram of an image region forming two directed lines in one embodiment of the present invention.

[0083] Figure 6 This is a schematic diagram of determining the position of a first vertex among four vertices of a minimum circumscribed rectangle of an image area in one embodiment of the present invention.

[0084] Figure 7 This is a schematic diagram of determining the position of a first vertex among four vertices of a minimum circumscribed rectangle of an image area in another embodiment of the present invention.

[0085] Figure 8 This is a schematic diagram of determining the position of a first vertex among four vertices of a minimum circumscribed rectangle of an image area in yet another embodiment of the present invention.

[0086] Figure 9 Schematic diagram of a first rotation angle of a minimum circumscribed rectangle of an image area in one embodiment of the present invention.

[0087] Figure 10 3 is a schematic diagram of a first rotation angle of a minimum circumscribed rectangle of an image area in another embodiment of the present invention.

[0088] Figure 11 3 is a schematic diagram of a first rotation angle of a minimum circumscribed rectangle of an image area in yet another embodiment of the present invention.

[0089] Figure 12 It is a schematic diagram of intercepting an image area by perspective transformation in one embodiment of the present invention.

[0090] Figure 13 It is a schematic diagram of a text box captured in one embodiment of the present invention.

[0091] Figure 14 Schematic diagram of the flip angle of a text box in one embodiment of the present invention.

[0092] Figure 15 This is a schematic diagram of performing the first splicing of text contents of each text box in an image area in one embodiment of the present invention.

[0093] Figure 16 3 is a schematic diagram of the second splicing of text content in each image area in one embodiment of the present invention. DETAILED DESCRIPTION

[0094] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0095] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0097] like Figure 1 The flowchart of a method for extracting text content from an image of the present invention is shown. According to an embodiment of the present invention, a method for extracting text content from an image is provided, comprising the following steps:

[0098] Step S1: perform layout segmentation and recognition on the image to obtain the minimum circumscribed rectangle of each image area on the image.

[0099] like Figure 2 The figure shows a schematic diagram of multiple image areas on a picture in one embodiment of the present invention. In the present invention, the picture t has multiple image areas w, and the image areas w are document images (document materials), such as identity card images.

[0100] According to an embodiment of the present invention, a layout segmentation recognition model is trained to perform layout segmentation recognition on the image t, and the minimum bounding rectangle of each image region w on the image t is obtained. Figure 3FIG2 is a schematic diagram of obtaining the minimum circumscribed rectangle of an image area on a picture in one embodiment of the present invention. In the embodiment, the front and back ID card images are extracted as an example. In the embodiment, the front ID card image (image area w) is named as F area, and the back ID card image (image area w) is named as G area (as shown in FIG2 ). Figure 16 Detect the image area w of the F region and the image area w of the G region, and obtain the minimum bounding rectangle of the image area w of the F region and the image area w of the G region on the image.

[0101] It should be understood that for each image region w on the picture t, the minimum bounding rectangle of the image region w is obtained. The minimum bounding rectangle refers to the smallest rectangle surrounding the image region w.

[0102] Step S2: Calculate the first rotation angle of each image region w using the minimum circumscribed rectangle of each image region w, and intercept each image region w by perspective transformation.

[0103] According to an embodiment of the present invention, intercepting each image region w by perspective transformation includes the following method steps:

[0104] Step S201: Determine the order of the four vertices of the minimum bounding rectangle of each image region w.

[0105] like Figure 4 The figure shows a schematic diagram of sorting the four vertices of the minimum bounding rectangle of an image area in one embodiment of the present invention. The minimum bounding rectangle of each image area w has four vertices, numbered as the first vertex a1, the second vertex a2, the third vertex a3 and the fourth vertex a4 respectively.

[0106] The order of the four vertices of the minimum bounding rectangle of each image region w is determined as follows:

[0107] Any one of the four vertices of the minimum circumscribed rectangle of each image region w is selected as the preset first vertex a1.

[0108] Calculate the distance between the preset first vertex a1 and the other three vertices, take the preset first vertex a1 as the starting point, and determine the sequence numbers of the other three vertices according to the clockwise rule or the left-hand rule.

[0109] Taking the minimum bounding rectangle of an image region w as an example, any one of the four vertices of the minimum bounding rectangle of the image region w is selected as the preset first vertex a1, and the distances between the preset first vertex a1 and the other three vertices are calculated. The vertex farthest from the preset first vertex a1 is the third vertex a3. According to the clockwise rule, the first vertex a1, the second vertex a2, the third vertex a3 and the fourth vertex a4 are sorted in a clockwise direction, as shown in FIG. Figure 4 shown.

[0110] In one embodiment, any one of the four vertices of the minimum circumscribed rectangle of the image area w is selected as the preset first vertex a1, and the distances between the preset first vertex a1 and the other three vertices are calculated. The vertex farthest from the preset first vertex a1 is the third vertex a3. According to the left-hand rule, the palm of the left hand is facing upward, and the palm is regarded as the first vertex a1. The direction of the middle finger tip is the direction of the third vertex a3. The side where the thumb is located is the side where the second vertex a2 is located. In this way, the first vertex a1, the second vertex a2, the third vertex a3 and the fourth vertex a4 are sorted as follows: Figure 4 shown.

[0111] Step S202 : Establish a rectangular coordinate system for the minimum bounding rectangle of each image region w, and determine the position of the first vertex a1 and the positions of the four vertices of the minimum bounding rectangle of each image region w.

[0112] like Figure 5 The schematic diagram of an image region forming two directed lines in one embodiment of the present invention is shown. The position of the first vertex is determined among the four vertices of the minimum circumscribed rectangle of each image region w according to the following method:

[0113] Determine the coordinates of the midpoints of the four sides of the minimum bounding rectangle of each image region w;

[0114] Connect the midpoints of the two opposite sides of the minimum bounding rectangle of each image region w;

[0115] In the two opposite sides of the minimum circumscribed rectangle of each image region w, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two opposite sides of the minimum circumscribed rectangle of each image region w, forming two directed lines;

[0116] By using the left-hand rule and two directed lines, the position of the first vertex a1 is determined among the four vertices of the minimum circumscribed rectangle of each image region w.

[0117] like Figure 5 As shown in the figure, taking the minimum bounding rectangle of an image region w as an example, the midpoints of the four sides of the minimum bounding rectangle of image region w are points b, c, d, and e. Points b and c are the midpoints of two opposite sides of the minimum bounding rectangle of image region w, and points d and e are the midpoints of two opposite sides of the minimum bounding rectangle of image region w. The coordinates of point b are (x1, y1), the coordinates of point c are (x2, y2), the coordinates of point d are (x3, y3), and the coordinates of point e are (x4, y4).

[0118] Connect points b and c, and d and e. Compare the abscissas of b and c, and d and e. If b's abscissa x1 is less than c's abscissa x2, then b points to c, forming a directed line bc. If d's abscissa x3 is less than e's abscissa x4, then d points to e, forming a directed line de.

[0119] In some embodiments, when the horizontal coordinate values ​​of point b and point c are equal, the vertical coordinate values ​​of point b and point c are compared; when the horizontal coordinate values ​​of point d and point e are equal, the vertical coordinate values ​​of point d and point e are compared, and a directed line is formed by pointing from the midpoint with a smaller vertical coordinate value to the midpoint with a larger vertical coordinate value.

[0120] For example, when the abscissa x1 of point b is equal to the abscissa x2 of point c, and the ordinate y1 of point b is less than the ordinate y2 of point c, then point b points to point c, forming a directed line bc. Similarly, when the abscissa x3 of point d is equal to the abscissa x4 of point e, and the ordinate y3 of point d is less than the ordinate y4 of point e, then point d points to point e, forming a directed line de.

[0121] Using the left-hand rule and two directed lines, the position of the first vertex a1 is determined among the four vertices of the minimum bounding rectangle of the image region w. Specifically, the left palm faces upward, with the middle finger pointing to the directed lines bc and de. The two directions pointed by the thumb determine the position of the first vertex a1 among the four vertices of the minimum bounding rectangle of the image region w, as shown in the figure below: Figure 6 、 Figure 7 and Figure 8 shown.

[0122] Figure 6 is a schematic diagram of determining the position of the first vertex among the four vertices of the minimum circumscribed rectangle of the image area in one embodiment of the present invention. Figure 7 is a schematic diagram of determining the position of a first vertex among four vertices of a minimum circumscribed rectangle of an image area in another embodiment of the present invention. Figure 8 is a schematic diagram of determining the position of the first vertex among the four vertices of the minimum circumscribed rectangle of the image area in another embodiment of the present invention. Figure 6 、 Figure 7 and Figure 8 It is the position of the first vertex a1 among the four vertices of the minimum circumscribed rectangle of the image area determined when the image area w is placed in three different directions.

[0123] In step S201, the order of the four vertices of the minimum bounding rectangle of the image area w has been determined. After the position of the first vertex a1 is determined in step S202, the positions of the second vertex a2, the third vertex a3 and the fourth vertex a4 of the minimum bounding rectangle of the image area w can be determined according to the order of the four vertices of the minimum bounding rectangle of the image area w, thereby determining the positions of the four vertices of the minimum bounding rectangle of the image area w.

[0124] Step S203: Determine a first rotation angle of the minimum circumscribed rectangle of each image region w.

[0125] According to an embodiment of the present invention, the first rotation angle of the minimum circumscribed rectangle of each image region w is determined by the following method:

[0126] With the first vertex a1 as the center, rotate the line connecting the first vertex a1 and the second vertex a2 counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the counterclockwise rotation of the line connecting the first vertex a1 and the second vertex a2 is used as the first rotation angle A of the minimum circumscribed rectangle of each image area w.

[0127] like Figure 9 FIG. 1 is a schematic diagram showing a first rotation angle of a minimum circumscribed rectangle of an image area according to an embodiment of the present invention. Figure 10 Schematic diagram of the first rotation angle of the minimum circumscribed rectangle of an image area in another embodiment of the present invention, Figure 11 FIG. 1 is a schematic diagram showing a first rotation angle of a minimum circumscribed rectangle of an image area in another embodiment of the present invention, Figure 6 、 Figure 7 and Figure 8 It is the first rotation angle A of the minimum circumscribed rectangle of the image area w when the image area w is placed in three different directions.

[0128] Step S204 : rotating each image region w counterclockwise at a first rotation angle A with the first vertex a1 of the minimum circumscribed rectangle of each image region w as the center.

[0129] Step S205: intercept each rotated image region w by perspective transformation.

[0130] In step S204, each image region w is rotated counterclockwise according to the first rotation angle A, and each rotated image region w is intercepted by perspective transformation. Figure 12 The figure shows a schematic diagram of intercepting an image area by perspective transformation in one embodiment of the present invention. In the embodiment, an ID card image (image area w) placed on the reverse side is taken as an example, and the image area w is rotated according to a first rotation angle A and then intercepted.

[0131] In some embodiments, when the area of ​​the image region w is smaller than 1 / 20 of the area of ​​the picture t, the image region w is not captured, thereby filtering out unnecessary information blocks generated by the layout segmentation recognition model trained in step S1.

[0132] Step S3: Obtain all text boxes of each intercepted image area w, determine the second rotation angle of each text box, and calculate the third rotation angle of each image area w based on the second rotation angle, and perform non-cropping rotation and straightening on each intercepted image area w according to the third rotation angle.

[0133] like Figure 13 The diagram below shows a schematic diagram of extracting a text box in one embodiment of the present invention. This embodiment uses an image of an ID card (image region w) placed on the reverse side as an example. The text within text box k is the text to be extracted in the present invention. An image region w has multiple text boxes k. All text boxes k for each image region w extracted in step S2 are obtained. Text box k is a rectangular box with four text box points (the four vertices of text box k).

[0134] According to an embodiment of the present invention, performing non-cropping rotation and straightening on each intercepted image region w according to a third rotation angle includes:

[0135] Step S301: Obtain all text boxes k in each captured image region w.

[0136] Step S302: For all text boxes in each intercepted image region w, intercept each text box by perspective transformation, including:

[0137] Step S3021: Determine the order of the four text box points of each text box k.

[0138] The ordering of the four text box points of each text box k in step S3021 is the same as the ordering of the four vertices of the minimum bounding rectangle of each image region w in step S201. Specifically, the ordering of the four text box points of each text box k is determined as follows:

[0139] Select any one of the four text box points of each text box k as the preset first text box point;

[0140] Calculate the distance between the preset text box point and the other three text box points, take the preset first text box point as the starting point, and determine the sequence numbers of the other three text box points according to the clockwise rule or the left-hand rule.

[0141] Step S3022: Establish a rectangular coordinate system for each text box k, and determine the position of the first text box point among the four text box points of each text box k, as well as the positions of the four text box points of each text box k.

[0142] The method for determining the position of the first text box point and the positions of the four text box points of each text box k in step S3022 is the same as the method for determining the position of the first vertex a1 and the positions of the four vertices of the minimum bounding rectangle of each image region w in step S202. Specifically, the position of the first text box point is determined among the four text box points of each text box k in the following manner:

[0143] Determine the coordinates of the midpoints of the four sides of each text box k;

[0144] Connect the midpoints of the two opposite sides of each text box k;

[0145] In the two sides opposite to each text box k, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two sides opposite to each text box k, forming two directed lines;

[0146] Using the left-hand rule and two directed lines, the position of the first text box point is determined among the four text box points of each text box k.

[0147] Step S3023: Determine the second rotation angle of each text box k.

[0148] According to an embodiment of the present invention, the second rotation angle of each text box k is determined by the following method:

[0149] With the first text box point as the center, the longer side of the two adjacent sides of the first text box point is rotated counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the longer side of the two adjacent sides of the first text box point is used as the second rotation angle B of each text box k, as shown in Figure 13 shown.

[0150] Step S3024: rotate each text box k counterclockwise according to the second rotation angle B with the first text box point of each text box k as the center.

[0151] Step S3025: intercept each rotated text box k by perspective transformation.

[0152] Step S303: using a binary classification image direction determiner, determine each intercepted text box k, and determine the flip angle (180° or 0°) of each text box k based on the determination result.

[0153] According to an embodiment of the present invention, each intercepted text box k may have the text content in the text box k upside down. The present invention uses a binary classification image direction determiner to determine each intercepted text box k, and flips the text box k by 180° or 0° according to the judgment result. In the embodiment, the flip angle is recorded as flip angle C.

[0154] like Figure 14The diagram shows the flip angle of a text box in one embodiment of the present invention. For example, when the text content in text box k is not reversed, the flip angle C = 0°. When the text content in text box k is reversed, the flip angle C = 180°.

[0155] Step S304: Taking the sum of the second rotation angle B and the flip angle C as the third rotation angle, and performing non-cropping rotation and straightening on each intercepted image region w according to the third rotation angle.

[0156] In the embodiment, the third rotation angle is recorded as D, and each intercepted image region w is rotated and straightened according to the third rotation angle D without being cropped (the image region w is rotated and straightened counterclockwise).

[0157] Step S4: Perform OCR recognition on each rotated and straightened image region w, and perform a first splicing on the text content of each text box k in each image region w.

[0158] like Figure 15 The figure shows a schematic diagram of performing the first splicing of the text contents of each text box in the image area in one embodiment of the present invention. All text boxes k in each image area w are sorted in order from top to bottom and from left to right, and the text contents in the sorted text boxes k are spliced ​​for the first time.

[0159] In the embodiment, an ID card image (image area w) placed on the reverse side is used as an example. There are 6 text boxes k in the image area w. After the image area w is rotated and straightened, the 6 text boxes k are sorted from top to bottom and from left to right as: text box k 1, text box k 2, text box k 3, text box k 4, text box k 5 and text box k 6. The text contents in the sorted text boxes k are spliced ​​for the first time, as shown in FIG. Figure 15 shown.

[0160] When the text contents in the sorted text boxes k are first concatenated, if there are spaces in the text contents of a single text box k, the spaces need to be removed. When different text boxes k are concatenated, they are separated by a space.

[0161] Step S5: Calculate the standard rotation angle of the image t, and perform rotation and straightening on the image t without cropping.

[0162] According to an embodiment of the present invention, the standard rotation angle of the image t is calculated by the following method:

[0163] Step S501: Obtain all image regions w captured from a picture t and a third rotation angle D of all image regions w.

[0164] Step S502: The sum of the third rotation angle D and the first rotation angle A of each image region w is used as the fourth rotation angle E of each image region w.

[0165] Step S503: Perform simple clustering on all image regions w, and obtain the standard rotation angle of the image t through the fourth rotation angle E of all image regions w.

[0166] According to an embodiment of the present invention, a simple clustering of all image regions w includes:

[0167] Among all image regions w captured in the picture t, the image regions w in which the positive and negative differences of the fourth rotation angle E are less than 5° are classified into one category.

[0168] From the class containing the largest image area w, find the fourth rotation angle E with the smallest difference between the fourth rotation angle E and the mean of all fourth rotation angles E in the class, and use it as the standard rotation angle of image t.

[0169] In some embodiments, when there are two classes containing the most image areas w, the number of text boxes k in each of the two classes containing the most image areas w is counted. In the class with the most text boxes k, the fourth rotation angle E having the smallest difference between the fourth rotation angle E and the mean of all fourth rotation angles E in that class is found, and this fourth rotation angle E is used as the standard rotation angle for image t.

[0170] In some embodiments, when there are two classes containing the most image areas w, and when the number of text boxes k of the two classes containing the most image areas w is the same, the standard rotation angle is selected based on the variance of the fourth rotation angles E of all image areas w in the two classes containing the most image areas w.

[0171] Step S6: sort each image region w of the rotated and straightened picture t, and perform a second splicing on the text content of each image region w after the first splicing according to the sorting.

[0172] like Figure 16 The diagram shows a second stitching of text content in each image region in one embodiment of the present invention. After the image t is rotated and straightened according to a standard rotation angle without cropping, each image region w of the rotated and straightened image t is sorted according to the following method:

[0173] A rectangular coordinate system is established for the image t, and each image region on the rotated and straightened image is sorted in ascending order according to the coordinates of the first vertex a1 of the minimum circumscribed rectangle of each image region w.

[0174] In this embodiment, the extraction of the front and back images of an ID card is taken as an example. Figure 16As shown, after image t is rotated and straightened according to the standard rotation angle without cropping, the coordinates of the first vertex a1 of each image area are recalculated in the rectangular coordinate system established for image t. In this embodiment, the coordinates of the first vertex a1 of the minimum bounding rectangle of image area w in region F are (xF, yF), and the coordinates of the first vertex a1 of the minimum bounding rectangle of image area w in region G are (xG, yG).

[0175] Compare the coordinates of the first vertex a1 of the minimum circumscribed rectangle of the image area w in region F (xF, yF) with the coordinates of the first vertex a1 of the minimum circumscribed rectangle of the image area w in region G (xG, yG), and arrange the image area w with a larger vertical coordinate of the first vertex a1 before the image area w with a smaller vertical coordinate of the first vertex a1.

[0176] For example, when the ordinate yF of the first vertex a1 of the minimum circumscribed rectangle of the image area w in area F is smaller than the ordinate yG of the first vertex a1 of the minimum circumscribed rectangle of the image area w in area G, the image area w in area G is placed before the image area w in area F.

[0177] In some embodiments, when the ordinates of the first vertices a1 of the image regions w are the same, the image region w with the larger ordinate of the first vertex a1 is arranged before the image region w with the smaller ordinate of the first vertex a1.

[0178] For example, when the horizontal coordinate xF of the first vertex a1 of the minimum circumscribed rectangle of the image area w in area F is smaller than the horizontal coordinate xG of the first vertex a1 of the minimum circumscribed rectangle of the image area w in area G, the image area w in area G is arranged before the image area w in area F.

[0179] The text content of each sorted image region w is spliced ​​for the second time, and two spaces are used to separate the text content of different image regions w.

[0180] Step S7: output the text content after the second splicing.

[0181] After sorting each image region w of the rotated and straightened picture t according to step S6, the text content of each image region w after the first splicing is spliced ​​for the second time, and the spliced ​​text content is output.

[0182] According to an embodiment of the present invention, a system for extracting text content from an image is provided, which is used to execute a method for extracting text content from an image provided by the present invention, comprising:

[0183] The layout segmentation and recognition module is used to perform layout segmentation and recognition on the image and obtain the minimum circumscribed rectangle of each image area on the image.

[0184] The image region interception module is used to calculate the first rotation angle of each image region using the minimum circumscribed rectangle of each image region, and intercept each image region by perspective transformation.

[0185] The image area rotation and straightening module is used to obtain all text boxes in each intercepted image area, determine the second rotation angle of each text box, and calculate the third rotation angle of each image area based on the second rotation angle; and perform non-cropping rotation and straightening on each intercepted image area according to the third rotation angle.

[0186] The first stitching module is used to perform OCR recognition on each rotated and straightened image area, and to perform the first stitching on the text content of each text box in each image area.

[0187] The image rotation and straightening module is used to calculate the standard rotation angle of the image and rotate and straighten the image without cropping.

[0188] The second stitching module is used to sort each image area of ​​the rotated and straightened picture, stitch the text content of each image area after the first stitching for the second time according to the sorting, and output the text content after the second stitching.

[0189] According to an embodiment of the present invention, a computer storage medium is provided for storing computer-executable instructions, wherein the computer-executable instructions are used to execute a method for extracting text content from an image provided by the present invention.

[0190] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for extracting text content from an image, characterized in that: The extraction method comprises the following steps: S1. Performing layout segmentation and recognition on the image to obtain the minimum bounding rectangle of each image area on the image; S2. Calculate the first rotation angle of each image region using the minimum circumscribed rectangle of each image region, and intercept each image region by perspective transformation; S3, acquiring all text boxes of each intercepted image area, determining a second rotation angle of each text box, and calculating a third rotation angle of each image area according to the second rotation angle; Performing non-cropping rotation and straightening on each intercepted image region according to a third rotation angle; S4, performing OCR recognition on each rotated and straightened image area, and performing a first splicing of the text content of each text box in each image area; S5. Calculate the standard rotation angle of the image and rotate and straighten the image without cropping it; S6. Sort each image region of the rotated and straightened image, and perform a second splicing on the text content of each image region after the first splicing according to the sorting; S7, outputting the text content after the second splicing; Wherein, step S2 includes the following method steps: S201, determining the order of the four vertices of the minimum bounding rectangle of each image area, including: The order of the four vertices of the minimum bounding rectangle of each image region is determined as follows: Select any one of the four vertices of the minimum circumscribed rectangle of each image area as a preset first vertex; Calculate the distance between the preset first vertex and the other three vertices, take the preset first vertex as the starting point, and determine the sequence numbers of the other three vertices according to the clockwise rule or the left-hand rule; S202, establishing a rectangular coordinate system for the minimum circumscribed rectangle of each image area, and determining the position of a first vertex among the four vertices of the minimum circumscribed rectangle of each image area, as well as the positions of the four vertices of the minimum circumscribed rectangle of each image area; The position of the first vertex is determined among the four vertices of the minimum circumscribed rectangle of each image region according to the following method: Determine the coordinates of the midpoints of the four sides of the minimum bounding rectangle of each image region; Connect the midpoints of the two opposite sides of the minimum circumscribed rectangle of each image area; In the two opposite sides of the minimum circumscribed rectangle of each image area, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two opposite sides of the minimum circumscribed rectangle of each image area, forming two directed lines; Using the left-hand rule and two directed lines, determine the position of the first vertex among the four vertices of the minimum circumscribed rectangle of each image region; S203: Determine a first rotation angle of a minimum circumscribed rectangle of each image region, wherein the first rotation angle of the minimum circumscribed rectangle of each image region is determined by the following method: With the first vertex as the center, rotate the line connecting the first and second vertices counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the counterclockwise rotation of the line connecting the first and second vertices is used as the first rotation angle of the minimum circumscribed rectangle of each image area. S204, rotating each image region counterclockwise according to a first rotation angle with the first vertex of the minimum circumscribed rectangle of each image region as the center; S205 , intercepting each rotated image region by perspective transformation.

2. The extraction method according to claim 1, wherein Step S3 includes the following method steps: S301, obtaining all text boxes in each intercepted image area; S302: For all text boxes in each intercepted image area, intercept each text box by perspective transformation, including: S3021. Determine the order of the four text box points of each text box. The order of the four text box points of each text box is determined according to the following method: Select any one of the four text box points of each text box as the preset first text box point; Calculate the distance between the preset text box point and the other three text box points, take the preset first text box point as the starting point, and determine the sequence numbers of the other three text box points according to the clockwise rule or the left-hand rule; S3022. Establish a rectangular coordinate system for each text box, and determine the position of a first text box point among the four text box points of each text box, as well as the positions of the four text box points of each text box. The position of the first text box point among the four text box points of each text box is determined according to the following method: Determine the coordinates of the midpoints of the four sides of each text box; Connect the midpoints of the two opposite sides of each text box; In the two opposite sides of each text box, the midpoint with the smaller coordinate value points to the midpoint with the larger coordinate value as the direction of the line connecting the midpoints of the two opposite sides of each text box, forming two directed lines; Using the left-hand rule and two directed lines, determine the position of the first text box point among the four text box points of each text box; S3023: Determine the second rotation angle of each text box, wherein the second rotation angle of each text box is determined by the following method: With the first text box point as the center, rotate the longer side of the two adjacent sides of the first text box point counterclockwise until it coincides with the X-axis of the rectangular coordinate system. The angle of the longer side of the two adjacent sides of the first text box point counterclockwise rotation is used as the second rotation angle of each text box; S3024, rotating each text box counterclockwise according to a second rotation angle with the first text box point of each text box as the center; S3025, intercepting each rotated text box by perspective transformation; S303, using a binary classification image direction determiner to determine each intercepted text box, and determining the flip angle of each text box based on the determination result; S304: Taking the sum of the second rotation angle and the flip angle as a third rotation angle, and performing non-cropping rotation and straightening on each intercepted image region according to the third rotation angle.

3. The extraction method according to claim 1, wherein In step S4, all text boxes in each image area are sorted in order from top to bottom and from left to right. Perform the first splicing of the text contents in the sorted text boxes.

4. The extraction method according to claim 1, wherein In step S5, the standard rotation angle of the image is calculated by the following method: S501, obtaining all image regions captured from the image and the third rotation angles of all image regions; S502: taking the sum of the third rotation angle and the first rotation angle of each image region as the fourth rotation angle of each image region; S503 , performing simple clustering on all image regions, and obtaining a standard rotation angle of the image through the fourth rotation angles of all image regions.

5. The extraction method according to claim 4, characterized in that In step S503, simple clustering of all image regions includes: Among all image regions, the image regions where the positive and negative difference of the fourth rotation angle is less than 5° are classified as one category; From the class containing the largest image area, find the fourth rotation angle with the smallest difference between the fourth rotation angle and the mean of the fourth rotation angles, and use it as the standard rotation angle of the image.

6. The extraction method according to claim 5, characterized in that When the number of classes containing the most image areas is two, Then, count the number of text boxes in each of the two classes containing the largest image area; in the class with the largest number of text boxes, find the fourth rotation angle with the smallest difference between the fourth rotation angle and the mean of the fourth rotation angles, and use this as the standard rotation angle for the image; When the number of text boxes of the two classes containing the largest image area is the same, The standard rotation angle is selected based on the variance of the fourth rotation angles of all image areas in the two classes containing the largest image areas.

7. The extraction method according to claim 1, characterized in that In step S6, each image region of the rotated and straightened image is sorted according to the following method: A rectangular coordinate system is established for the image, and each image area of ​​the rotated and straightened image is sorted in ascending order according to the coordinates of the first vertex of the minimum circumscribed rectangle of each image area.

8. A system for extracting text content from an image, characterized in that: The extraction system is used to perform the extraction method according to any one of claims 1 to 7, include, A layout segmentation and recognition module is used to perform layout segmentation and recognition on the image and obtain the minimum circumscribed rectangle of each image area on the image; An image region interception module, configured to calculate a first rotation angle of each image region using a minimum circumscribed rectangle of each image region, and intercept each image region by perspective transformation; An image region rotation and straightening module is used to obtain all text boxes of each intercepted image region, determine a second rotation angle of each text box, and calculate a third rotation angle of each image region based on the second rotation angle; Performing non-cropping rotation and straightening on each intercepted image region according to a third rotation angle; The first stitching module is used to perform OCR recognition on each rotated and straightened image area and perform the first stitching on the text content of each text box in each image area; Image rotation and straightening module, used to calculate the standard rotation angle of the image and rotate and straighten the image without cropping; The second stitching module is used to sort each image area of ​​the rotated and straightened picture, stitch the text content of each image area after the first stitching for the second time according to the sorting, and output the text content after the second stitching.

9. A computer storage medium, characterized in that Used to store computer-executable instructions, wherein the computer-executable instructions are used to execute the extraction method according to any one of claims 1 to 7.

Images