Text image recognition method, device, equipment and storage medium

Abstract

The present application discloses a text image recognition method, device, equipment and storage medium, the method comprising: obtaining a text image, the text image comprising M text boxes, obtaining attribute information of each text box in the M text boxes according to a pre-trained detection model, determining association information of each text box in the M text boxes according to the attribute information of the M text boxes, and determining text structure information of the text image according to the attribute information and association information of each text box in the M text boxes, the text structure information comprising the typesetting structure of the M text boxes and the text category of each text box. Thus, the text structure information of the text image can be accurately identified, and the applicability of text image recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a text image recognition method, device, equipment and storage medium. Background Art

[0002] With the widespread application of artificial intelligence (AI) technology in educational scenarios and the increasing demand for electronic teaching in schools, optical character recognition (OCR) technology has been increasingly widely used in scenarios such as automatic grading, analysis of supplementary teaching questions, and automatic search of questions. Among them, OCR refers to the process in which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text. OCR technology can be used to perform character recognition on images containing text and output the text in the image as editable text.

[0003] In an existing method for text image recognition combined with OCR technology, a template database is pre-established based on sample document images of different typesetting styles, the border of the text image is first extracted, the title of the text image is recognized through OCR, and the document type of the text image is obtained based on the border and title of the text image. Then, according to the document type of the recognized text image, the template in the template database is called to perform OCR recognition on the text image to obtain the text and text structure information (i.e., typesetting style) in the text image.

[0004] In the above method, the text in the text image needs to be recognized by OCR to obtain semantic information, and then combined with template matching to obtain the text structure information of the text image. On the one hand, the templates in the template database are limited, so the applicability is poor. On the other hand, the text images taken by users are tilted, perspective and paper distorted. In these cases, semantic information cannot be accurately obtained through OCR recognition, resulting in low accuracy in text structure information recognition. Summary of the invention

[0005] The present application provides a text image recognition method, apparatus, device and storage medium to solve the problems of poor applicability and low accuracy of text image recognition.

[0006] In a first aspect, the present application provides a text image recognition method, comprising:

[0007] Acquire a text image, wherein the text image includes M text boxes, where M is a positive integer;

[0008] According to the pre-trained detection model, obtain attribute information of each text box in the M text boxes;

[0009] Determine association information of each of the M text boxes according to the attribute information of the M text boxes;

[0010] According to the attribute information and the association information of each text box in the M text boxes, text structure information of the text image is determined, and the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

[0011] In a second aspect, the present application provides a text image recognition device, comprising:

[0012] A first acquisition module is used to acquire a text image, wherein the text image includes M text boxes, where M is a positive integer;

[0013] A second acquisition module is used to acquire attribute information of each text box in the M text boxes according to a pre-trained detection model;

[0014] A first determining module, configured to determine association information of each of the M text boxes according to the attribute information of the M text boxes;

[0015] The second determination module is used to determine the text structure information of the text image according to the attribute information and the association information of each text box in the M text boxes, and the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

[0016] In a third aspect, the present application provides an electronic device, including:

[0017] Processor; and

[0018] A memory, configured to store executable instructions of the processor;

[0019] The processor is configured to execute the text image recognition method described in any one of the first aspect or possible implementations of the first aspect by executing the executable instructions.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the text image recognition method described in the first aspect or any possible implementation manner of the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text image recognition method described in the first aspect or any possible implementation manner of the first aspect.

[0022] The text image recognition method, device, equipment and storage medium provided by the present application obtain the attribute information of each text box in the M text boxes of the text image according to the pre-trained detection model, determine the associated information of each text box in the M text boxes according to the attribute information of the M text boxes, and determine the text structure information of the text image according to the attribute information and associated information of each text box in the M text boxes. Since the text structure information of the text image is determined based on the attribute information and associated information of the text box, the attribute information and associated information of the text box can still be accurately obtained when the text image is tilted, distorted or perspective, and there is no need to perform text recognition to obtain semantic information. Therefore, the text structure information of the text image can be accurately identified, and compared with the template matching method, the applicability of text image recognition is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of an application scenario of the text image recognition method provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a text image recognition method provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of attribute information of a text box provided in an embodiment of the present application;

[0026] Figure 4 A flowchart of a text image recognition method provided in an embodiment of the present application;

[0027] Figure 5 Schematic diagram of the constructed text tree;

[0028] Figure 6 A schematic diagram of the structure of a text image recognition device provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of the structure of a text image recognition device provided in an embodiment of the present application;

[0030] Figure 8 It is a schematic block diagram of an electronic device 200 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0033] In the related art, when performing text image recognition, it is necessary to obtain semantic information from the text in the text image through OCR recognition, and then combine it with the template matching in the pre-established template database to obtain the text structure information of the text image. The text structure information here can be text typesetting information. However, the templates in the template database are limited. For text images that exceed the templates in the template database, the text structure information of the text image cannot be recognized, so the applicability is poor. For example, the text image to be recognized is an image obtained by photographing teaching supplementary materials. Teaching supplementary materials are various study guidance, examination guidance and other publications that are matched with textbooks and used by students, including textbook synchronous exercise publications, winter and summer vacation homework publications, primary and secondary school study questions, test paper publications, and other study and examination guidance publications for primary and secondary school students recognized by the press and publication administrative departments at or above the provincial level. The product forms of teaching supplementary materials include books, newspapers, periodicals, audio-visual products and electronic publications. There are various types of layout styles for questions in supplementary teaching materials, and there is no fixed type of layout style. The number of small questions in each type of question (such as multiple-choice questions, true-or-false questions, fill-in-the-blank questions, calculation questions, answer questions, and compound questions, etc.) is not fixed. Therefore, the above method cannot accurately identify the text structure information of text images in supplementary teaching materials, and the applicability of the above method is poor. On the other hand, the text images taken by users are tilted, perspective, and paper distorted. In these cases, OCR recognition cannot accurately obtain semantic information, resulting in low accuracy in text structure information recognition.

[0034] To solve this problem, the embodiments of the present application provide a text image recognition method, apparatus, device and storage medium. When identifying text structure information of a text image, the attribute information of each of the M text boxes of the text image is first obtained according to a pre-trained detection model, and then the associated information of each of the M text boxes is determined according to the attribute information of the M text boxes. Then, the text structure information of the text image is determined according to the attribute information and associated information of each of the M text boxes. Since the text structure information of the text image is determined according to the attribute information and associated information of the text box, the attribute information and associated information of the text box can still be accurately obtained when the text image is tilted, distorted or perspective, and there is no need to perform text recognition to obtain semantic information. Therefore, the text structure information of the text image can be accurately identified. Compared with the template matching method, the applicability of text image recognition is also improved.

[0035] Next, the application scenarios involved in the embodiments of the present application are illustrated.

[0036] The text image recognition method provided in the embodiment of the present application can be applied to at least the following application scenarios, which are described below with reference to the accompanying drawings.

[0037] For example, Figure 1 A schematic diagram of an application scenario of the text image recognition method provided in the embodiment of the present application, such as Figure 1 As shown, the application scenario involves an electronic device 101, and the electronic device 101 can be various terminal devices, such as a mobile phone, a personal computer, a tablet computer, a learning machine, a tutoring machine or a portable device. The electronic device 101 can obtain a text image, specifically, the electronic device 101 can obtain a text image by shooting itself, or receive a text image sent by another electronic device with a shooting function, and then execute the text image recognition method provided by the embodiment of the present application to recognize the text image and recognize the text structure information of the text image. The text structure information includes the typesetting structure of multiple text boxes included in the text image and the text category of each text box. Further, the text image can also be recognized according to the text structure information, and an editable document that is typeset according to the text structure information is finally output. Further, the electronic device 101 can also display the text structure information of the text image or the editable document corresponding to the text image (i.e., the recognition result of the text image). It is convenient for users to query, analyze or correct the editable document corresponding to the text image.

[0038] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0039] Figure 2 The flowchart of a text image recognition method provided in an embodiment of the present application can be performed by a text image recognition device, which can be implemented by software and / or hardware. The text image recognition device can be an electronic device or a chip or circuit of an electronic device. Figure 2 As shown, the method of this embodiment may include:

[0040] S101. Acquire a text image, where the text image includes M text boxes, where M is a positive integer.

[0041] Specifically, obtaining a text image can be directly photographing the text image by the electronic device, or receiving a text image sent by other devices. The text image recognition method provided in the embodiment of the present application mainly recognizes images containing text and identifies text structure information, which can be text typesetting structure information.

[0042] S102. Obtain attribute information of each text box in the M text boxes according to a pre-trained detection model.

[0043] Specifically, the attribute information of the text box is information used to represent at least one of the size, outline and text category of the text box. The attribute information of the text box may, for example, include at least one of the circumscribed rectangle of the text box, the text category of the text box and the outer outline of the text box.

[0044] The detection model is pre-trained based on sample data, the sample data includes multiple sample text images, the sample text images are input to the detection model, and the original attribute information of each text box included in the sample text image is output from the detection model, wherein the original attribute information of the text box may include the circumscribed rectangle of the text box, the text category of the text box, and the segmentation mask corresponding to the text box. Specifically, S102 may include:

[0045] S1021. Input the text image into a pre-trained detection model, and output the original attribute information of M text boxes included in the text image. The original attribute information of each text box includes the circumscribed rectangle of the text box, the text category of the text box, and the segmentation mask corresponding to the text box.

[0046] Among them, the bounding rectangle of the text box is a tuple composed of (x1, y1, x2, y2), where (x1, y1) and (x2, y2) are the coordinates of the upper left corner and the lower right corner of the bounding rectangle of the text box respectively. The text category of the text box is the question type classification corresponding to the text in the text box, which can be an enumeration variable composed of all question types in the teaching materials. The text category of the text box can include multiple-choice questions, true-or-false questions, fill-in-the-blank questions, calculation questions, solution questions, compound questions and answer areas, etc., where the answer area refers to the answer area in any text category. It can be understood that, generally speaking, there is one answer area for multiple-choice questions, true-or-false questions, solution questions and calculation questions, and there are one or more answer areas for fill-in-the-blank questions. The segmentation mask corresponding to the text box can be a 28*28 matrix, in which each element is the confidence of a pixel point, and the confidence can be a value between 0 and 1.

[0047] S1022: Obtain the outer contour of each text box according to the circumscribed rectangle of each text box and the segmentation mask corresponding to each text box.

[0048] Specifically, taking a text box as an example, the segmentation mask corresponding to the text box is adjusted to a first segmentation mask of the same size as the circumscribed rectangle of the text box, and the value of each element in the first segmentation mask is set, specifically, according to a preset threshold (such as 0.5), the value of the element whose confidence is greater than or equal to the preset threshold is modified to 1, and the value of the element whose confidence is less than or equal to the preset threshold is modified to 0, to obtain a second segmentation mask, and then, the contour extraction algorithm is used to extract the outer contour of the text box from the second segmentation mask. At this time, the coordinate points of the outer contour of the text box are the coordinates relative to the circumscribed rectangle of the text box. The offset (x1, y1) is added to all points in the outer contour of the extracted text box, and the outer contour of the text box in absolute coordinates can be obtained, and the outer contour of the text box in absolute coordinates is determined as the outer contour of the text box.

[0049] Figure 3 A schematic diagram of attribute information of a text box provided in an embodiment of the present application, such as Figure 3 As shown in the figure, there are three types of text boxes, one is multiple-choice questions, one is true-or-false questions, and the other is answer area. Multiple-choice questions have 3 text boxes (including a text box corresponding to a large question and two text boxes corresponding to small questions), true-or-false questions have 3 text boxes (including a text box corresponding to a large question and two text boxes corresponding to small questions), and there are a total of 4 answer area text boxes (two answer area text boxes for multiple-choice questions and two answer area text boxes for true-or-false questions). The attribute information of each text box includes Figure 3 Shown: the bounding rectangle of the text box, the text class of the text box, and the outer contour of the text box.

[0050] S103: Determine association information of each of the M text boxes according to the attribute information of the M text boxes.

[0051] Specifically, the associated information of a text box is information used to represent the hierarchical relationship of a text box and / or the answer area list of a text box. The associated information of a text box may, for example, include at least one of the level of the text box, the next-level text box list of the text box, and the answer area list of the text box. It can be understood that if a text box is an answer area, since the answer area has no level, no next-level text box list, and no answer area list, the associated information of the text box does not include the above three items of information. Therefore, in an implementable method, the text boxes whose text category is the answer area can be first determined, and the number of text boxes other than the text category of the answer area in the M text boxes is recorded as N, and then the associated information of each of the N text boxes is determined based on the attribute information of the N text boxes.

[0052] Optionally, after acquiring the attribute information of each text box in the M text boxes included in the text image through S102, the text category of each text box can be obtained based on the attribute information of each text box. The method of this embodiment may also include: extracting the text box with the text category of the answer area from the M text boxes, and storing the attribute information of the text box with the text category of the answer area into the answer area text box set.

[0053] In this embodiment, the level of the text box can be the first level, the second level, the third level, etc., for example Figure 1 The text box whose text category is multiple-choice question has a first-level text box and a second-level text box. The next-level text box list of the text box refers to a list composed of one or more next-level text boxes of the text box, such as Figure 1 The first level multiple choice question text box shown has 3 sub-level text boxes, and the first level true or false question text box has 4 sub-level text boxes. The answer area list of a text box refers to a list composed of text boxes in the answer area of ​​a text box.

[0054] As an implementable manner, determining the associated information of each of the M text boxes according to the attribute information of the M text boxes in S103 may specifically include:

[0055] S1031. Construct a text node according to the attribute information of each text box in N text boxes, where the N text boxes are text boxes other than the text box with the answering area as the text category among the M text boxes, and N is a positive integer.

[0056] The attribute information of each text node may include a node identifier, a bounding rectangle, a text category, and an outer contour. The bounding rectangle, text category, and outer contour of a text node are the bounding rectangle, text category, and outer contour of the corresponding text box. The node identifier is an identifier assigned to the text node when the text node is constructed.

[0057] S1032. Determine a text node set according to the attribute information of each text node and the text category, wherein the text node set includes the text category and a text node list of the text category.

[0058] For example, the text image includes three text categories, namely, multiple choice questions, true or false questions, and fill-in-the-blank questions. There are 6 text nodes whose text category is multiple choice questions, 4 text nodes whose text category is true or false questions, and 3 text nodes whose text category is fill-in-the-blank questions. Then the text node set of the determined text image can be {multiple choice questions: text node 1, text node 2, text node 3, text node 5, text node 6, and text node 7; true or false questions: text node 4, text node 8, text node 10, and text node 12; fill-in-the-blank questions: text node 9, text node 11, text node 13}.

[0059] S1033. Determine the main text node and the sub-text node list of the main text node in the text node list of each text category based on the text node set, determine the level of the text box corresponding to the main text node as the first level, and determine the next-level text box list of the text box corresponding to each main text node based on the sub-text node list of each main text node.

[0060] Specifically, S1033 may include:

[0061] For each text node list of a text category, take any two text nodes in the text node list (such as node i and node j ), the outer contour of each text node can be obtained according to the attribute information of the text node, and node i The outer contour is contour i , remember node j The outer contour is contour j , calculate the intersection area Si of the two outer contours j , calculate contour i The area enclosed is S i , calculate contour j The area enclosed is S j , define r i =S ij / S i , r j =S ij / S j , if r i >r j >Preset threshold (for example, 0.7), then determine contour i Contour j If r j >ri >Preset threshold (for example, 0.7), then determine contour j Contour i .

[0062] If contour i Contour j , then determine the node j For node i The child text node of node i is the main text node, and node j Add to node i In the list of child text nodes of contour j Contour i , then determine the node i For node j The child text node of node j is the main text node, and node i Add to node j The list of child text nodes.

[0063] Through the above method, the main text node and the sub-text node list of the main text node in the text node list of each text category are obtained. Furthermore, the level of the text box corresponding to each main text node is the first level, and the text box list corresponding to the sub-text node list of each main text node is the next level text box list of the text box corresponding to the main text node.

[0064] S1034. Determine the answer area list of the text box corresponding to each text node according to the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set and the answer area text box set.

[0065] The answer area text box set may include an answer area text box identifier and attribute information of the answer area text box.

[0066] Optionally, in S1034, based on the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set and the answer area text box set, the answer area list of the text box corresponding to each text node is determined, which can be:

[0067] S10341. Construct a text tree according to the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set.

[0068] Specifically, as an implementable method, the text image can be used as the root node, the main text node in the text node list of all text categories in the text node set can be used as the child node of the root node, and the child text node of each main text node can be used as the child node of the main text node.

[0069] S10342. Determine the text node to which each answer area text box in the answer area text box set belongs by covering the outer contour according to the text tree, determine the answer area list of each text node according to the text node to which each answer area text box belongs, and determine the answer area list of each text node as the answer area list of the text box corresponding to each text node.

[0070] As an implementable method, the text node to which each answer area text box in the answer area text box set belongs is determined by covering the outer contour according to the text tree, which can be specifically:

[0071] For each answer area text box in the answer area text box set, all child nodes of the root node in the text tree are traversed in turn, and the currently visited target node is obtained, which is a child node of the root node.

[0072] The answer area text box is determined to be included in the target node by covering the outer contour. The outer contour covering method is:

[0073] Calculate the contour coverage of the target node to each answer area text box, where the contour coverage is the result of dividing the outer contour of the answer area text box by the outer contour of the target node.

[0074] If the coverage rate of the target node on the outline of the answer area is greater than a preset coverage rate threshold, it is determined that the text box of the answer area is included by the target node;

[0075] Continue to traverse the child nodes of the target node, determine the last child node containing the answer area text box by covering the outer contour, and determine the last child node containing the answer area text box as the text node to which the answer area text box belongs.

[0076] In the above manner, the answer area list of each text box is determined. It may happen that the same answer area is added to the answer area list by multiple text nodes at the same time (similar to A→B, B→C, A→C). This situation will cause confusion in subsequent structured processing. It is necessary to decouple the above repeated inclusion situation. Optionally, after S1033, the method of this embodiment may further include:

[0077] For each text node, the child text node set is constructed according to the child text node list of each text node in the following way:

[0078] According to the preset traversal method, all the child text nodes in the child text node list of the text node are traversed in turn, and the visited nodes are added to the child text node set.

[0079] Deletes a child text node that exists in both the child text node list and the child text node collection from the child text node list.

[0080] S104 . Determine text structure information of the text image according to the attribute information and association information of each of the M text boxes. The text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

[0081] Specifically, generally speaking, for the layout of supplementary teaching materials, there is usually a column structure, and the questions in the same column are arranged from top to bottom. On the same page, the columns are arranged from left to right. As an implementable method, S104 may include:

[0082] S1041. Determine the column to which each text box in the N text boxes belongs based on the attribute information and associated information of each text box in the N text boxes, where the N text boxes are the text boxes in the M text boxes except for the text boxes with the answering area as the text category, and N is a positive integer.

[0083] Optionally, as an implementable manner, S1041 may include:

[0084] S10411. Obtain a target set according to the attribute information of each text box in the N text boxes, where the target set includes the upper left corner vertex of each text box in the N text boxes.

[0085] S10412. Determine the estimated width of the column according to the text box of the first level among the N text boxes.

[0086] Optionally, the median of the widths of the bounding rectangles of the first-level text boxes among the N text boxes may be used as the estimated column width. For example, if there are five first-level text boxes among the N text boxes, the median of the widths of the bounding rectangles of the five text boxes may be used as the estimated column width, that is, in descending order or descending order, the width of the bounding rectangle of the third text box may be used as the estimated column width.

[0087] S10413. Connect all upper left corner vertices in the target set in pairs to obtain multiple line segments, and calculate the inclination angle of each line segment.

[0088] S10414. Cluster the multiple line segments according to the inclination angle of each line segment, the inclination angle threshold, and the estimated column width to obtain at least one category, each category corresponds to a column, and each category includes at least one line segment.

[0089] The multiple line segments are clustered according to the inclination angle of each line segment, the inclination angle threshold, and the estimated column width to obtain at least one classification, which may specifically include:

[0090] The line segments whose inclination angle is less than a preset value in each line segment are added to the line segment set, wherein the preset value is, for example, 45 degrees.

[0091] Repeat the following steps until the segment set is empty and K categories are obtained, where K is a positive integer:

[0092] A first classification including the line segment to be clustered is constructed for a line segment to be clustered in the line segment set, and the line segment to be clustered is deleted from the line segment set.

[0093] The line segments belonging to the first category in the line segment set are determined as follows:

[0094] Determine a target line segment in the line segment set that is paired with a first line segment in the first category, wherein an endpoint of the target line segment coincides with an endpoint of a line segment in the first category;

[0095] If the target line segment meets the preset condition, it is determined that the target line segment belongs to the first category. If the target line segment does not meet the first preset condition, the target line segment is deleted from the line segment set. The preset condition is: the angle between the target line segment and the first line segment is greater than 180°-threshold θ Or less than 180°-threshold θ , threshold θ is the tilt angle threshold, for example, 30 degrees.

[0096] For K categories, two categories that meet the second preset condition are merged into one category to obtain at least one category, and the second preset condition is: the horizontal distance between the leftmost endpoint of the text box included in the first category and the leftmost endpoint of the text box included in the second category is less than q times the estimated width of the column, where q is a preset value, where q is, for example, 2 / 3.

[0097] S10415. Determine the column to which each of the N text boxes belongs based on at least one category.

[0098] Specifically, the endpoints of the line segments in each category are traversed, and the text box corresponding to the endpoint is determined as the text box in the column of the category.

[0099] S1042. Determine a first sorting result and a second sorting result according to the column to which each text box belongs and the vertical coordinate of the preset position vertex of each text box, wherein the first sorting result includes the sorting of each column, and the second sorting result includes the sorting of the text boxes of the first level among the N text boxes.

[0100] S1043. Sort the text boxes in the next-level text box list of the first-level text box according to a preset sorting rule to obtain a third sorting result.

[0101] Specifically, when sorting the text boxes in the next-level text box list of text boxes whose text categories are multiple-choice questions, fill-in-the-blank questions, and true-or-false questions, you can extract the upper left corner vertex of each text box, sort the vertex sequence from small to large according to the vertical coordinate y, and according to the sorting result, replace the vertices in the sorted list with their corresponding text boxes.

[0102] When sorting the text boxes in the next level text box list whose text category is calculation question text boxes, the upper left corner vertex of each text box can be extracted, and the vertex sequence is first threshold-sorted according to the vertical coordinate y from small to large, and then each row is sorted according to the horizontal coordinate x from small to large. According to the sorting result, the vertices in the sorted list are replaced with their corresponding text boxes.

[0103] S1044. Obtain a layout structure of M text boxes according to the first sorting result, the second sorting result, and the third sorting result.

[0104] S1045 . Obtain text structure information of the text image according to the typesetting structure of the M text boxes and the text category of each of the M text boxes.

[0105] Furthermore, if the text image includes a text box of a preset text category, and the preset text category is, for example, a fill-in-the-blank question, the method of this embodiment may further include, before S1044:

[0106] S1046. Sort the answer area list of the text boxes of the preset text category to obtain a fourth sorting result.

[0107] Optionally, sorting the answer area list of the text boxes of the preset text category to obtain a fourth sorting result may specifically include:

[0108] S10461, obtaining the minimum bounding rectangle of the text box of the text category and the inclination angle α of the minimum bounding rectangle, and the center point of the minimum bounding rectangle of each answer area in the answer area list of the text box of the preset text category;

[0109] S10462. Determine the position order of the answer areas in the same row in the answer area list according to the ascending order of the horizontal coordinates of the center points of the answer areas. The answer areas in the same row in the answer area list are determined in the following manner:

[0110] S10463: If the absolute value of the difference between the inclination angles θ and α of the line connecting the center points of the minimum circumscribed rectangle is less than a preset angle threshold, it is determined that the two answering areas are located in the same row;

[0111] S10464. Determine the position order of the answer areas located in different rows by the following method:

[0112] like and Then the answer area of ​​the mth row is determined to be above the answer area of ​​the nth row; if and Then the answer area of ​​the mth row is determined to be below the answer area of ​​the nth row, where P ml is the center point of the first answer area on the left side of the mth row, P mr is the center point of the last answer area on the right side of the mth row, P nl is the center point of the first answer area on the left side of the nth row, P nr It is the center point of the last answer area on the right end of the nth row.

[0113] S10465. Obtain a fourth sorting result based on the position sorting of the answer areas located in the same row and the position sorting of the answer areas located in different rows in the answer area list.

[0114] Correspondingly, S1045 may be: obtaining a layout structure of M text boxes according to the first sorting result, the second sorting result, the third sorting result, and the fourth sorting result.

[0115] Furthermore, the method of this embodiment may also include:

[0116] S105: Output text structure information of the text image.

[0117] Specifically, when outputting the text structure information of the text image, the information of each text box in the N text boxes can be output according to the typesetting structure of the M text boxes. The information of each text box includes the outer contour of the text box (specifically, the contour points of the outer contour of the text box and the polygonal slices cut out in the text image according to the outer contour of the text box can be output in sequence), the text category, the outer contour of the answer area of ​​the text box (specifically, the contour points of the outer contour of the answer area and the polygonal slices cut out in the text image according to the outer contour of the answer area can be output in sequence), and the information of each text box in the next-level text box list of the text box. It can be understood that the information of each text box in the next-level text box list of the text box can also include the outer contour of the text box, the text category, the outer contour of the answer area of ​​the text box and the information of each text box in the next-level text box list of the text box (if there are only two levels of text boxes of one text category, there is no information of each text box in the next-level text box list here).

[0118] Furthermore, in an implementable manner, the method of this embodiment may also include:

[0119] S106 . Perform text recognition on the text image according to the text structure information of the text image to obtain text recognition results of M text boxes.

[0120] S107 . Obtain a recognition result of the text image according to the text structure information of the text image and the text recognition results of the M text boxes.

[0121] S108: Output the recognition result of the text image.

[0122] The text recognition of the text image can be performed by using existing text recognition methods, such as OCR recognition, or other text recognition methods, which are not limited in this embodiment. The recognition result of the text image output in S108 includes both the text structure information of the text image and the text recognition result of each text box.

[0123] The text image recognition method provided in this embodiment obtains the attribute information of each of the M text boxes of the text image according to the pre-trained detection model, determines the associated information of each of the M text boxes according to the attribute information of the M text boxes, and determines the text structure information of the text image according to the attribute information and associated information of each of the M text boxes. Since the text structure information of the text image is determined according to the attribute information and associated information of the text box, the attribute information and associated information of the text box can still be accurately obtained when the text image is tilted, distorted or perspective, and there is no need to perform text recognition to obtain semantic information. Therefore, the text structure information of the text image can be accurately identified, and compared with the template matching method, the applicability of text image recognition is also improved.

[0124] The following is a detailed description of the text image recognition method provided in the embodiment of the present application in conjunction with a specific embodiment.

[0125] Figure 4 This is a flowchart of a text image recognition method provided in an embodiment of the present application. The text image recognition method of this embodiment can be performed by a text image recognition device, which can be implemented in software and / or hardware. The text image recognition device can be an electronic device or a chip or circuit of an electronic device. Figure 4 As shown, the method of this embodiment includes the following steps:

[0126] S201. Acquire a text image, where the text image includes M text boxes.

[0127] Specifically, the text image may be acquired by the electronic device directly photographing the text image, or by receiving the text image sent by other devices.

[0128] S202: Input the text image into a pre-trained detection model, and output original attribute information of M text boxes, where the original attribute information of each text box includes a bounding rectangle of the text box, a text category of the text box, and a segmentation mask corresponding to the text box.

[0129] Among them, the bounding rectangle of the text box is a tuple composed of (x1, y1, x2, y2), where (x1, y1) and (x2, y2) are the coordinates of the upper left corner and the lower right corner of the bounding rectangle of the text box respectively. The text category of the text box is the question type classification corresponding to the text in the text box, which can be an enumeration variable composed of all question types in the teaching materials. The text category of the text box can include multiple-choice questions, true-or-false questions, fill-in-the-blank questions, calculation questions, answer questions, compound questions and answer areas, etc. is the confidence corresponding to the text category of the text box. For example, if the text category of a text box is a multiple-choice question, this is the confidence (also probability) that the text category of the text box is a multiple-choice question, and the confidence can be a value between 0-1. The segmentation mask corresponding to the text box can be a 28*28 matrix, and each element in the matrix is ​​the confidence of a pixel point, which can be a value between 0-1.

[0130] S203 : Obtain the outer contour of each text box according to the circumscribed rectangle of each text box and the segmentation mask corresponding to each text box.

[0131] Specifically, taking a text box as an example, the segmentation mask corresponding to the text box is adjusted to a first segmentation mask of the same size as the circumscribed rectangle of the text box, and the value of each element in the first segmentation mask is set, specifically, according to a preset threshold (such as 0.5), the value of the element whose confidence is greater than or equal to the preset threshold is modified to 1, and the value of the element whose confidence is less than or equal to the preset threshold is modified to 0, to obtain a second segmentation mask, and then, the contour extraction algorithm is used to extract the outer contour of the text box from the second segmentation mask. At this time, the coordinate points of the outer contour of the text box are the coordinates relative to the circumscribed rectangle of the text box. The offset (x1, y1) is added to all points in the outer contour of the extracted text box, and the outer contour of the text box in absolute coordinates can be obtained, and the outer contour of the text box in absolute coordinates is determined as the outer contour of the text box.

[0132] S204, extracting text boxes whose text categories are answering areas from the M text boxes, and storing attribute information of the extracted text boxes whose text categories are answering areas into the answering area text box set.

[0133] The attribute information of the text box whose text category is the answer area includes the bounding rectangle of the text box and the outer contour of the text box. For example, Table 1 shows an example of an answer area text box set:

[0134] Table 1 Answer area text box collection

[0135] Answer area logo Attribute information of the answer area Answer area text box 1 Attribute information of answer area text box 1 Answer area text box 2 Attribute information of answer area text box 2 Answer area text box Attribute information of the answer area text box n

[0136] As shown in Table 1, there are n answer areas, and the answer area text box set corresponds to the attribute information of the n answer areas. When an answer area is stored in the answer area text box set, an answer area identifier is assigned to the answer area.

[0137] It can be understood that after extracting the text boxes whose text categories are answer areas, the remaining text boxes are text boxes for various questions.

[0138] S205. Determine association information of each of the N text boxes according to attribute information of the N text boxes except the answer area among the M text boxes, the association information including the level of the text box, the list of next-level text boxes of the text box, and the list of answer areas of the text box.

[0139] Specifically, since there is an inclusive relationship between large and small questions, or there are multiple levels of nested self-questions, in this embodiment, a multi-branch tree can be used to summarize the structural information of the questions. As an implementable method, S205 can specifically include:

[0140] S2051. Construct a text node according to the attribute information of each of the N text boxes except the answer area among the M text boxes. The attribute information of each text node includes a node identifier, a circumscribed rectangle, a text category, and an outer contour.

[0141] The bounding rectangle, text category and outer contour of a text node are the bounding rectangle, text category and outer contour of the corresponding text box, and the node identifier is an identifier assigned to the text node when constructing the text node.

[0142] S2052: Determine a text node set according to the attribute information of each text node and the text category, wherein the text node set includes the text category and a text node list of the text category.

[0143] For example, the text image includes three text categories, namely, multiple choice questions, true or false questions, and fill-in-the-blank questions. There are 6 text nodes whose text category is multiple choice questions, 4 text nodes whose text category is true or false questions, and 3 text nodes whose text category is fill-in-the-blank questions. Then the text node set of the determined text image can be {multiple choice questions: text node 1, text node 2, text node 3, text node 5, text node 6, and text node 7; true or false questions: text node 4, text node 8, text node 10, and text node 12; fill-in-the-blank questions: text node 9, text node 11, text node 13}.

[0144] S2053. Determine the main text node and the sub-text node list of the main text node in the text node list of each text category based on the text node set, determine the level of the text box corresponding to the determined main text node as the first level, and determine the next-level text box list of the text box corresponding to each main text node based on the sub-text node list of each main text node.

[0145] Among them, the specific implementation of S2053 can refer to the specific description in S1033, which will not be repeated here.

[0146] S2054: Construct a text tree according to the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set.

[0147] Specifically, as an implementable method, the text image can be used as the root node, the main text node in the text node list of all text categories in the text node set can be used as the child node of the root node, and the child text node of each main text node can be used as the child node of the main text node.

[0148] Still taking the text node set of the text image as {multiple choice question: text node 1, text node 2, text node 3, text node 5, text node 6 and text node 7; true or false question: text node 4, text node 8, text node 10 and text node 12; fill-in-the-blank question: text node 9, text node 11, text node 13} as an example, for example, the main text node and the sub-text node list of the main text node in the text node list of each text category determined by S20512 are shown in Table 2 below:

[0149] Table 2 Main text node and sub-text node list of the main text node in the text node list of each text category in the text node set

[0150]

[0151] The text tree constructed based on the main text node and the sub-text node list of the main text node in the text node list of each text category in the text node set shown in Table 2 is as follows: Figure 5 As shown, Figure 5Schematic diagram of the constructed text tree. Specifically, the text image is taken as the root node, text node 1, text node 4 and text node 9 are respectively used as the three child nodes of the root node, the child text nodes of text node 1 (such as text node 2, text node 3, text node 5, text node 6 and text node 7 in Table 2) are used as the child nodes of text node 1, the child text nodes of text node 4 (such as text node 8, text node 10 and text node 12 in Table 2) are used as the child nodes of text node 4, and the child text nodes of text node 9 (such as text node 11 and text node 13 in Table 2) are used as the child nodes of text node 9, and the construction is obtained. Figure 5 The text tree shown.

[0152] S2055. Determine the text node to which each answer area text box in the answer area text box set belongs by covering the outer contour according to the text tree, determine the answer area list of each text node according to the text node to which each answer area text box belongs, and determine the answer area list of each text node as the answer area list of the text box corresponding to each text node.

[0153] Specifically, one answer area text box is taken out from the answer area text box set in turn, and the text node to which the answer area text box belongs is determined according to the following method, until the text nodes to which all answer area text boxes in the answer area text box set belong are determined.

[0154] Optionally, the text node to which a text box in the answer area belongs can be determined in the following manner:

[0155] S1. Traverse all child nodes of the root node in the text tree in turn, and take the currently visited target node, which is a child node of the root node.

[0156] S2. Calculate the contour coverage of the target node to the text box in the answer area.

[0157] The contour coverage is the result of dividing the outer contour of the text box in the answer area by the outer contour of the target node.

[0158] S3. If the outline coverage of the target node on the answer area text box is greater than a preset coverage threshold, it is determined that the answer area text box is included in the target node, and the child nodes of the target node are continued to be traversed.

[0159] S4, repeat S2-S3 until no child node of the target node is found that includes the answer area text box, and the last child node that includes the answer area text box is determined as the text node to which the answer area text box belongs.

[0160] Through S1-S4, the text node to which each answer area text box in the answer area text box set belongs is obtained. According to the text node to which each answer area text box belongs, the answer area list of each text node can be determined. The answer area list of each text node is the answer area list of the text box corresponding to each text node.

[0161] Through the above method, it may happen that the same answer area is added to the answer area list by multiple text nodes at the same time (similar to A→B, B→C, A→C), which will cause confusion in subsequent structured processing. It is necessary to decouple the above repeated inclusion. Optionally, the method of this embodiment may also include:

[0162] S11, for each text node, according to each text node's child text node list child_list = [note 0 …note j ] Construct the subtext node set S in the following way i :

[0163] According to the preset traversal method (for example, it can be a traversal method such as depth-limited, breadth-first, etc., or other traversal methods), all child text nodes in the child text node list child_list of the text node are traversed accordingly, and the visited nodes are added to the child text node set S i middle.

[0164] S12, the child text node list child_list and the child text node set S i The nodes in are deleted from the child text node list child_list.

[0165] Generally, for the layout of supplementary teaching materials, there is usually a column structure, and the questions in the same column are arranged from top to bottom. On the same page, the columns are arranged from left to right. The method of this embodiment may also include:

[0166] S206 , determining text structure information of the text image according to the attribute information and association information of each text box in the N text boxes, where the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

[0167] Specifically, as an implementable manner, S206 may include:

[0168] S2061. Obtain a target set according to the attribute information of each text box in the N text boxes, where the target set includes the upper left corner vertex of each text box in the N text boxes.

[0169] S2062. Determine an estimated column width according to a text box at the first level among the N text boxes.

[0170] Optionally, the median of the widths of the bounding rectangles of the first-level text boxes among the N text boxes may be used as the estimated column width. For example, if there are five first-level text boxes among the N text boxes, the median of the widths of the bounding rectangles of the five text boxes may be used as the estimated column width, that is, in descending order or descending order, the width of the bounding rectangle of the third text box may be used as the estimated column width.

[0171] S2063. Connect all upper left corner vertices in the target set in pairs to obtain multiple line segments, and calculate the inclination angle θ of each line segment.

[0172] S2064. Cluster the multiple line segments according to the inclination angle of each line segment, the inclination angle threshold, and the estimated column width to obtain at least one category, each category corresponds to a column, and each category includes at least one line segment.

[0173] Specifically, the clustering process in S2064 can be seen in Figure 2 The description in the illustrated embodiment will not be repeated here.

[0174] S2065. Determine the column to which each of the N text boxes belongs based on at least one category.

[0175] Specifically, the endpoints of the line segments in each category are traversed, and the text box corresponding to the endpoint is determined as the text box in the column of the category.

[0176] S2066. Determine a first sorting result and a second sorting result according to the column to which each text box belongs and the vertical coordinate of the preset position vertex of each text box, wherein the first sorting result includes the sorting of each column, and the second sorting result includes the sorting of the text boxes of the first level among the N text boxes.

[0177] S2067. Sort the text boxes in the next-level text box list of the first-level text box according to a preset sorting rule to obtain a third sorting result.

[0178] Specifically, when sorting the text boxes in the next-level text box list of text boxes whose text categories are multiple-choice questions, fill-in-the-blank questions, and true-or-false questions, you can extract the upper left corner vertex of each text box, sort the vertex sequence from small to large according to the vertical coordinate y, and according to the sorting result, replace the vertices in the sorted list with their corresponding text boxes.

[0179] When sorting the text boxes in the next level text box list whose text category is the calculation question text box, the upper left corner vertex of each text box can be extracted, and the vertex sequence is first threshold sorted according to the vertical coordinate y from small to large, and then the horizontal coordinate x is sorted from small to large for each row. According to the sorting result, the vertices in the sorted list are replaced with their corresponding text boxes. The threshold sorting means that if the absolute value of the difference between the y coordinate values ​​of two vertices is less than the preset threshold, the two vertices are determined to be in the same row. Otherwise, they are not in the same row.

[0180] If the text image includes a text box of a preset text category, such as a fill-in-the-blank question, the method of this embodiment may further include:

[0181] S2068. Sort the answer area list of the text boxes of the preset text category to obtain a fourth sorting result.

[0182] Specifically, due to the inclination of the text image and the fact that the answer area of ​​the fill-in-the-blank question is often narrow in height, the answer area cannot be simply sorted in the order of top to bottom and left to right.

[0183] For a text box with a fill-in-the-blank question as the text category, there are multiple answer areas, there may be multiple answer areas in one row, or there may be multiple rows, each row has multiple answer areas. For a text box with a fill-in-the-blank question as the text category, the specific process of S2068 can be found in Figure 2 The description of S10461-S10465 in the illustrated embodiment will not be repeated here.

[0184] S2069. Obtain a layout structure of M text boxes according to the first sorting result, the second sorting result, the third sorting result, and the fourth sorting result.

[0185] S207: Output text structure information of the text image.

[0186] Optionally, after S207, the following may also be included:

[0187] Text recognition is performed on the text image according to the text structure information of the text image to obtain text recognition results of M text boxes, and the text recognition results of the M text boxes are output according to the typesetting structure of the M text boxes and the text category of each text box.

[0188] In this embodiment, when outputting the text recognition results of M text boxes according to the typesetting structure of the M text boxes and the text category of each text box, the text recognition results of the text boxes and the text recognition results of the answer area can also be output separately according to the text category. For example, for a question, the question and the answer to the question can be output separately, which is convenient for query, analysis and correction.

[0189] The following is an embodiment of the device of the present application, which can be used to execute the above method embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, reference can be made to the above method embodiment of the present application.

[0190] Figure 6 A structural diagram of a text image recognition device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device of this embodiment may include: a first acquisition module 11, a second acquisition module 12, a first determination module 13 and a second determination module 14, wherein:

[0191] The first acquisition module 11 is used to acquire a text image, where the text image includes M text boxes, where M is a positive integer;

[0192] The second acquisition module 12 is used to acquire attribute information of each text box in the M text boxes according to the pre-trained detection model;

[0193] The first determination module 13 is used to determine the associated information of each text box in the M text boxes according to the attribute information of the M text boxes;

[0194] The second determination module 14 is used to determine text structure information of the text image according to the attribute information and association information of each text box in the M text boxes, where the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

[0195] Optionally, the first determining module 13 is used to:

[0196] Construct a text node according to the attribute information of each text box in N text boxes, respectively, the N text boxes are the text boxes in M ​​text boxes except the text box with the text category of answering area, and N is a positive integer;

[0197] According to the attribute information of each text node, a text node set is determined according to the text category, and the text node set includes the text category and a text node list of the text category;

[0198] Determine a main text node and a sub-text node list of the main text node in a text node list of each text category according to the text node set, determine the level of a text box corresponding to the main text node as the first level, and determine a next-level text box list of a text box corresponding to each main text node according to the sub-text node list of each main text node;

[0199] The answer area list of the text box corresponding to each text node is determined based on the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set and the answer area text box set. The answer area text box set includes the answer area text box identifier and the attribute information of the answer area text box.

[0200] Optionally, the second acquisition module 12 is further used for:

[0201] The text boxes whose text categories are answering areas are extracted from the M text boxes, and the attribute information of the text boxes whose text categories are answering areas is stored in the answering area text box set.

[0202] Optionally, the first determining module 13 is specifically configured to:

[0203] Construct a text tree according to the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set;

[0204] The text node to which each answer area text box in the answer area text box set belongs is determined according to the text tree by covering the outer contour, the answer area list of each text node is determined according to the text node to which each answer area text box belongs, and the answer area list of each text node is determined as the answer area list of the text box corresponding to each text node.

[0205] Optionally, the first determining module 13 is specifically configured to:

[0206] For each answer area text box in the answer area text box set, all child nodes of the root node in the text tree are traversed in turn, and the currently visited target node is obtained, which is a child node of the root node;

[0207] The answer area text box is determined to be included in the target node by covering the outer contour. The outer contour covering method is:

[0208] Calculate the contour coverage of the target node to each text box in the answer area;

[0209] If the coverage rate of the target node on the outline of the answer area is greater than a preset coverage rate threshold, it is determined that the text box of the answer area is included by the target node;

[0210] Continue to traverse the child nodes of the target node, determine the last child node containing the answer area text box by covering the outer contour, and determine the last child node containing the answer area text box as the text node to which the answer area text box belongs.

[0211] Optionally, the first determining module 13 is further configured to:

[0212] For each text node, the child text node set is constructed according to the child text node list of each text node in the following way:

[0213] According to the preset traversal method, all the child text nodes in the child text node list of the text node are traversed in turn, and the visited nodes are added to the child text node set;

[0214] Deletes a child text node that exists in both the child text node list and the child text node collection from the child text node list.

[0215] Optionally, the second determining module 14 is used to:

[0216] According to the attribute information and the associated information of each text box in the N text boxes, the column to which each text box in the N text boxes belongs is determined, the N text boxes are the text boxes in the M text boxes except the text boxes with the answering area as the text category, and N is a positive integer;

[0217] Determine a first sorting result and a second sorting result according to the column to which each text box belongs and the ordinate of the preset position vertex of each text box, wherein the first sorting result includes the sorting of each column, and the second sorting result includes the sorting of the text boxes of the first level among the N text boxes;

[0218] Sort the text boxes in the next level text box list of the first level text box according to the preset sorting rule to obtain a third sorting result;

[0219] According to the first sorting result, the second sorting result and the third sorting result, a typesetting structure of M text boxes is obtained;

[0220] According to the typesetting structure of the M text boxes and the text category of each of the M text boxes, the text structure information of the text image is obtained.

[0221] Optionally, if the text image includes a text box of a preset text category, the second determining module 14 is further configured to:

[0222] Sorting the answer area list of the text boxes of the preset text category to obtain a fourth sorting result;

[0223] The second determining module 14 is used to obtain the layout structure of M text boxes according to the first sorting result, the second sorting result, the third sorting result and the fourth sorting result.

[0224] Optionally, the second determining module 14 is specifically configured to:

[0225] Obtaining the minimum bounding rectangle of the text box of the preset text category and the inclination angle α of the minimum bounding rectangle, and the center point of the minimum bounding rectangle of each answer area in the answer area list of the text box of the preset text category;

[0226] The positions of the answer areas in the same row in the answer area list are determined in ascending order of the horizontal coordinates of the center points of the answer areas. The answer areas in the same row in the answer area list are determined in the following manner:

[0227] If the absolute value of the difference between the inclination angles θ and α of the line connecting the center points of the minimum circumscribed rectangles of the two answering areas is less than a preset angle threshold, it is determined that the two answering areas are located in the same row;

[0228] The position order of the answer areas in different rows is determined as follows:

[0229] like and Then the answer area of ​​the mth row is determined to be above the answer area of ​​the nth row; if and Then the answer area of ​​the mth row is determined to be below the answer area of ​​the nth row, where P ml is the center point of the first answer area on the left side of the mth row, P mr is the center point of the last answer area on the right side of the mth row, P nl is the center point of the first answer area on the left side of the nth row, P nr The center point of the last answer area on the right side of the nth row;

[0230] A fourth sorting result is obtained according to the position sorting of the answer areas located in the same row and the position sorting of the answer areas located in different rows in the answer area list.

[0231] Optionally, the second determining module 14 is used to:

[0232] Acquire a target set according to the attribute information of each text box in the N text boxes, where the target set includes the upper left corner vertex of each text box in the N text boxes;

[0233] Determine the estimated width of the column according to the text box at the first level among the N text boxes;

[0234] Connect all the upper left corner vertices in the target set in pairs to obtain multiple line segments, and calculate the inclination angle of each line segment;

[0235] Clustering the multiple line segments according to the inclination angle of each line segment, the inclination angle threshold and the estimated column width to obtain at least one classification, each classification corresponds to a column, and each classification includes at least one line segment;

[0236] According to at least one category, determine the column to which each text box in the N text boxes belongs.

[0237] Optionally, the second determining module 14 is used to:

[0238] The line segments whose inclination angle is less than a preset value in each line segment are added to the line segment set, wherein the preset value is, for example, 45 degrees.

[0239] Repeat the following steps until the segment set is empty and K categories are obtained, where K is a positive integer:

[0240] A first classification including the line segment to be clustered is constructed for a line segment to be clustered in the line segment set, and the line segment to be clustered is deleted from the line segment set.

[0241] The line segments belonging to the first category in the line segment set are determined as follows:

[0242] Determine a target line segment in the line segment set that is paired with a first line segment in the first category, wherein an endpoint of the target line segment coincides with an endpoint of a line segment in the first category;

[0243] If the target line segment meets the preset condition, it is determined that the target line segment belongs to the first category. If the target line segment does not meet the first preset condition, the target line segment is deleted from the line segment set. The preset condition is: the angle between the target line segment and the first line segment is greater than 180°-threshold θ Or less than 180°-threshold θ , threshold θ is the tilt angle threshold.

[0244] For K categories, two categories that meet the second preset condition are merged into one category to obtain at least one category, and the second preset condition is: the horizontal distance between the leftmost endpoint of the text box included in the first category and the leftmost endpoint of the text box included in the second category is less than q times the estimated width of the column, where q is a preset value, where q is, for example, 2 / 3.

[0245] The device provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principle and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.

[0246] Figure 7 A structural diagram of a text image recognition device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device of this embodiment is Figure 6 On the basis of the device shown, further, it can also include:

[0247] The processing module 15 is used for:

[0248] Output text structure information of text images.

[0249] Optionally, the processing module 15 is used to perform text recognition on the text image according to the text structure information of the text image to obtain text recognition results of M text boxes; obtain recognition results of the text image according to the text structure information of the text image and the text recognition results of the M text boxes; and output the recognition results of the text image.

[0250] The device provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principle and technical effects can be found in the above method embodiment, and this embodiment will not be repeated here.

[0251] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 6 or Figure 7 The text image recognition device shown can perform Figure 2 The corresponding method embodiments, and the aforementioned and other operations and / or functions of each module in the text image recognition device are respectively to achieve Figure 2 For the sake of brevity, the corresponding processes in each method are not repeated here.

[0252] The text image recognition device of the embodiment of the present application is described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0253] Figure 8 It is a schematic block diagram of an electronic device 200 provided in an embodiment of the present application.

[0254] like Figure 8 As shown, the electronic device 200 may include:

[0255] The memory 210 and the processor 220, the memory 210 is used to store the computer program and transmit the program code to the processor 220. In other words, the processor 220 can call and run the computer program from the memory 210 to implement the method in the embodiment of the present application.

[0256] For example, the processor 220 may be configured to execute the above method embodiments according to instructions in the computer program.

[0257] In some embodiments of the present application, the processor 220 may include but is not limited to:

[0258] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0259] In some embodiments of the present application, the memory 210 includes but is not limited to:

[0260] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0261] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 210 and executed by the processor 220 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0262] like Figure 8 As shown, the electronic device may also include:

[0263] The transceiver 230 may be connected to the processor 220 or the memory 210 .

[0264] The processor 220 may control the transceiver 230 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 230 may include a transmitter and a receiver. The transceiver 230 may further include an antenna, and the number of antennas may be one or more.

[0265] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0266] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0267] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0268] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0269] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0270] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0271] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A text image recognition method, characterized in that: include: Acquire a text image, wherein the text image includes M text boxes, where M is a positive integer; According to the pre-trained detection model, the attribute information of each text box in the M text boxes is obtained, wherein the attribute information of the text box includes at least one of a circumscribed rectangle of the text box, a text category of the text box, and an outer contour of the text box, and the text category of the text box is a question type classification corresponding to the text in the text box; Determine, according to the attribute information of the M text boxes, association information of each text box in the M text boxes, wherein the association information of the text boxes includes at least one of the level of the text box, a list of text boxes at a lower level of the text box, and a list of answer areas of the text box; According to the attribute information and the association information of each text box in the M text boxes, text structure information of the text image is determined, and the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

2. The method according to claim 1, characterized in that The method further comprises: Output text structure information of the text image.

3. The method according to claim 1, characterized in that The method further comprises: Performing text recognition on the text image according to the text structure information of the text image to obtain text recognition results of the M text frames; Obtaining a recognition result of the text image according to the text structure information of the text image and the text recognition results of the M text boxes; The recognition result of the text image is output.

4. The method according to claim 1, characterized in that: The determining, according to the attribute information of the M text boxes, the associated information of each text box of the M text boxes comprises: Constructing a text node according to the attribute information of each text box in N text boxes, respectively, wherein the N text boxes are text boxes other than the text boxes with the text category of the answering area in the M text boxes, and N is a positive integer; According to the attribute information of each text node, a text node set is determined according to the text category, and the text node set includes the text category and a text node list of the text category; Determine a main text node and a sub-text node list of the main text node in a text node list of each text category according to the text node set, determine the level of a text box corresponding to the main text node as a first level, and determine a next-level text box list of a text box corresponding to each main text node according to the sub-text node list of each main text node; According to the main text nodes and the sub-text node lists of the main text nodes in the text node lists of all text categories in the text node set and the answer area text box set, the answer area list of the text box corresponding to each text node is determined.

5. The method according to claim 1, characterized in that The method further comprises: A text box whose text category is the answer area is extracted from the M text boxes, and attribute information of the text box whose text category is the answer area is stored in the answer area text box set.

6. The method according to claim 4, characterized in that Determining the answer area list of the text box corresponding to each text node according to the main text node and the sub-text node list of the main text node and the answer area text box set in the text node list of all text categories in the text node set includes: Constructing a text tree according to the main text node and the sub-text node list of the main text node in the text node list of all text categories in the text node set; The text node to which each answer area text box in the answer area text box set belongs is determined by covering the outer contour according to the text tree, the answer area list of each text node is determined according to the text node to which each answer area text box belongs, and the answer area list of each text node is determined as the answer area list of the text box corresponding to each text node.

7. The method according to claim 6, characterized in that Determining the text node to which each answer area text box in the answer area text box set belongs by covering the text tree with an outer contour includes: For each answer area text box in the answer area text box set, all child nodes of the root node in the text tree are traversed in turn, and the currently visited target node is obtained, and the target node is a child node of the root node; The answer area text box is determined to be included in the target node by the outer contour covering method, and the outer contour covering method is: Calculating the contour coverage of the target node to each text box in the answer area; If the coverage rate of the target node on the outline of the answer area is greater than a preset coverage rate threshold, it is determined that the text box of the answer area is included by the target node; Continue to traverse the child nodes of the target node, determine the child node that finally contains the answer area text box by the outer contour covering method, and determine the child node that finally contains the answer area text box as the text node to which the answer area text box belongs.

8. The method according to claim 4, characterized in that After determining the main text node and the sub-text node list of the main text node in the text node list of each text category according to the text node set, the method further includes: For each text node, a child text node set is constructed according to the child text node list of each text node in the following manner: According to a preset traversal method, all child text nodes in the child text node list of the text node are traversed in turn, and the visited nodes are added to the child text node set; The child text nodes existing in both the child text node list and the child text node set are deleted from the child text node list.

9. The method according to claim 1, characterized in that: The determining of the text structure information of the text image according to the attribute information and the association information of each text box in the M text boxes includes: Determine the column to which each text box in the N text boxes belongs according to the attribute information and the association information of each text box in the N text boxes, wherein the N text boxes are text boxes other than the text box with the text category of the answering area in the M text boxes, and N is a positive integer; Determine a first sorting result and a second sorting result according to the column to which each text box belongs and the ordinate of the preset position vertex of each text box, wherein the first sorting result includes the sorting of each column, and the second sorting result includes the sorting of the text boxes of the first level among the N text boxes; Sorting the text boxes in the next-level text box list of the first-level text box according to a preset sorting rule to obtain a third sorting result; Obtaining a layout structure of the M text boxes according to the first sorting result, the second sorting result, and the third sorting result; The text structure information of the text image is obtained according to the typesetting structure of the M text boxes and the text category of each text box in the M text boxes.

10. The method according to claim 9, characterized in that If the text image includes a text box of a preset text category, the method further includes: Sorting the answer area list of the text boxes of the preset text category to obtain a fourth sorting result; The obtaining, according to the first sorting result, the second sorting result and the third sorting result, the typesetting structure of the M text boxes comprises: The layout structure of the M text boxes is obtained according to the first sorting result, the second sorting result, the third sorting result and the fourth sorting result.

11. The method according to claim 9, characterized in that The determining, according to the attribute information and the association information of each text box in the N text boxes, the column to which each text box in the N text boxes belongs includes: Acquire a target set according to the attribute information of each text box in the N text boxes, the target set including the upper left corner vertex of each text box in the N text boxes; Determine the estimated column width according to the text box of the first level among the N text boxes; Connect all upper left corner vertices in the target set in pairs to obtain multiple line segments, and calculate the inclination angle of each line segment; Clustering the plurality of line segments according to the inclination angle of each line segment, the inclination angle threshold and the estimated width of the column to obtain at least one classification, each of the classifications corresponds to a column, and each of the classifications includes at least one line segment; According to the at least one classification, the column to which each of the N text boxes belongs is determined.

12. A text image recognition device, characterized in that: include: A first acquisition module is used to acquire a text image, wherein the text image includes M text boxes, where M is a positive integer; A second acquisition module is used to acquire attribute information of each of the M text boxes according to a pre-trained detection model, wherein the attribute information of the text box includes at least one of a circumscribed rectangle of the text box, a text category of the text box, and an outer contour of the text box, and the text category of the text box is a question type classification corresponding to the text in the text box; A first determination module is used to determine the associated information of each of the M text boxes according to the attribute information of the M text boxes, wherein the associated information of the text boxes includes at least one of the level of the text box, a list of next-level text boxes of the text box, and a list of answer areas of the text box; The second determination module is used to determine the text structure information of the text image according to the attribute information and the association information of each text box in the M text boxes, and the text structure information includes the typesetting structure of the M text boxes and the text category of each text box.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the text image recognition method according to any one of claims 1 to 11 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text image recognition method according to any one of claims 1 to 11 is implemented.

Images