Text content extraction method and apparatus

Abstract

The present specification provides a text content extraction method and device, wherein the text content extraction method comprises: performing character recognition processing on a target image, and determining a mark image containing character labels according to a processing result; inputting the mark image containing the character labels into an image processing model for processing, and obtaining a reference image containing predicted lines output by the image processing model; determining mark information corresponding to the character labels in the mark image and line information corresponding to the predicted lines in the reference image; calculating connection parameters according to the mark information and the line information, and constructing a target relationship graph corresponding to the target image based on the connection parameters; and extracting text content in the target image by using the target relationship graph, thereby improving the text completeness degree of text content extraction in the target image, and improving the efficiency and accuracy of text extraction.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method for extracting text content. This specification also relates to a text content extraction apparatus, a computing device, and a computer-readable storage medium. Background Technology

[0002] With the development of computer technology and the arrival of the big data era, more and more data is stored electronically, that is, data is entered into computers and stored in databases for easy retrieval and backup. Currently, data entry for paper forms is usually done manually, with staff reading the forms and entering the data into the computer. However, this method consumes a lot of human resources. Due to limited human energy, incomplete data entry is inevitable, and manual entry is inefficient and inaccurate. Therefore, there is an urgent need for an effective text extraction method to solve these problems. Summary of the Invention

[0003] In view of this, embodiments of this specification provide a method for extracting text content. This specification also relates to a text content extraction apparatus, a computing device, and a computer-readable storage medium, to address the technical deficiencies existing in the prior art.

[0004] According to a first aspect of the embodiments of this specification, a method for extracting text content is provided, comprising:

[0005] Perform text recognition processing on the target image, and determine the marker image containing text identifiers based on the processing results;

[0006] The marked image containing the text identifier is input into the image processing model for processing to obtain a reference image containing predicted lines output by the image processing model;

[0007] Determine the marker information corresponding to the text identifier in the marker image, and the line information corresponding to the predicted line in the reference image;

[0008] Calculate connection parameters based on the marker information and the line information, and construct a target relationship graph corresponding to the target image based on the connection parameters;

[0009] The text content is extracted from the target image using the target relationship graph.

[0010] Optionally, the step of performing text recognition processing on the target image and determining a marker image containing text identifiers based on the processing result includes:

[0011] The text entities in the target image are subjected to text recognition processing, and the text box and center point corresponding to each text entity in the target image are determined based on the processing results;

[0012] Based on the text box and center point corresponding to each text entity in the target image, a marker image containing the text identifier is generated.

[0013] Optionally, generating a marker image containing the text identifier based on the text box and center point corresponding to each text entity in the target image includes:

[0014] Semantic recognition processing is performed on the text entities in the target image, and the entity category information corresponding to each text entity in the target image is determined based on the processing results;

[0015] Based on the entity category information of each text entity in the target image, determine the attribute information of the text box corresponding to the text entity in the target image, and determine the attribute information of the target image;

[0016] Use the text box and center point corresponding to each text entity as the initial text identifier;

[0017] The initial text identifier is updated according to the attribute information of the target image and the attribute information of the text box, and a marker image containing the text identifier is generated based on the update result.

[0018] Optionally, the step of inputting the marker image containing the text identifier into an image processing model for processing to obtain a reference image containing predicted lines output by the image processing model includes:

[0019] The marker image containing the text identifier is converted into an initial marker image of a first preset dimension;

[0020] The initial labeled image is processed by network layers and convolutional layers in the image processing model, and a second preset dimension reference image containing predicted lines is determined based on the processing results.

[0021] Optionally, determining the marker information corresponding to the text identifier in the marker image and the line information corresponding to the predicted line in the reference image includes:

[0022] Multiple text boxes are determined based on the text identifiers contained in the marked image;

[0023] Multiple text boxes are grouped together, and multiple text box pairs are determined based on the grouping result;

[0024] The determination of the marking information for any text identifier includes:

[0025] The first position information of the first text box and the second position information of the second text box contained in the text box pair are used as the marking information of the text identifier.

[0026] Optionally, determining any one of the sub-connection parameters includes:

[0027] Determine the associated marker information and associated line information that have a relationship;

[0028] The first associated position parameter and the second associated position parameter are determined based on the associated marker information;

[0029] Calculate a reference position parameter based on the first associated position parameter and the second associated parameter, and use the reference position parameter as a reference value;

[0030] The line pixel values ​​corresponding to the line information are determined based on the reference image;

[0031] The sub-connection parameters are calculated based on the reference values ​​and the line pixel values.

[0032] Optionally, constructing the target relationship graph corresponding to the target image based on the connection parameters includes:

[0033] Each sub-connection parameter in the connection parameters is compared with a preset parameter threshold, and the target sub-connection parameter is determined based on the comparison result.

[0034] The target text box is determined in the marking diagram based on the target sub-connection parameters;

[0035] Extract the target text boxes, connect the target text boxes, and generate a target relationship graph based on the connection results.

[0036] Optionally, generating the target relationship graph based on the connection results includes:

[0037] Generate an initial relationship graph corresponding to the target image based on the connection results;

[0038] The target relationship is determined based on the neural network model in the initial relationship graph;

[0039] Based on the target relationship, the initial relationship graph is filtered, and the relationship to be deleted is determined according to the filtering results.

[0040] The relationships to be deleted are deleted in the initial relationship graph, and a target relationship graph is generated based on the processing results.

[0041] Optionally, extracting text content from the target image using the target relationship graph includes:

[0042] Entity pair construction is performed on two entity points with a connection relationship in the target relationship graph, and multiple entity pairs are determined based on the processing results.

[0043] The text content corresponding to each entity pair in the target image is determined and extracted.

[0044] Optionally, the step of performing text recognition processing on the text entities in the target image, and determining the text box and center point corresponding to each text entity in the target image based on the processing result, includes:

[0045] The text entities in the target image are subjected to text recognition processing, and the entity information corresponding to each text entity in the target image is determined based on the processing result;

[0046] Based on the entity information of each text entity in the target image, a text box and center point corresponding to the text entity in the target image are generated.

[0047] According to a second aspect of the embodiments of this specification, a text content extraction apparatus is provided, comprising:

[0048] The processing module is configured to perform text recognition processing on the target image and determine a marker image containing text identifiers based on the processing results;

[0049] The input module is configured to input a marker image containing the text identifier into an image processing model for processing, and obtain a reference image containing predicted lines output by the image processing model;

[0050] The determination module is configured to determine the marker information corresponding to the text identifier in the marker image, and the line information corresponding to the predicted line in the reference image;

[0051] The construction module is configured to calculate connection parameters based on the marker information and the line information, and construct a target relationship graph corresponding to the target image based on the connection parameters;

[0052] The extraction module is configured to extract text content from the target image using the target relationship graph.

[0053] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0054] Memory and processor;

[0055] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the memory, implement the steps of the text content extraction method.

[0056] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the text content extraction method.

[0057] The text content extraction method provided in this specification involves performing character recognition processing on a target image, determining a marker map containing text identifiers based on the processing results, inputting the marker map containing text identifiers into an image processing model for processing, obtaining a reference map containing predicted lines output by the image processing model, determining the marker information corresponding to the text identifiers in the marker map, and the line information corresponding to the predicted lines in the reference map, calculating connection parameters based on the marker information and line information, and constructing a target relationship graph corresponding to the target image based on the connection parameters, and extracting text content from the target image using the target relationship graph, thereby improving the completeness of text content extraction from the target image and improving the efficiency and accuracy of text extraction. Attached Figure Description

[0058] Figure 1 A schematic diagram of a text content extraction method according to an embodiment of this specification is shown;

[0059] Figure 2 This is a flowchart of a text content extraction method provided in one embodiment of this specification;

[0060] Figure 3 This is a schematic diagram of a text content extraction method provided in one embodiment of this specification;

[0061] Figure 4 This is a flowchart illustrating a text content extraction method for form information extraction provided in one embodiment of this specification.

[0062] Figure 5 This is a schematic diagram of the structure of a text content extraction device provided in one embodiment of this specification;

[0063] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0064] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0065] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0066] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0067] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0068] Convolutional Neural Networks (CNNs) are a class of feedforward neural networks that incorporate convolutional computations and have a deep structure. They are one of the representative algorithms of deep learning.

[0069] Graph Convolutional Network (GCN): In essence, it functions similarly to a CNN, acting as a feature extractor, except that its target data is graph data. GCN ingeniously designs a method for extracting features from graph data, allowing us to use these features to perform node classification, graph classification, and link prediction on graph data.

[0070] Figure 1 A schematic diagram of a text content extraction method according to an embodiment of this specification is shown, such as... Figure 1As shown, text recognition processing is performed on the target image, and a marker map containing text boxes and their center points is determined based on the processing results. This marker map is then input into an image processing model to obtain a reference map containing predicted lines. The marker information corresponding to the text boxes in the marker map and the line information corresponding to the predicted lines in the reference map are determined. Connection parameters are calculated based on the marker and line information, and a target relationship graph corresponding to the target image is constructed based on these parameters. The relationships between text entities in the target relationship graph are used to extract the corresponding text content from the target image, thereby improving the completeness of the extracted text and enhancing the efficiency and accuracy of text extraction.

[0071] In practical applications, target images include, but are not limited to, forms, resumes, transcripts, etc., in fields such as finance, technology, and academia. Target images can be obtained by photographing or scanning handwritten forms. One embodiment of this specification provides a text content extraction method that can be applied to intelligent form analysis scenarios. By inputting a complex form image, intelligent form information extraction is performed, extracting the matching relationships of various semantic entities within the form, such as: Name -> Wang Er, Gender -> Male, Home Address -> XX City XX Street, Purchase Date -> Year X Month X Day, etc. The form data is first processed using OCR text recognition to obtain text information; then, semantic entity recognition technology is used to obtain semantic entity classification results for different text regions; finally, entity matching technology is used to establish matching relationships for the classified semantic entities. Information is then extracted from the form based on these matching relationships.

[0072] This specification provides a method for extracting text content, and also relates to a text content extraction device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0073] Figure 2 A flowchart of a text content extraction method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0074] Step S202: Perform text recognition processing on the target image, and determine the marker image containing text identifiers based on the processing results.

[0075] Specifically, the target image refers to the image that needs to undergo text recognition processing. Target images include, but are not limited to, forms, resumes, transcripts, and shopping lists generated in shopping scenarios in fields such as finance, technology, and academia. Target images can also be obtained by taking photos or scanning handwritten forms. Correspondingly, text recognition processing typically includes several parts such as text information collection, information analysis and processing, and information classification. The method used can be OCR recognition to identify the text content in the target image. The processing result is the recognition result obtained by performing text recognition processing on the target image. When only the text in the target image is recognized, the processing result can be the recognized text content in the target image. In scenarios where the text in the target image needs to be labeled, the processing result can be adding text boxes and center points to the recognized text content, using the text boxes and center points as text markers to identify the text length, width, and other attribute information of the text in the target image. Correspondingly, the labeled image is the image generated based on the text markers determined by the text recognition processing, and the labeled image contains the text markers corresponding to each part of the text content.

[0076] Based on this, a target image containing text content is obtained, and the text content in the target image has a text layout structure. When performing text recognition processing on the target image, text recognition processing is performed on the target image according to the text layout structure, the text identifiers corresponding to the recognized text are determined, and a marker map containing the text identifiers is constructed based on the text identifiers.

[0077] In practical applications, when performing text recognition processing on target images, the text content in the target image usually has a text layout structure. When the target image is a form image, it typically contains a header and at least one corresponding text entry. For example, if the header is the price of a product, the corresponding text entry would be the price of each product. If there are at least two products, then one header entry corresponds to at least two text entries. Therefore, when performing text recognition processing on the target image, one header entry and the two corresponding text entries can be identified. Then, based on the text length information of the identified text content, a text identifier can be determined, and a marker image containing the text identifier can be generated.

[0078] Furthermore, considering that directly using text recognition to process the target image only yields the text content, subsequent text extraction operations based solely on this content cannot achieve text extraction based on the relationships between text elements. Therefore, a text box can be drawn based on the recognized text content, and then the center point of the text box can be determined. The specific implementation is as follows:

[0079] The text entities in the target image are subjected to text recognition processing, and the text box and center point corresponding to each text entity in the target image are determined according to the processing result; a marker image containing the text identifier is generated according to the text box and center point corresponding to each text entity in the target image.

[0080] Specifically, a text entity refers to a piece of text or a piece of data contained in the target image. When the target image is a grade sheet, each subject and each subject score is a text entity. When the target image is a form, the header and the corresponding form content are both text entities. A text box is a box that contains text entities. The size of the text box covers the size of the text entities it contains, meaning that the text content corresponding to the text entities is all within the text box. Correspondingly, the center point is the center point of the text box, which can be the intersection of the two center lines of the text box.

[0081] Based on this, text recognition processing is performed on the text entities in the target image to identify all text entities contained in the target image. The identified text entities are then used as the objects to be processed. Boxes that completely cover the text entities are identified as the corresponding text boxes, and the midpoint of each text box is used as the center point. This process is repeated to determine the text box and center point corresponding to each text entity in the target image. Based on the text box and center point corresponding to each text entity in the target image, a marker image containing text labels is generated.

[0082] For example, in a scenario where the form is a resume, information extraction and analysis of the structured resume is required. OCR and other text recognition methods are used to identify text entities in the resume. These text entities include resume items and resume content. Resume items are fields such as "Name," "Gender," and "Education Level," while resume content consists of personal information such as "Li Si," "Male," and "Bachelor's Degree." After identifying the fields and personal information, text boxes matching the length of each field and personal information are determined, and the midpoint of each text box is used as the center point of the text entity. A marker image is then generated based on the text box and center point of each text entity.

[0083] In summary, based on the text box and center point corresponding to each text entity in the target image, a marker map containing text labels is generated. This enables the text entities in the target image to be marked in the form of text boxes and center points, which facilitates subsequent text extraction based on the text boxes and center points corresponding to the text entities.

[0084] Furthermore, text recognition processing is performed on the target image. After identifying the text entities contained in the target image, considering that the number of characters in each text entity is different, the length of each text entity is different. After determining the entity information of each text entity, the text box corresponding to that text entity can be drawn, and then the center point of the text entity can be determined based on the text box. The specific implementation is as follows:

[0085] The text entities in the target image are subjected to text recognition processing, and the entity information corresponding to each text entity in the target image is determined according to the processing result; based on the entity information of each text entity in the target image, a text box and center point corresponding to the text entity in the target image are generated.

[0086] Specifically, entity information includes, but is not limited to, the entity length, entity width, and entity position information of the text entity. The text box corresponding to the text entity can be determined based on the entity length, entity width, and entity position information.

[0087] Based on this, text recognition processing is performed on the text entities in the target image. The processing results determine the entity information, such as the entity length, entity width, and entity position, for each text entity in the target image. Based on the entity information of each text entity in the target image, a text box corresponding to the text entity is generated, and the center point is determined based on the side length of the text box.

[0088] Following the previous example, after identifying the text entities in the resume, such as "name", "gender", "education", "Li Si", "male", and "bachelor's degree", the length and width of each text entity are determined based on the number of characters corresponding to each text entity. Then, the text box and center point of the text entity are determined based on the length and width of the text entity.

[0089] In summary, based on the entity information of each text entity in the target image, a text box and center point corresponding to the text entity in the target image are generated, so that all the text contained in the text entity can be contained in the text box, thus achieving complete selection of the text entity.

[0090] Furthermore, considering the diverse types of text entities in the target image, to facilitate subsequent processing of the marker map, different attribute information can be determined based on different types of text entities during marker map generation, and then the marker map can be generated based on the attribute information. The specific implementation is as follows:

[0091] Semantic recognition processing is performed on the text entities in the target image. Based on the processing results, the entity category information corresponding to each text entity in the target image is determined. Based on the entity category information of each text entity in the target image, the attribute information of the text box corresponding to the text entity in the target image is determined, as well as the attribute information of the target image is determined. The text box and center point corresponding to each text entity are used as initial text identifiers. The attributes of the initial text identifiers are updated according to the attribute information of the target image and the attribute information of the text box. Based on the update results, a marker map containing the text identifiers is generated.

[0092] Specifically, semantic recognition refers to identifying the semantic features of the text content corresponding to text entities. This can be achieved through named entity recognition, neural network models, and other methods to process semantic features of each text entity, thereby determining its entity category information. Entity category information represents the entity type of the text entity, which includes fixed entities and variable entities. For example, in the case of a report card, headers such as "Name" and "Class" are fixed entities, while "Wang Er" and "Class 3, Grade 1" are variable entities. Attribute information refers to color channel information. In the case of multiple color channels, any one color channel can be used as the attribute information of the target image, while other channels can be used as the attribute information of the text boxes corresponding to the text entities. For instance, if the entity category information of the text entities is divided into fixed and variable entities, the R channel can be used as the attribute information of the variable entities, the G channel as the attribute information of the fixed entities, and the B channel as the attribute information of the target image, thus generating a marker map. This marker map uses various color channels to color-mark the entity types, thereby improving the accuracy of text extraction in subsequent processes.

[0093] Based on this, semantic recognition processing is performed on each text entity in the target image to identify the entity category information corresponding to each text entity, and the text entities in the target image are classified according to the entity category. Based on the entity category information of each text entity in the target image, the attribute information of the text box corresponding to the text entity in the target image, as well as the attribute information of the target image, are determined. The text box and center point corresponding to each text entity are used as initial text identifiers. The initial text identifiers are updated according to the attribute information of the target image and the attribute information of the text box to obtain updated text identifiers. Then, a marker map containing the updated text identifiers is generated based on the updated text identifiers.

[0094] Following the previous example, after identifying the text entities "Name," "Gender," "Education," "Li Si," "Male," and "Bachelor's Degree" in the resume using OCR and other text recognition methods, semantic recognition is performed on each text entity to determine its entity type. "Name," "Gender," and "Education" are considered fixed entities, while "Li Si," "Male," and "Bachelor's Degree" are considered variable entities. A color channel is assigned to each of the fixed entities, variable entities, and the resume image, and these color channels are used as attribute information. The R channel is used as the attribute information for the variable entities "Li Si," "Male," and "Bachelor's Degree," the G channel is used as the attribute information for the fixed entities "Name," "Gender," and "Education," and the B channel is used as the attribute information for the resume image. This process generates a labeled image corresponding to the resume image with color channel information.

[0095] In summary, by updating the attributes of the initial text labels according to the attribute information of the target image and the text box, a label map containing the text labels is obtained. This allows different text entities in the label map to have different attribute features, thereby improving processing efficiency when the label map is subsequently input into the image processing model.

[0096] Step S204: Input the marked image containing the text identifier into the image processing model for processing to obtain a reference image containing predicted lines output by the image processing model.

[0097] Specifically, after performing text recognition processing on the target image and determining the marker map containing text identifiers based on the processing results, the marker map containing text identifiers can be input into the image processing model for processing to obtain a reference map containing predicted lines output by the image processing model. The image processing model can be a model corresponding to a convolutional neural network, used to establish relationships based on the text identifiers contained in the marker map. The relationships between text identifiers with relationships in the marker map are established, and these relationships are represented by lines, which are the predicted lines. The image processing model can predict the relationships between text identifiers. If there is a relationship between text identifiers, the predicted lines between the text identifiers are drawn, thereby obtaining a reference map containing the predicted lines. The reference map contains the predicted lines between each pair of text identifiers with relationships.

[0098] Based on this, after performing text recognition processing on the target image and determining the marker image containing text labels according to the processing results, the marker image containing text labels is input into the image processing model for processing. The image processing model draws predicted lines based on the text labels in the marker image. By drawing predicted lines between two related text labels, a reference image containing predicted lines can be obtained. The reference image is the output of the image processing model, and the predicted lines in the reference image correspond to the text labels in the marker image.

[0099] In practical applications, considering that the text content in the target image has different lengths, in order to improve the accuracy of the predicted connection between text labels when the text labels corresponding to the text content are text boxes, the center point of the text box can be determined after the text content is determined. The text box and the center point are used as text labels. Then, when there is a relationship between the two text labels, the center points of the two text labels can be connected to realize the drawing of the predicted line.

[0100] Furthermore, when processing the labeled image using an image processing model, considering that different image processing models have different requirements for the dimensionality of the labeled image, the dimensions of the labeled image need to be adjusted before inputting it into the image processing model. The specific implementation is as follows:

[0101] The marker image containing the text identifier is converted into an initial marker image of a first preset dimension; the initial marker image is processed by the network layer and convolutional layer in the image processing model, and a second preset dimension reference image containing the predicted lines is determined based on the processing result.

[0102] Specifically, the first preset dimension refers to the dimension pre-set for the marker image. The preset dimension can be any value. When the preset dimension is 3, dimension 1 can be used to represent the color of the marker image, dimension 2 can be used to represent the size of the marker image, and dimension 3 can be used to represent the depth value of the marker image. Correspondingly, the second preset dimension is the dimension pre-set for the output of the image processing model. Each dimension can correspond to an image dimension. The image dimension can represent the attributes of the image, including but not limited to image color, image size, and image resolution.

[0103] Based on this, the marker image containing text labels is converted into an initial marker image of a first preset dimension. The initial marker image of the first preset dimension is then input into an image processing model for processing. The initial marker image is processed through network layers and convolutional layers in the image processing model. Based on the processing results, a reference image of a second preset dimension containing predicted lines is determined. The image processing model can be a CNN model. The initial marker image is processed by convolutional sampling through the CNN model to obtain the reference image containing predicted lines. This enables the construction of connections between text boxes based on information such as text boxes, center points, and text contained in the marker image, thus obtaining the reference image containing predicted lines.

[0104] Following the previous example, after obtaining the marker image corresponding to the resume image with color channel information, the marker image can be input into the image processing model for processing. In the case of a CNN model, the marker image is first processed into a W*H*3 resume image to be processed. Then, the network layers and convolutional layers in the CNN process the resume image sequentially to obtain a (W / 2)*(H / 2)*1 resume connection diagram. In this diagram, lines can be used to represent the relationships between text boxes.

[0105] In summary, by inputting a fixed-dimensional labeled image into an image processing model for processing, the resulting reference image also has a fixed dimension. Processing through a model can improve the speed of image processing and further enhance its standardization.

[0106] Step S206: Determine the marker information corresponding to the text identifier in the marker diagram, and the line information corresponding to the predicted line in the reference diagram.

[0107] Specifically, after inputting the marked image containing text labels into the image processing model for processing and obtaining the reference image containing predicted lines output by the image processing model, the label information corresponding to the text labels in the marked image and the line information corresponding to the predicted lines in the reference image can be determined to facilitate the subsequent calculation of connection parameters. The label information includes, but is not limited to, the position information, length information, and width information of the text labels; the line information refers to the position information, line length information, and pixel information corresponding to the lines in the reference image. Based on the label information and line information, the subsequent calculation of connection parameters can be realized.

[0108] Based on this, after inputting the marked image containing text labels into the image processing model to obtain a reference image containing predicted lines, the model determines label information such as the label position corresponding to the text labels in the marked image, and determines line information such as the pixel points corresponding to the predicted lines in the reference image. This facilitates the subsequent calculation of connectivity parameters based on the label information and line information.

[0109] In practical applications, the marking information can be the location of a text label, which can be represented by coordinates. A coordinate system is established in the marking image, and the coordinate information corresponding to the text label is determined based on its position in the coordinate system. Since the predicted line is a line drawn based on the relationship between text labels, the predicted line has corresponding pixels in the reference image, and the line information can be the number of pixels occupied by the predicted line.

[0110] Furthermore, considering that the marker image contains a large number of text boxes, and whether there is a correspondence between the text boxes cannot be determined solely by the marker image, the text boxes in the marker image can be arranged and combined to obtain multiple text box pairs. The specific implementation is as follows:

[0111] Multiple text boxes are determined based on the text identifiers contained in the marked image; the multiple text boxes are combined, and multiple text box pairs are determined based on the combination result; the determination of the marking information of any text identifier includes: using the first position information of the first text box and the second position information of the second text box contained in the text box pair as the marking information of the text identifier.

[0112] Specifically, the combination processing refers to the permutation and combination of multiple text boxes contained in the marker image. Combining any two text boxes yields a text box pair, and combining all the text boxes in the marker image yields multiple text box pairs corresponding to the marker image. The first text box is any one of the text boxes in a text box pair, and correspondingly, the second text box is the second text box in the text box pair that corresponds to the first text box. The first position information is the position information of the first text box in the marker image; the second position information is the position information of the second text box in the marker image. Both the first and second position information can be represented in coordinate form.

[0113] Based on this, multiple text boxes are determined according to the text identifiers contained in the marker diagram; the multiple text boxes are combined, and multiple text box pairs are determined according to the combination result; the determination of the marking information of any text identifier includes: taking the first position information of the first text box and the second position information of the second text box contained in the text box pair as the marking information of the text identifier.

[0114] Following the previous example, in the marked graph containing text boxes corresponding to the text entities "Name", "Gender", "Education Level", "Li Si", "Male", and "Undergraduate", we determine all the text boxes contained in the marked graph. By combining any two text boxes from multiple text boxes, we can obtain multiple text box pairs consisting of "Name" and "Li Si", as well as "Gender" and "Li Si". The position information of the two text boxes contained in a text box pair is used as the marked information. For the text box pair consisting of "Name" and "Li Si", we determine the position coordinates of the two vertices of the "Name" text box (2,2) and (3,1), and the position coordinates of the two vertices of the "Li Si" text box (5,5) and (6,4). We use these two sets of coordinates as the marked information. When selecting the vertices of a text box, we can choose a set of vertices at the diagonal position of the text box.

[0115] In summary, by combining multiple text boxes and determining multiple text box pairs based on the combination results, it is possible to list possible text box pairs when the relationship between the text boxes is unknown, and then identify the text box pairs with related relationships among the multiple text box pairs.

[0116] Step S208: Calculate connection parameters based on the marker information and the line information, and construct a target relationship graph corresponding to the target image based on the connection parameters.

[0117] Specifically, after determining the marker information corresponding to the text identifiers in the marker image and the line information corresponding to the predicted lines in the reference image, connection parameters can be calculated based on the marker information and line information. A target relationship graph corresponding to the target image can then be constructed based on these connection parameters. The connection parameter refers to the parameter value between two text identifiers corresponding to the marker information, calculated based on the marker information and line information. The connection parameter represents the degree of correlation between the two text identifiers. For example, if the text content corresponding to the two text identifiers is "gender" and "male," since "gender" and "male" have a high degree of correlation, the parameter value of the connection parameter between these two text identifiers will be high. However, if the text content corresponding to the two text identifiers is "gender" and "Wang Er," since "Wang Er" represents a name, there is no correlation between "gender" and "Wang Er," or if there is a correlation, the degree of correlation is low. Therefore, the parameter value of the connection parameter between these two text identifiers will be low. Correspondingly, the target relationship graph is an undirected graph containing entities and relationships. If two entities have a correlation, an undirected edge is drawn between them.

[0118] Based on this, after determining the marker information corresponding to the text identifiers in the marker image and the line information corresponding to the predicted lines in the reference image, the connection parameters between the text identifiers corresponding to the marker information are calculated based on the marker information and the line information. After calculating the connection parameters between each pair of related text identifiers in the marker image, the target relationship graph corresponding to the target image can be constructed based on the connection parameters.

[0119] In practical applications, two related text labels can be identified in the labeling graph through traversal. Two text labels are arbitrarily selected in the labeling graph, and it is determined in the reference graph whether there is a predicted line between them. If so, the labeling information corresponding to each of the two text labels is determined, and the predicted line between them is determined based on the reference graph. Then, connection parameters are calculated based on the line information of the predicted line and the labeling information corresponding to each of the two text labels, facilitating the subsequent construction of the target relationship graph based on these connection parameters.

[0120] Furthermore, considering that the marker graph contains many pairs of text boxes, each pair corresponding to one marker message, and since the distance between two text boxes in multiple pairs is different, the connection parameters between the two text boxes in a pair can be determined by calculation. Based on these connection parameters, it can then be determined whether there is a relationship between the two text boxes in a pair. The specific implementation is as follows:

[0121] Determine the associated marker information and associated line information that have a relationship; determine the first associated position parameter and the second associated position parameter based on the associated marker information; calculate the reference position parameter according to the first associated position parameter and the second associated parameter, and use the reference position parameter as the reference value; determine the line pixel value corresponding to the line information based on the reference image; calculate the sub-connection parameter based on the reference value and the line pixel value.

[0122] Specifically, the association marker information refers to the attribute information of the two text boxes contained in the text box pair. The association marker information includes, but is not limited to, the position coordinates of the two text boxes in the text box pair, and the size of the text boxes. Correspondingly, the first association position parameter refers to the position coordinate parameter and text box size parameter corresponding to any one text box in the text box pair, and the second association position parameter refers to the position coordinate parameter and text box size parameter corresponding to the other text box in the text box pair. The reference position parameter is the parameter value obtained after calculating the first association position parameter and the second association position parameter. It can be calculated by summing the horizontal and vertical coordinates of the text boxes and taking the average. The position coordinates can be the coordinates of the vertex of the text box or the coordinates of any point on the edge of the text box. The association line information refers to the line information corresponding to the text box pair in the reference image. The line pixel value refers to the number of pixels corresponding to the line information in the reference image, that is, the number of pixels occupied by the line between the two text boxes contained in the text box pair in the reference image. The sub-connection parameter is the connection parameter corresponding to the two text boxes in the text box pair, which is used to represent the association relationship between the two text boxes.

[0123] Based on this, text box pairs are identified in the marker image, and lines associated with these text box pairs are identified in the reference image, thereby determining the associated marker information and associated line information. The first and second associated position parameters contained in the associated marker information are determined. A reference position parameter is calculated based on the first and second associated position parameters and used as a reference value. The line pixel values ​​corresponding to the line information are determined based on the reference image. Sub-connection parameters, representing the degree of association between the two text boxes in the text box pair, are calculated using the reference value and the line pixel values.

[0124] Following the previous example, determine the position coordinates of the two vertices of the "Name" text box (2, 2) and (3, 1), and the position coordinates of the two vertices of the "Li Si" text box (5, 5) and (6, 4). Then, determine the connection between the "Name" text box and the "Li Si" text box in the resume connection diagram, and determine the number of pixels (81) occupied by this connection in the resume connection diagram. Use the following formula (1) to calculate the sub-connection parameters between the "Name" text box and the "Li Si" text box:

[0125]

[0126] Where R represents the sub-connection parameter, N represents the number of pixels, and the position coordinates of the two vertices of the "Name" text box are (Cx0) and (Nx0) respectively. i Cy0 i ), (Cx1 i Cy1 i The coordinates of the two vertices of the text box "Li Si" are (Cx0)j Cy0 j ), (Cx1 j Cy1 j ), i represents the “Name” text box, j represents the “Li Si” text box, and abs represents taking the absolute value. Therefore, by calculating through formula (1), the sub-connection parameter is: R=81 / (abs(((((2+3) / 2)-((5+6) / 2))*(((2+1) / 2)-((5+4) / 2))))=9.

[0127] In summary, by calculating the sub-connection parameters between two text boxes in a text box pair, the degree of association between the text boxes can be determined, thus improving the efficiency of determining whether text boxes are related.

[0128] Furthermore, considering that the calculated values ​​of each sub-connection parameter may differ significantly, if each sub-connection parameter is used as the basis for drawing the target relationship graph, the target relationship graph will contain lines connecting text boxes with low correlation. Therefore, after determining each sub-connection parameter, the sub-connection parameters can be filtered. The specific implementation is as follows:

[0129] Each sub-connection parameter in the connection parameters is compared with a preset parameter threshold, and the target sub-connection parameter is determined based on the comparison result; the target text box is determined in the marker graph based on the target sub-connection parameter; the target text box is extracted, and the target text box is connected by lines, and a target relationship graph is generated based on the connection result.

[0130] Specifically, the preset parameter threshold refers to a pre-set threshold used for comparison with each sub-connection parameter. Through comparison, the sub-connection parameters can be divided into two categories: sub-connection parameters that are greater than the preset parameter threshold and sub-connection parameters that are less than the preset parameter threshold. Correspondingly, the target sub-connection parameter is at least one sub-connection parameter determined after comparing each sub-connection parameter with the preset parameter threshold. The target text box refers to the text box pair corresponding to the target sub-connection parameter. Since a sub-connection parameter is calculated based on a text box pair, a sub-connection parameter corresponds to two text boxes with a connection relationship.

[0131] Based on this, each sub-connection parameter in the connection parameters is compared with a preset parameter threshold, and the sub-connection parameters that are greater than the preset parameter threshold are designated as target sub-connection parameters. Based on the target sub-connection parameters, two target text boxes corresponding to the target sub-connection parameters are determined in the marker diagram. The two target text boxes are extracted, and a connection is drawn between them. Then, a target relationship diagram is generated based on the connection results.

[0132] Continuing with the previous example, a resume connection diagram and a marker diagram were determined based on the resume. The connection parameters for each text pair can be calculated based on the text pairs contained in the marker diagram and the lines corresponding to those text pairs. After calculating 50 connection parameters for 50 text pairs, each connection parameter is compared with a preset parameter threshold. The 30 connection parameters with values ​​greater than the threshold are determined, and a relationship diagram is then drawn based on each of these 30 determined connection parameters.

[0133] In summary, by comparing each sub-connection parameter in the connection parameters with a preset parameter threshold, text box pairs with a high degree of correlation are selected. Then, a connection is drawn between the two target text boxes in each text box pair, eliminating text box pairs with low correlation. This improves the accuracy of the target relationship graph.

[0134] Furthermore, considering that the relationships between text boxes may not match the relationships between text entities in the actual target image before generating the target relationship graph, it is necessary to filter the relationships, delete useless relationships, and then generate the target relationship graph. The specific implementation is as follows:

[0135] An initial relationship graph corresponding to the target image is generated based on the connection results; target relationships are determined in the initial relationship graph based on a neural network model; the target relationships are filtered in the initial relationship graph, and relationships to be deleted are determined based on the filtering results; the relationships to be deleted are deleted in the initial relationship graph, and a target relationship graph is generated based on the processing results.

[0136] Specifically, the initial relationship graph refers to an undirected graph containing relationships between text entities; the target relationship refers to the valid relationship in the initial relationship graph, which can be used for text extraction later. For example, there is a relationship between "gender" and "female", and there is also a relationship between "gender" and "20 years old". However, in actual applications, even if there is a relationship between "gender" and "20 years old", text extraction cannot be performed based on this relationship because "gender" and "20 years old" cannot be matched. Accordingly, the relationship between "gender" and "20 years old" is the relationship to be deleted.

[0137] Based on this, an initial relationship graph corresponding to the target image is generated according to the connection results. The target relationship is selected by filtering within the initial relationship graph using a neural network model. Then, the initial relationship graph is filtered again based on the target relationship to select the relationships to be deleted. Finally, the relationships to be deleted are removed from the initial relationship graph to obtain the target relationship graph.

[0138] Following the previous example, before drawing the relationship graph based on each of the 30 connection parameters, we can also use the GCN network to discriminate these 30 relationships, identifying the 9 useless relationships among them. Then, based on the remaining 21 relationships, we can generate the target relationship graph, achieving the desired result. Figure 3 The relationship filtering shown is for... Figure 3 The 30 relations contained in (a) were filtered to determine 21 valid relations, which were then used to generate the following: Figure 3 The target relationship diagram shown in (b) is as follows.

[0139] In summary, by using a neural network model to remove some useless relationships from the initial relationship graph and generate a target relationship graph, the efficiency and accuracy of relationship filtering in the initial relationship graph can be improved.

[0140] Step S210: Extract text content from the target image using the target relationship graph.

[0141] Specifically, after calculating the connection parameters based on the marker and line information, and constructing the target relationship graph corresponding to the target image based on the connection parameters, the target relationship graph can be used to extract text content from the target image. The text content refers to the text in the target image that needs to be extracted and stored. In the case of a form, the extracted text content can include the header and the body of the form, or it can be only the body of the form. In the scenario where the target image is a student's report card, the subject name, student name, and class information constitute the header, while the scores for each subject, names such as "Wang Er," and information such as "Class 3, Grade 1" constitute the body of the form. When extracting text content, all text in the report card can be extracted, or only the specific scores can be extracted and directly stored in the completed electronic document, thus populating the electronic document with content.

[0142] Based on this, after calculating the connection parameters according to the marking information and line information, and constructing the target relationship graph corresponding to the target image based on the connection parameters, the text content is extracted from the target image using the target relationship graph. That is, two entity points with a connection relationship are identified in the target relationship graph as entity pairs. Based on the connection lines between entity points in the entity pairs, the text content corresponding to the two entity points in the target image is determined, and then the text content is extracted. All entity pairs contained in the target relationship graph are identified, and the text content corresponding to the entity pairs is extracted from the target image based on all entity pairs.

[0143] In practical applications, when extracting text content from a target image, all text content corresponding to the target relationship graph in the target image can be extracted, or only a portion of the text content can be extracted. An entity pair contains two entities, one of which can be regarded as the head entity and the other as the content entity. When extracting text content, both the text content corresponding to the head entity and the text content corresponding to the content entity can be extracted as text content, or only the text content corresponding to the content entity can be extracted.

[0144] In the scenario of extracting resume content, the header entities are "name", "age", "work experience", "education", etc., and the content entities are "Zhang San", "28 years old", "3 years", "Bachelor's degree", etc. Therefore, when extracting text content, you can extract "name", "age", "work experience", "education", and their corresponding text content such as "Zhang San", "28 years old", "3 years", "Bachelor's degree", etc., or you can extract only the text content such as "Zhang San", "28 years old", "3 years", "Bachelor's degree", etc., and then directly store the text content such as "Zhang San", "28 years old", "3 years", "Bachelor's degree".

[0145] Furthermore, considering that the target relationship graph contains multiple entities and the relationships between these entities are complex, when extracting text content based on the target relationship graph, it is necessary to clearly identify the entity pairs contained in the target relationship graph, and then determine and extract the text content from the target image based on these entity pairs. The specific implementation is as follows:

[0146] Entity pair construction is performed on two entity points with a connection relationship in the target relationship graph, and multiple entity pairs are determined based on the processing results; the text content corresponding to each entity pair is determined and extracted in the target image.

[0147] Specifically, an entity point refers to a node contained in the target relationship graph; two nodes with edges in the target relationship graph are connected, and an entity pair containing the two nodes can be constructed based on the edges between the two nodes; multiple entity pairs contained in the target relationship graph can be determined based on the target relationship graph; based on the entity pairs, two text entities with a relationship can be identified in the target image, and then the text content corresponding to the two text entities with a relationship can be extracted.

[0148] Based on this, entity points are identified in the target relationship graph. Two entity points with a connection relationship are identified based on the edges in the target relationship graph. Entity pair construction processing is performed on the two entity points with a connection relationship in the target relationship graph. Based on the processing results, multiple entity pairs contained in the target relationship graph are identified. The text content corresponding to each entity pair is determined in the target image, and the text content of each entity pair is extracted.

[0149] Using the previous example, when extracting the text content corresponding to the resume, you can extract only personal information such as "Zhang San," "28 years old," "3 years," and "Bachelor's degree." After extracting the personal information from the resume, the resume information can be stored. Alternatively, information can be extracted and stored in a "key-value pair" manner. That is, extract "Name" - "Zhang San," "Age" - "28 years old," "Work Experience" - "3 years," and "Education" - "Bachelor's degree," etc.

[0150] In practical applications, when extracting text content corresponding to text entities in a form, since text entities include both fixed and variable entities, in scenarios where both fixed and variable entities in the form are entered into a computer for storage, the text content corresponding to the fixed and variable entities with connections can be extracted based on the target relationship graph and then persistently stored. When storing the extracted text content into a table that already contains the header information of the form, only the text content corresponding to the variable entities can be extracted, and the text content corresponding to the variable entities can be stored in the corresponding positions in the table based on the target relationship graph.

[0151] Continuing with the previous example, we can extract and store the text entities "Name", "Gender", "Education", "Li Si", "Male", and "Bachelor's Degree" from the resume. We can extract "Name"-"Li Si", "Gender"-"Male", and "Education"-"Bachelor's Degree" as text content, and then store the extracted text content in a table or similar format. Alternatively, if a table with header information such as "Name", "Gender", and "Education" has already been created, we can extract only the text content of "Li Si", "Male", and "Bachelor's Degree", and then store this text content in the corresponding location within the table, achieving persistent and standardized storage of resume information.

[0152] In summary, the text content extraction method provided in this specification involves performing character recognition processing on a target image, determining a marker map containing text identifiers based on the processing results, inputting the marker map containing text identifiers into an image processing model for processing, obtaining a reference map containing predicted lines output by the image processing model, determining the marker information corresponding to the text identifiers in the marker map, and the line information corresponding to the predicted lines in the reference map, calculating connection parameters based on the marker information and line information, and constructing a target relationship graph corresponding to the target image based on the connection parameters, and extracting text content from the target image using the target relationship graph, thereby improving the completeness of text content extraction from the target image and improving the efficiency and accuracy of text extraction.

[0153] The following is in conjunction with the appendix Figure 4 Taking the text content extraction method provided in this specification as an example of its application in form information extraction, the text content extraction method will be further explained. Among other things, Figure 4 This specification illustrates a flowchart of a text content extraction method for form information extraction according to an embodiment of this specification, which specifically includes the following steps:

[0154] Step S402: Obtain the target image.

[0155] In intelligent form analysis scenarios, by inputting a complex form image, an intelligent form information extraction system can extract the matching relationships of various semantic entities within the form, such as: Name -> Wang Er, Gender -> Male, Home Address -> XX City XX Street, Purchase Date -> Year X Month X Day, etc. This involves acquiring the form image in the context of form information extraction.

[0156] Step S404: Perform recognition processing on the target image, and determine the text box and center point corresponding to each text entity in the target image based on the recognition processing results.

[0157] By using OCR text recognition to identify text in the form image, each text entity contained in the form image is identified, and then the text box and center point corresponding to each text entity are drawn.

[0158] Step S406: Generate a marker map based on the text box and center point corresponding to each text entity in the target image.

[0159] Before drawing the labeled image, semantic entity text information and center points are added to the image. The semantic entity category is added to different image channels. For example, if the original form image has three channels, and the semantic entities are divided into two categories, then only one channel is retained, and the other two channels represent the text boxes for each category of semantic entities, thus generating a labeled image containing channel information, text boxes, and center points. It should be noted that if the semantic entity categories include other categories, the number of channels can be increased.

[0160] Step S408: Input the labeled image into the image processing model and output the reference image.

[0161] A CNN network is constructed. The original image (W*H*3) is input as input. After processing through network layers and convolutional layers, the output is a line graph of (W / 2)*(H / 2)*1. The CNN network structure includes ResNet18, deconvolutional upsampling layers, and convolutional layers. The original image (W*H*3) is input, processed by ResNet18 to obtain (W / 8)*(H / 8)*512, then upsampled by deconvolutional layers to obtain (W / 4)*(H / 4)*256, then upsampled again by deconvolutional layers to obtain (W / 2)*(H / 2)*128, and finally processed by a 1*1 convolutional layer to obtain (W / 2)*(H / 2)*1, which is then output.

[0162] Step S410: Traverse the marked graph and calculate the connection parameters based on the traversal results and the reference graph.

[0163] Traverse the marker graph and identify text boxes i and j corresponding to any two text entities contained in the graph. Based on the position of the text boxes in the marker graph, determine the coordinates of the two vertices located on the diagonal, i.e., the coordinates of the top-left corner (Cx0) of text box i. i Cy0 i ) and the coordinates of the lower right corner (Cx1) i Cy1 i The coordinates of the top left corner of text box j are (Cx0). j Cy0 j ) and the coordinates of the lower right corner (Cx1) j Cy1 j Based on the line drawing, determine the reference lines corresponding to text boxes i and j, determine the number of pixels N occupied by the reference lines, and then calculate the connection parameter R using the following formula (1):

[0164]

[0165] Where R represents the sub-connection parameter, N represents the number of pixels, and abs represents the absolute value. The calculated connection parameter R is compared with a preset parameter threshold, and multiple connection parameters greater than the preset threshold are obtained. This process continues until all text entities in the marked image have been traversed, at which point multiple connection parameters are obtained.

[0166] Step S412: Draw the initial relationship diagram based on the connection parameters.

[0167] Step S414: Filter the initial relationship graph to obtain the target relationship graph.

[0168] The invalid relationships in the initial relationship graph are identified and deleted based on GCN (Graph Convolutional Neural Network), thus obtaining the target relationship graph corresponding to the form image.

[0169] Step S416: Extract text based on the target relationship graph and the target image.

[0170] In summary, the text content extraction method provided in this specification involves performing character recognition processing on a target image, determining a marker map containing text identifiers based on the processing results, inputting the marker map containing text identifiers into an image processing model for processing, obtaining a reference map containing predicted lines output by the image processing model, determining the marker information corresponding to the text identifiers in the marker map, and the line information corresponding to the predicted lines in the reference map, calculating connection parameters based on the marker information and line information, and constructing a target relationship graph corresponding to the target image based on the connection parameters, and extracting text content from the target image using the target relationship graph, thereby improving the completeness of text content extraction from the target image and improving the efficiency and accuracy of text extraction.

[0171] Corresponding to the above method embodiments, this specification also provides embodiments of a text content extraction device. Figure 5 A schematic diagram of a text content extraction device according to an embodiment of this specification is shown. Figure 5 As shown, the device includes:

[0172] Processing module 502 is configured to perform text recognition processing on the target image and determine a marker image containing text identifiers based on the processing result;

[0173] The input module 504 is configured to input a marker image containing the text identifier into an image processing model for processing, and obtain a reference image containing predicted lines output by the image processing model.

[0174] The determining module 506 is configured to determine the marking information corresponding to the text identifier in the marking image, and the line information corresponding to the predicted line in the reference image;

[0175] The construction module 508 is configured to calculate connection parameters based on the marker information and the line information, and construct a target relationship graph corresponding to the target image based on the connection parameters;

[0176] Extraction module 510 is configured to extract text content from the target image using the target relationship graph.

[0177] In an optional embodiment, the input module 504 is further configured to:

[0178] The text entities in the target image are subjected to text recognition processing, and the text box and center point corresponding to each text entity in the target image are determined according to the processing result; a marker image containing the text identifier is generated according to the text box and center point corresponding to each text entity in the target image.

[0179] In an optional embodiment, the input module 504 is further configured to:

[0180] Semantic recognition processing is performed on the text entities in the target image. Based on the processing results, the entity category information corresponding to each text entity in the target image is determined. Based on the entity category information of each text entity in the target image, the attribute information of the text box corresponding to the text entity in the target image is determined, as well as the attribute information of the target image is determined. The text box and center point corresponding to each text entity are used as initial text identifiers. The attributes of the initial text identifiers are updated according to the attribute information of the target image and the attribute information of the text box. Based on the update results, a marker map containing the text identifiers is generated.

[0181] In an optional embodiment, the input module 504 is further configured to:

[0182] The marker image containing the text identifier is converted into an initial marker image of a first preset dimension; the initial marker image is processed by the network layer and convolutional layer in the image processing model, and a second preset dimension reference image containing the predicted lines is determined based on the processing result.

[0183] In an optional embodiment, the determining module 506 is further configured to:

[0184] Multiple text boxes are determined based on the text identifiers contained in the marked image; the multiple text boxes are combined, and multiple text box pairs are determined based on the combination result; the determination of the marking information of any text identifier includes: using the first position information of the first text box and the second position information of the second text box contained in the text box pair as the marking information of the text identifier.

[0185] In an optional embodiment, the construction module 508 is further configured to:

[0186] Determine the associated marker information and associated line information that have a relationship; determine the first associated position parameter and the second associated position parameter based on the associated marker information; calculate the reference position parameter according to the first associated position parameter and the second associated parameter, and use the reference position parameter as the reference value; determine the line pixel value corresponding to the line information based on the reference image; calculate the sub-connection parameter based on the reference value and the line pixel value.

[0187] In an optional embodiment, the construction module 508 is further configured to:

[0188] Each sub-connection parameter in the connection parameters is compared with a preset parameter threshold, and the target sub-connection parameter is determined based on the comparison result; the target text box is determined in the marker graph based on the target sub-connection parameter; the target text box is extracted, and the target text box is connected by lines, and a target relationship graph is generated based on the connection result.

[0189] In an optional embodiment, the construction module 508 is further configured to:

[0190] An initial relationship graph corresponding to the target image is generated based on the connection results; target relationships are determined in the initial relationship graph based on a neural network model; the target relationships are filtered in the initial relationship graph, and relationships to be deleted are determined based on the filtering results; the relationships to be deleted are deleted in the initial relationship graph, and a target relationship graph is generated based on the processing results.

[0191] In an optional embodiment, the extraction module 510 is further configured to:

[0192] Entity pair construction is performed on two entity points with a connection relationship in the target relationship graph, and multiple entity pairs are determined based on the processing results; the text content corresponding to each entity pair is determined and extracted in the target image.

[0193] In an optional embodiment, the input module 504 is further configured to:

[0194] The text entities in the target image are subjected to text recognition processing, and the entity information corresponding to each text entity in the target image is determined according to the processing result; based on the entity information of each text entity in the target image, a text box and center point corresponding to the text entity in the target image are generated.

[0195] In summary, the text content extraction device provided in this specification performs text recognition processing on a target image, determines a marker image containing text identifiers based on the processing results, inputs the marker image containing text identifiers into an image processing model for processing, obtains a reference image containing predicted lines output by the image processing model, determines the marker information corresponding to the text identifiers in the marker image, and the line information corresponding to the predicted lines in the reference image, calculates connection parameters based on the marker information and line information, and constructs a target relationship graph corresponding to the target image based on the connection parameters, and extracts text content from the target image using the target relationship graph, thereby improving the completeness of text content extraction from the target image and improving the efficiency and accuracy of text extraction.

[0196] The above is an illustrative scheme of a text content extraction device according to this embodiment. It should be noted that the technical solution of this text content extraction device and the technical solution of the above-described text content extraction method belong to the same concept. For details not described in detail in the technical solution of the text content extraction device, please refer to the description of the technical solution of the above-described text content extraction method.

[0197] Figure 6A structural block diagram of a computing device 600 according to an embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0198] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0199] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0200] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0201] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described text content extraction method.

[0202] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the text content extraction method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the text content extraction method described above.

[0203] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the text content extraction method described above.

[0204] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the text content extraction method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the text content extraction method described above.

[0205] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0206] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0207] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0208] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0209] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for extracting text content, characterized in that, include: Perform text recognition processing on the target image, and determine the marker image containing text identifiers based on the processing results; The marked image containing the text identifiers is input into the image processing model for processing to obtain a reference image containing predicted lines output by the image processing model. The predicted lines are the lines connecting the text identifiers and are used to indicate that the connected text identifiers have an association relationship. Multiple text boxes are determined based on the text identifiers contained in the marked image. The multiple text boxes are combined. Multiple text box pairs are determined based on the combination results. The first position information of the first text box and the second position information of the second text box contained in each text box pair are used as the marking information of the text identifiers. The line information corresponding to the predicted lines in the reference image is also determined. Connection parameters are calculated based on the marker information and the line information, and a target relationship graph corresponding to the target image is constructed based on the connection parameters. The connection parameters include sub-connection parameters, and the determination of the sub-connection parameters includes: determining associated marker information and associated line information with a relationship; determining a first associated position parameter and a second associated position parameter based on the associated marker information; calculating a reference position parameter based on the first associated position parameter and the second associated position parameter, and using the reference position parameter as a reference value; determining the line pixel value corresponding to the associated line information based on the reference graph; and calculating the sub-connection parameters based on the reference value and the line pixel value. The text content is extracted from the target image using the target relationship graph.

2. The method according to claim 1, characterized in that, The step of performing text recognition processing on the target image and determining a marker image containing text identifiers based on the processing results includes: The text entities in the target image are subjected to text recognition processing, and the text box and center point corresponding to each text entity in the target image are determined based on the processing results; Based on the text box and center point corresponding to each text entity in the target image, a marker image containing the text identifier is generated.

3. The method according to claim 2, characterized in that, The step of generating a marker image containing the text identifier based on the text box and center point corresponding to each text entity in the target image includes: Semantic recognition processing is performed on the text entities in the target image, and the entity category information corresponding to each text entity in the target image is determined based on the processing results; Based on the entity category information of each text entity in the target image, determine the attribute information of the text box corresponding to the text entity in the target image, and determine the attribute information of the target image; Use the text box and center point corresponding to each text entity as the initial text identifier; The initial text identifier is updated according to the attribute information of the target image and the attribute information of the text box, and a marker image containing the text identifier is generated based on the update result.

4. The method according to claim 1, wherein inputting the marker image containing the text identifier into an image processing model for processing to obtain a reference image containing predicted lines output by the image processing model includes: The marker image containing the text identifier is converted into an initial marker image of a first preset dimension; The initial labeled image is processed by network layers and convolutional layers in the image processing model, and a second preset dimension reference image containing predicted lines is determined based on the processing results.

5. The method according to claim 1, characterized in that, The step of constructing the target relationship graph corresponding to the target image based on the connection parameters includes: Each sub-connection parameter in the connection parameters is compared with a preset parameter threshold, and the target sub-connection parameter is determined based on the comparison result. The target text box is determined in the marking diagram based on the target sub-connection parameters; Extract the target text boxes, connect the target text boxes, and generate a target relationship graph based on the connection results.

6. The method according to claim 5, characterized in that, The step of generating the target relationship graph based on the connection results includes: Generate an initial relationship graph corresponding to the target image based on the connection results; The target relationship is determined based on the neural network model in the initial relationship graph; Based on the target relationship, the initial relationship graph is filtered, and the relationship to be deleted is determined according to the filtering results. The relationships to be deleted are deleted in the initial relationship graph, and a target relationship graph is generated based on the processing results.

7. The method according to claim 1, characterized in that, The step of extracting text content from the target image using the target relationship graph includes: Entity pair construction is performed on two entity points with a connection relationship in the target relationship graph, and multiple entity pairs are determined based on the processing results. The text content corresponding to each entity pair in the target image is determined and extracted.

8. The method according to claim 2, characterized in that, The step of performing text recognition processing on the text entities in the target image, and determining the text box and center point corresponding to each text entity in the target image based on the processing result, includes: The text entities in the target image are subjected to text recognition processing, and the entity information corresponding to each text entity in the target image is determined based on the processing result; Based on the entity information of each text entity in the target image, a text box and center point corresponding to the text entity in the target image are generated.

9. A text content extraction device, characterized in that, include: The processing module is configured to perform text recognition processing on the target image and determine a marker image containing text identifiers based on the processing results; The input module is configured to input the marker image containing the text identifiers into the image processing model for processing, and obtain a reference image containing predicted lines output by the image processing model, wherein the predicted lines are the lines connecting the text identifiers, and the predicted lines are used to indicate that the connected text identifiers have an association relationship. The determination module is configured to determine multiple text boxes based on the text identifiers contained in the marked image, combine the multiple text boxes, determine multiple text box pairs based on the combination result, use the first position information of the first text box and the second position information of the second text box contained in each text box pair as the marking information of the text identifiers, and determine the line information corresponding to the predicted lines in the reference image. A construction module is configured to calculate connection parameters based on the marker information and the line information, and to construct a target relationship graph corresponding to the target image based on the connection parameters. The connection parameters include sub-connection parameters, and the determination of the sub-connection parameters includes: determining associated marker information and associated line information with a relationship; determining a first associated position parameter and a second associated position parameter based on the associated marker information; calculating a reference position parameter based on the first associated position parameter and the second associated position parameter, and using the reference position parameter as a reference value; determining the line pixel value corresponding to the associated line information based on the reference graph; and calculating the sub-connection parameters based on the reference value and the line pixel value. The extraction module is configured to extract text content from the target image using the target relationship graph.

10. A computing device, characterized in that, It includes a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the text content extraction method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the text content extraction method according to any one of claims 1 to 8.

Images