Method, device, equipment and medium for reconstructing table rows of form images

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, and medium for reconstructing a table row from a form image. The method is applied to the medical field. In this application, text recognition is performed on a form image to be reconstructed to obtain N text contents in the form image to be reconstructed, and a text box containing the corresponding text contents is constructed. A corresponding directed graph is constructed based on the coordinates of the text boxes in the form, and candidate combinations in the same row as the text boxes are selected from the directed graph. The text boxes in the corresponding row are searched based on the candidate combinations to obtain a row text box in the row where the corresponding text box is located. This improves the accuracy of row reconstruction in the table, extracts the text content after row reconstruction as the text content of the corresponding row, and thus improves the accuracy of text extraction from the table.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for reconstructing table rows of a form image. Background Art

[0002] Digital medicine is a new, modern approach to healthcare that applies modern computer and information technology to the entire medical process. It represents the development direction and management goal of public healthcare. In digital healthcare research, various formatted data sources, such as medical records, pathology data, and surgical data, are typically used as data sources. Information extraction is performed from these data sources to obtain the corresponding information. During medical treatment, patients often bring paper medical records, examination reports, and laboratory test results from previous visits to a new hospital. This requires the new hospital to process the paper information. Existing technology uses optical character recognition to automatically extract information from images corresponding to paper documents, effectively improving the efficiency of manual information entry. However, during this information extraction process, if the information displayed in the image is neatly arranged, the extraction result is more accurate. If the information in the image is not neatly arranged, or even deformed or misaligned, it can be assumed that the text content in the corresponding row has shifted, making it difficult to accurately extract the corresponding text content, resulting in lower text extraction accuracy. Therefore, improving the accuracy of text extraction from images has become an urgent issue. Summary of the Invention

[0003] In view of this, embodiments of the present application provide a method, apparatus, device, and medium for reconstructing table rows of a form image to solve the problem of low text extraction accuracy during the information extraction process.

[0004] A first aspect of an embodiment of the present application provides a method for reconstructing a table row of a form image, the method comprising:

[0005] Obtaining a form image to be reconstructed, performing text recognition on the form image to be reconstructed to obtain N text contents in the form image to be reconstructed, constructing a text box enclosing the corresponding text contents, and filtering out target text and a text box corresponding to the target text from the N text contents, where N is an integer greater than 1;

[0006] Determining, based on the coordinate information of the text box of the target text, a row target text box composed of multiple target text text boxes in the same row, obtaining row target text boxes corresponding to different rows, dividing the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to the different rows, and calculating, for any sub-table, a fitting curve of the row target text box based on the coordinate information of the corresponding row target text box in the sub-table, where M is an integer greater than 1;

[0007] Extracting multiple column text boxes in the same column as the text box of the target text, constructing a directed graph corresponding to any column text box based on a preset spacing, determining candidate alignment text boxes corresponding to the column text box based on the directed graph, and traversing all column text boxes to obtain a candidate alignment text box corresponding to each column text box;

[0008] For any column, determine the candidate combination consisting of the corresponding column text box and the candidate alignment text box, filter the candidate combination according to the preset filtering conditions, and obtain the K optimal combinations after filtering. According to the preset search algorithm and the K optimal combinations, search for the text box in the same row as the corresponding column text box to obtain the row text box in the row where the column text box is located. According to the text content in each text box in the row text box, obtain the reconstructed row text in the form image to be reconstructed, where K is an integer greater than 1.

[0009] A second aspect of an embodiment of the present application provides a table row reconstruction device for a form image, the table row reconstruction device comprising:

[0010] an acquisition module, configured to acquire a form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct a text box enclosing the corresponding text contents, and filter out target text and a text box corresponding to the target text from the N text contents, where N is an integer greater than 1;

[0011] a division module, configured to determine, based on the coordinate information of the text boxes of the target text, a row target text box composed of multiple target text text boxes in the same row, obtain row target text boxes corresponding to different rows, divide the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to the different rows, and for any sub-table, calculate a fitting curve for the row target text box based on the coordinate information of the corresponding row target text box in the sub-table, where M is an integer greater than 1;

[0012] A construction module is configured to extract multiple column text boxes in the same column as the text box of the target text, construct a directed graph corresponding to any column text box based on a preset spacing, determine candidate alignment text boxes corresponding to the column text box based on the directed graph, and traverse all column text boxes to obtain a candidate alignment text box corresponding to each column text box;

[0013] The search module is used to determine, for any column, a candidate combination consisting of a corresponding column text box and a candidate alignment text box, screen the candidate combinations according to preset screening conditions, and obtain K optimal combinations after screening. According to a preset search algorithm and the K optimal combinations, the module searches for a text box in the same row as the corresponding column text box to obtain a row text box in the row where the column text box is located, and obtains the reconstructed row text in the form image to be reconstructed based on the text content in each text box in the row text box, where K is an integer greater than 1.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the table row reconstruction method as described in the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the table row reconstruction method as described in the first aspect is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] Obtain the form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct a text box that wraps the corresponding text content, and filter out the target text and the text box corresponding to the target text from the N text contents. N is an integer greater than 1. According to the coordinate information of the text box of the target text, determine the row target text box composed of multiple target text text boxes in the same row, and obtain the row target text boxes corresponding to different rows. According to the row target text boxes corresponding to different rows, divide the form image to be reconstructed into M sub-tables. For any sub-table, calculate the fitting curve of the row target text box according to the coordinate information of the corresponding row target text box in the sub-table. M is an integer greater than 1. Extract the text box corresponding to the target text. For multiple column text boxes in the same column, for any column text box, a directed graph corresponding to the column text box is constructed according to a preset spacing. Based on the directed graph, the candidate alignment text box corresponding to the column text box is determined. All column text boxes are traversed to obtain the candidate alignment text box corresponding to each column text box. For any column, a candidate combination consisting of the corresponding column text box and the candidate alignment text box is determined. According to the preset screening conditions, the candidate combination is screened to obtain the K optimal combinations after screening. According to the preset search algorithm and the K optimal combinations, the text box in the same row as the corresponding column text box is searched to obtain the row text box in the row where the column text box is located. According to the text content in each text box in the row text box, the reconstructed row text in the form image to be reconstructed is obtained, where K is an integer greater than 1. In the present application, a corresponding directed graph is constructed according to the coordinates of the text boxes in the table, a candidate combination in the same row as the text box is selected from the directed graph, and the text box in the corresponding row is searched according to the candidate combination to obtain the row text box in the row where the corresponding text box is located. This improves the row reconstruction accuracy in the table, extracts the text content after row reconstruction, and thus improves the table text extraction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 This is a schematic diagram of an application environment of a table row reconstruction method for a form image provided by an embodiment of the present invention;

[0020] Figure 2 This is a flow chart of a method for reconstructing a table row of a form image provided by one embodiment of the present invention;

[0021] Figure 3 1 is a schematic structural diagram of a table row reconstruction device for a form image provided by an embodiment of the present invention;

[0022] Figure 4 It is a structural diagram of a terminal device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0025] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0026] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0027] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0029] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0030] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0031] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0032] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0033] An embodiment of the present invention provides a table row reconstruction method for a form image, which can be applied in the following situations: Figure 1 In an application environment, a local terminal communicates with a server terminal. The local terminal includes, but is not limited to, a palmtop computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook computer, a personal digital assistant (PDA), and other terminal devices. The server terminal can be implemented as a standalone server or a server cluster generated by multiple servers.

[0034] See also Figure 2 , is a flow chart of a table row reconstruction method for a form image provided by an embodiment of the present invention. The above table row reconstruction method for a form image can be applied to Figure 1 The server in the above mentioned server is connected to the corresponding local end, such as Figure 2 As shown, the table row reconstruction method of the form image may include the following steps.

[0035] S201: Obtain a form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct a text box that wraps the corresponding text contents, and filter out target text and the text box corresponding to the target text from the N text contents.

[0036] In step S201, a form image to be reconstructed is obtained, where the form image is an image containing a table, such as a case image or a physical examination image. Text recognition is performed on the form image to be reconstructed, obtaining N text contents and corresponding text boxes in the form image to be reconstructed. Based on preset text tags, matching target text and text boxes corresponding to the target text are filtered from the text contents, where N is an integer greater than 1.

[0037] In this embodiment, the form image to be reconstructed is obtained by scanning or photographing the paper form. Before taking the photo, the photographing parameters can be set above the photographing area. The photographing color can be set to color, black and white, or grayscale; the cropping method can be set to manual, automatic, or no cropping; and whether to remove shadows during the photographing can be set. The text area in the form image to be reconstructed contains printed and / or handwritten text.

[0038] When extracting N text boxes and the text content in each text box in the form image to be reconstructed, optical character recognition technology can be used to perform text recognition on the form image to be reconstructed to obtain multiple text fragments and multiple bounding boxes. Optical character recognition technology can refer to detecting and recognizing text in an image. For example, optical character recognition technology can include two parts: text detection (Text Detection) and text recognition (Text Recognition). Text detection locates the area where text exists in the image, that is, finds the text bounding box of the word segment or sentence; text recognition recognizes the located text to obtain the corresponding text content.

[0039] In another example, the form image to be reconstructed may be input into a recognition model to obtain multiple text segments and multiple bounding boxes. The recognition model may be implemented as any object detection algorithm.

[0040] It should be noted that the text segment extracted from the form image to be reconstructed can be a word segment or a sentence. Furthermore, the extracted text bounding box has a corresponding position, which can be composed of two pairs of horizontal and vertical coordinates. In one example, the two pairs of horizontal and vertical coordinates include the horizontal and vertical coordinates of the upper left vertex and the lower right vertex of the text bounding box.

[0041] Filter out target text and a text box corresponding to the target text from N text contents, obtain a preset text label, filter out text contents that match the preset text label as target text, and determine the text box corresponding to the target text based on the target text. For example, the preset text label can be the entity name in the form image to be reconstructed, such as "project title (pro-title)", "project English abbreviation title (enpro-title)", "examination value title (val-title)", "reference range title (range-title)", "unit title (unit-title)", "project (pro)", "project English abbreviation (enpro)", "examination value (val)", "reference range (range)", "unit (unit)", "other" in the physical examination form. The preset text label can include different texts such as Chinese and English.

[0042] Optionally, text recognition is performed on the form image to be reconstructed to obtain N text contents in the form image to be reconstructed, including:

[0043] Perform text detection on the form image to be reconstructed to obtain a text region representing text in the form image to be reconstructed;

[0044] Perform text recognition on the text area to obtain the text content of the text in the text area.

[0045] In this embodiment, text detection is performed on the form image to be reconstructed through a preset detection network to obtain a text area representing the text in the form image to be reconstructed. The preset detection network may include a convolution layer, a batch normalization layer, a fusion layer, and an activation function.

[0046] It should be noted that, before using the preset detection network to perform text detection on the form image to be reconstructed, the embodiment of the present invention also includes: obtaining an initial detection network and a training sample set, using the training sample set to perform supervised training on the initial detection network to obtain a trained detection network, and determining the trained detection network as the preset detection network, wherein the training sample set is training data containing the form image to be reconstructed and the corresponding text area label. Using the preset detection network to perform text detection on the text image to obtain a text detection frame includes: using a convolutional layer to extract image features of the form image to be reconstructed to obtain a feature image, using a batch normalization layer (Batch Normalization, BN) to perform a normalization operation on the feature image to obtain a standard feature image; using a fusion layer to fuse the underlying features of the form image to be reconstructed with the standard feature image to obtain a target feature image, using an activation function to output the detection result of the target feature image, and generating a text area representing the text based on the detection result.

[0047] Perform text recognition on a text region to obtain the text content of the text in the text region. When performing text recognition on a text region, a preset recognition network is used to extract and recognize the text. The recognition network may include a convolutional neural network, a long short-term memory network, and a temporal classification network. The convolutional neural network is used to identify the feature image of the text region, the long short-term memory network is used to extract the text sequence from the feature image, and the temporal classification network is used to resolve the problem of character alignment in the text feature sequence. Furthermore, the convolutional neural network includes a convolutional layer, a pooling layer, and a fully connected layer, and the long short-term memory network includes an input gate, a forget gate, and an output gate.

[0048] It should be noted that, before using the preset recognition network to perform text recognition on the text area, the embodiment of the present invention also includes: obtaining an initial recognition network and a training sample set, using the training sample set to perform supervised training on the initial recognition network to obtain a trained recognition network, and determining the trained recognition network as the preset recognition network, wherein the training sample set is label training data containing the text area and the corresponding text content.

[0049] It should be noted that when using a preset recognition network to extract and recognize text, a convolutional neural network is used to extract features from the text region to obtain a feature image. Specifically, the convolutional layer of the convolutional neural network is used to extract convolutional features from the text region to obtain an initial feature image. The initial feature image is then reduced in dimension using the pooling layer of the convolutional neural network to obtain a reduced-dimensional feature image. The reduced-dimensional feature image is then output using the fully connected layer of the convolutional neural network to obtain a feature image. A long short-term memory network is used to identify text position sequences from the feature image to generate initial text content. Specifically, the state value of the feature image is calculated using the input gate of the long short-term memory network, the activation value of the feature image is calculated using the forget gate of the long short-term memory network, and a state update value of the feature image is calculated based on the state value and activation value. The text position sequence of the state update value is calculated using the output gate of the long short-term memory network to generate initial text content. The temporal classification network is used to align the characters of the initial text content to generate the corresponding text content.

[0050] S202: Determine the row target text box composed of multiple target text text boxes in the same row based on the coordinate information of the target text text box, obtain the row target text boxes corresponding to different rows, divide the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to different rows, and for any sub-table, calculate the fitting curve of the row target text box based on the coordinate information of the corresponding row target text box in the sub-table.

[0051] In step S202, M is an integer greater than 1. According to the coordinate information of the text box of the target text, the row target text box composed of multiple target text text boxes in the same row is determined, that is, the position corresponding to the column title of each column in the table is determined. According to the row target text boxes corresponding to different rows, the form image to be reconstructed is divided into M sub-tables, that is, each column in each sub-table contains a target text box with a title, and each sub-table contains target texts with the same text. For any sub-table, according to the coordinate information of the corresponding row target text box in the sub-table, the fitting curve of the row target text box is calculated, that is, the curvature degree of each sub-table is determined.

[0052] In this embodiment, based on the coordinate information of the target text text boxes, a row target text box consisting of multiple target text text boxes in the same row is determined. The vertical coordinates of the target text text boxes are extracted. When the difference between the vertical coordinates of the target text text boxes is less than a preset threshold, the corresponding target text text boxes are considered to be in the same row. Then, based on the horizontal coordinates, the corresponding text boxes are sorted from left to right to obtain the row target text boxes. Based on the row target text boxes, the form image to be reconstructed is divided into M sub-tables, such that each sub-table contains a row target text box. For each sub-table, a fitting curve for the row target text box is calculated based on the coordinate information of the corresponding row target text box in the sub-table.

[0053] It should be noted that when the sub-table in the form image to be reconstructed is divided into left and right sub-tables, that is, the same target text row includes two identical target texts, for example, including two equal "project title (pro-title)", "project English abbreviation title (enpro-title)", "check value title (val-title)", "reference range title (range-title)", "unit title (unit-title)", "project (pro)", "project English abbreviation (enpro)", "check value (val)", "reference range (range)", "unit (unit)", "other", etc., it is necessary to determine the position of the corresponding sub-table. For example, the corresponding text boxes are divided into two groups according to the target text, and the mean of the horizontal coordinates in each group of text boxes can be calculated. When the minimum value of the horizontal coordinates of the text boxes in the row corresponding group is greater than the mean, the target text of the corresponding group is divided into the right half of the sub-table, and the text boxes of the other group corresponding to the target text are divided into the left half of the sub-table.

[0054] Optionally, the form image to be reconstructed is divided into M sub-forms according to row target text boxes corresponding to different rows, including:

[0055] For any row target text box, determine the adjacent row target text box adjacent to the row, extract the text box between the row target text box and the adjacent row target text box, and the column coordinate of the text box in the row target text box is smaller than the column coordinate of the text box in the adjacent row target text box;

[0056] The text box between the row target text box and the adjacent row target text box, as well as the row target text box, is determined as a subtable, and all row target text boxes are traversed to obtain M subtables.

[0057] In this embodiment, after the row target text box, each text box is traversed starting from the next row until the traversal reaches the text box of the corresponding target text again, and the text box between the row target text box and the adjacent row target text box is extracted, wherein the column coordinates of the text box in the row target text box are smaller than the column coordinates of the text box in the adjacent row target text box, and the text box between the row target text box and the adjacent row target text box, as well as the row target text box are determined as a subtable, and all row target text boxes are traversed to obtain M subtables, wherein the subtable consists of the row target text box and the corresponding text box.

[0058] For example, each subtable includes one or more target texts including "project title (pro-title)", "project abbreviation title (enpro-title)", "check value title (val-title)", "reference range title (range-title)", "unit title (unit-title)", "project (pro)", "project abbreviation (enpro)", "check value (val)", "reference range (range)", "unit (unit)", and "other".

[0059] Optionally, calculating a fitting curve of a row target text box according to coordinate information of a corresponding row target text box in the subtable includes:

[0060] Calculate the center coordinates of each target text in the target text box of the row;

[0061] Connect the center point coordinates of the text box of each target text to obtain the fitting curve of the row target text box.

[0062] In this embodiment, the coordinates of the text box of each target text are calculated, the center point coordinates of the text box of each target text in the row target text box are calculated, the center point coordinates of the text box of each target text are connected, and the connected curve is used as the fitting curve of the row target text box, that is, the local slope curve in the row target text box.

[0063] S203: Extract multiple column text boxes that are in the same column as the text box of the target text. For any column text box, construct a directed graph corresponding to the column text box based on a preset spacing. Determine the candidate alignment text box corresponding to the column text box based on the directed graph. Traverse all column text boxes to obtain the candidate alignment text box corresponding to each column text box.

[0064] In step S203, the target text is used as a column label, and the text content corresponding to the column label is extracted. Each target text may contain multiple text contents, and a text box corresponding to the text content is obtained, that is, the column text box that is in the same column as the text box of the target text is extracted. For any column text box, a directed graph corresponding to the column text box is constructed according to a preset spacing, wherein the directed graph is composed of texts related to the text content corresponding to the column text box. According to the directed graph, the candidate alignment text box corresponding to the column text box is determined, and all column text boxes are traversed to obtain the candidate alignment text box corresponding to each column text box.

[0065] In this embodiment, first, the form image to be reconstructed can be initially adjusted according to the slope of the fitting curve corresponding to the sub-table to obtain the form image to be reconstructed after the initial adjustment. In the form image to be reconstructed after the initial adjustment, the text box corresponding to each column is extracted to obtain multiple column text boxes in the same column as the text box of the target text.

[0066] For any column text box, according to the preset spacing, a directed graph corresponding to the column text box is constructed, wherein the directed graph is composed of column text boxes in adjacent columns. For example, for a column text box in the first column, the directed graph of the corresponding column text box is constructed by finding the candidate alignment text box associated with the corresponding column text box in the second column. According to the preset spacing, the mean of the vertical coordinates in the corresponding column text box is used as the base point, and the preset distance is moved up and down respectively to obtain the height of the directed graph. According to the height, the column text box in the second column within the corresponding height range is selected as the candidate alignment text box, wherein the column text box in the second column within the corresponding height range can be There are multiple candidate alignment text boxes, that is, there can be multiple candidate alignment text boxes. Each column text box in the first column is traversed to obtain the directed graph corresponding to each column text box. In this embodiment, the preset spacing can be 1.5 times the length of the corresponding column text box; for a column text box in the second column, the directed graph of the corresponding column text box is constructed by finding the candidate alignment text box associated with the corresponding column text box in the third column. According to the preset spacing, the mean of the vertical coordinates in the corresponding column text box is used as the base point, and the preset distances are moved up and down respectively to obtain the height of the directed graph. According to the height, the column text box in the third column within the corresponding height range is selected as the candidate alignment text box.

[0067] For example, if the target text in the first column is the project name, and the text content of the column text box is creatinine, fasting blood glucose, uric acid and TG / HDL, the target text in the second column is the abbreviation of the project name. In the second column, if the text content of the column text box that forms a directed graph with creatinine is CREA, GLU and UA, the text content of the column text box that forms a directed graph with fasting blood glucose is GLU and UA, the text content of the column text box that forms a directed graph with uric acid is UA, and the text content of the column text box that forms a directed graph with TG / HDL is empty, then the candidate alignment text boxes corresponding to creatinine are CREA, GLU and UA, the candidate alignment text boxes corresponding to fasting blood glucose are GLU and UA, the candidate alignment text box corresponding to uric acid is UA, and the candidate alignment text box corresponding to TG / HDL is empty.

[0068] Optionally, after extracting multiple column text boxes in the same column as the text box of the target text according to the coordinate information of the text box of the target text, the method further includes:

[0069] Calculate the row spacing between text boxes in adjacent columns to obtain multiple row spacing values;

[0070] According to multiple row spacing values, the median of the row spacing values ​​is calculated. If the row spacing value is greater than the median, empty text boxes are filled between adjacent column text boxes corresponding to the row spacing value to obtain multiple filled column text boxes.

[0071] In this embodiment, in order to ensure that each column text box has a corresponding candidate alignment text box when constructing a directed graph, new text boxes are filled between adjacent column text boxes with a larger distance. When filling, the line spacing between adjacent column text boxes is calculated to obtain multiple line spacing values. Based on the multiple line spacing values, the median of the line spacing values ​​is calculated, and the median is used as a judgment condition for determining whether the line spacing between adjacent column text boxes is a larger distance. If the line spacing value is greater than the median, empty text boxes are filled between the adjacent column text boxes corresponding to the line spacing value to obtain multiple column text boxes after filling. Using the median as the judgment condition, the spacing between the column text boxes in the corresponding column can be accurately analyzed.

[0072] S204: For any column, determine the candidate combination consisting of the corresponding column text box and the candidate alignment text box, filter the candidate combination according to the preset filtering conditions, and obtain the K optimal combinations after filtering. According to the preset search algorithm and the K optimal combinations, search for the text box in the same row as the corresponding column text box to obtain the row text box in the row where the column text box is located, and obtain the reconstructed row text in the form image to be reconstructed according to the text content in each text box in the row text box, where K is an integer greater than 1.

[0073] In step S204, based on the beam search mechanism, a text box that is in the same row as the corresponding column text box is selected from the candidate alignment text boxes to obtain the row text box in the row where the column text box is located. According to the text content in each text box in the row text box, the reconstructed row text in the form image to be reconstructed is obtained.

[0074] In this embodiment, for example, if the target text in the first column is the project name, and the text content of the column text box is creatinine, fasting blood glucose, uric acid and TG / HDL, the target text in the corresponding second column is the abbreviation of the project name. In the second column, if the text content of the column text box that forms a directed graph with creatinine is CREA, GLU and UA, the text content of the column text box that forms a directed graph with fasting blood glucose is GLU and UA, the text content of the column text box that forms a directed graph with uric acid is UA, and the text content of the column text box that forms a directed graph with TG / HDL is empty, traverse the column text boxes in the first column to obtain the candidate alignment text boxes corresponding to the column text boxes in the first column. For candidate alignment text boxes corresponding to any column text box, the column text box and the candidate alignment text box are combined to obtain corresponding candidate combinations. For example, the candidate combination corresponding to creatinine and CREA is (creatinine, CREA), the candidate combination corresponding to creatinine and GLU is (creatinine, GLU), the candidate combination corresponding to creatinine and CREA is (creatinine, CREA), and the candidate combination corresponding to creatinine and UA is (creatinine, UA). Creatinine corresponds to three candidate combinations: (creatinine, CREA), (creatinine, GLU), and (creatinine, UA). Fasting blood glucose corresponds to two candidate combinations: (fasting blood glucose, GLU) and (fasting blood glucose, UA). Uric acid corresponds to one candidate combination: (uric acid, UA), and TG / HDL corresponds to one candidate combination. Therefore, there are 3*2*1*1=6 candidate combinations of candidate alignment text box combinations corresponding to the first column.

[0075] Since each column text box corresponds to a row text box, the six candidate combinations are initially screened to obtain initial screening candidate combinations that meet the conditions. During the initial screening, adjacent column text boxes are combined with only one corresponding column text box to form a candidate combination.

[0076] For example, when creatinine and CREA form the corresponding candidate combination (creatinine, CREA), fasting blood glucose and GLU form the corresponding candidate combination (fasting blood glucose, GLU), uric acid can only form the corresponding candidate combination (uric acid, UA) with UA, and TG / HDL can only form the corresponding candidate combination (TG / HDL, empty); when creatinine and CREA form the corresponding candidate combination (creatinine, CREA), fasting blood glucose and UA form the corresponding candidate combination (fasting blood glucose, UA), uric acid can only form the corresponding candidate combination (uric acid, empty), TG / HDL can only form the corresponding candidate combination (uric acid, empty). Candidate combination (TG / HDL, empty); when creatinine and GLU form a candidate combination (creatinine, GLU), fasting blood glucose can only form a candidate combination with UA (fasting blood glucose, UA), TG / HDL can only form a candidate combination with empty (TG / HDL, empty); when creatinine and UA form a candidate combination (creatinine, UA), fasting blood glucose can only form a candidate combination with GLU (fasting blood glucose, GLU), uric acid can only form a candidate combination with empty (uric acid, empty), TG / HDL can only form a candidate combination with empty (TG / HDL, empty). After initial screening, there are 4 candidate combinations.

[0077] The optimal combination is selected from the four candidate combinations. Based on the preset search algorithm and the K optimal combinations, a text box in the same row as the corresponding column text box is searched to obtain the row text box in the row where the column text box is located. The preset search algorithm is a beam search algorithm. When searching for the third column text box based on the text boxes in the first and second columns, a text box matching each optimal combination is selected in the third column based on the K optimal candidate combinations selected in the first and second columns. For example, an optimal combination in the first and second columns is (creatinine, CREA), (fasting blood glucose, GLU), (uric acid, UA), (TG / HDL, empty). Based on the optimal combination, a text box matching the optimal combination is searched in the third column. The searches are performed sequentially to obtain the row text box corresponding to each row. Based on the text content in each text box in the row text box, the reconstructed row text in the form image to be reconstructed is obtained.

[0078] Optionally, K optimal combinations are selected from the candidate combinations, including:

[0079] Calculate the distance variance between the text box and the corresponding fitting curve in each candidate combination;

[0080] Sort the distance variances from small to large, and select the candidate combinations corresponding to the first K distance variances as the optimal combination.

[0081] In this embodiment, when selecting K optimal combinations, the distance variance between the text box in each candidate combination and the corresponding fitting curve is calculated. For example, for one candidate combination among the four candidate combinations in the above method (creatinine, CREA), (fasting blood glucose, GLU), (uric acid, UA), and (TG / HDL, empty), the distance difference between the text boxes corresponding to creatinine, CREA, fasting blood glucose, GLU, uric acid, UA, and TG / HD and the fitting curve in the corresponding subtable is calculated, and the distance variance is calculated based on the distance difference. The smaller the variance, the more stable the distance between the text box in the candidate combination and the fitting curve. Therefore, the distance variance between the text box and the corresponding fitting curve in each candidate combination is calculated respectively, the distance variance is sorted from small to large, and the candidate combinations corresponding to the top K distance variances are selected as the optimal combinations.

[0082] It should be noted that when calculating the distance between the corresponding text box and the fitting curve, the distance between the center point of each text box and the fitting curve is calculated.

[0083] Obtain the form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct a text box that wraps the corresponding text content, and filter out the target text and the text box corresponding to the target text from the N text contents. N is an integer greater than 1. According to the coordinate information of the text box of the target text, determine the row target text box composed of multiple target text text boxes in the same row, and obtain the row target text boxes corresponding to different rows. According to the row target text boxes corresponding to different rows, divide the form image to be reconstructed into M sub-tables. For any sub-table, calculate the fitting curve of the row target text box according to the coordinate information of the corresponding row target text box in the sub-table. M is an integer greater than 1. Extract the text box corresponding to the target text. For multiple column text boxes in the same column, for any column text box, a directed graph corresponding to the column text box is constructed according to a preset spacing. Based on the directed graph, the candidate alignment text box corresponding to the column text box is determined. All column text boxes are traversed to obtain the candidate alignment text box corresponding to each column text box. For any column, a candidate combination consisting of the corresponding column text box and the candidate alignment text box is determined. According to the preset screening conditions, the candidate combination is screened to obtain the K optimal combinations after screening. According to the preset search algorithm and the K optimal combinations, the text box in the same row as the corresponding column text box is searched to obtain the row text box in the row where the column text box is located. According to the text content in each text box in the row text box, the reconstructed row text in the form image to be reconstructed is obtained, where K is an integer greater than 1. In the present application, a corresponding directed graph is constructed according to the coordinates of the text boxes in the table, a candidate combination in the same row as the text box is selected from the directed graph, and the text box in the corresponding row is searched according to the candidate combination to obtain the row text box in the row where the corresponding text box is located. This improves the row reconstruction accuracy in the table, extracts the text content after row reconstruction, and thus improves the table text extraction accuracy.

[0084] See also Figure 3 , Figure 3 Schematic diagram of the structure of a table row reconstruction device for a form image provided by an embodiment of the present invention. In this embodiment, the terminal includes various units for executing Figure 2 Each step in the corresponding embodiment. Please refer to Figure 2 as well as Figure 2 For the sake of convenience, only the parts related to this embodiment are shown. Figure 3 As shown, the table row reconstruction device 30 includes: an acquisition module 31 , a division module 32 , a construction module 33 , and a search module 34 .

[0085] An acquisition module 31 is configured to acquire a form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct text boxes enclosing the corresponding text contents, and select target text and a text box corresponding to the target text from the N text contents, where N is an integer greater than 1;

[0086] a division module 32 for determining, based on the coordinate information of the text boxes of the target text, a row target text box consisting of multiple target text text boxes in the same row, obtaining row target text boxes corresponding to different rows, dividing the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to different rows, and calculating, for any sub-table, a fitting curve for the row target text box based on the coordinate information of the corresponding row target text box in the sub-table, where M is an integer greater than 1;

[0087] A construction module 33 is configured to extract multiple column text boxes that are in the same column as the text box of the target text, construct a directed graph corresponding to each column text box based on a preset spacing, determine a candidate alignment text box corresponding to the column text box based on the directed graph, and traverse all column text boxes to obtain a candidate alignment text box corresponding to each column text box;

[0088] The search module 34 is used to determine, for any column, a candidate combination consisting of a corresponding column text box and a candidate alignment text box, screen the candidate combinations according to preset screening conditions, and obtain K optimal combinations after screening. According to a preset search algorithm and the K optimal combinations, the module searches for a text box in the same row as the corresponding column text box to obtain a row text box in the row where the column text box is located, and obtains the reconstructed row text in the form image to be reconstructed based on the text content in each text box in the row text box, where K is an integer greater than 1.

[0089] Optionally, the acquisition module 31 includes:

[0090] A detection unit, configured to perform text detection on the form image to be reconstructed, and obtain a text region representing text in the form image to be reconstructed;

[0091] The recognition unit is used to perform text recognition on the text area to obtain the text content of the text in the text area.

[0092] Optionally, the division module 32 includes:

[0093] an extraction unit configured to determine, for any row target text box, an adjacent row target text box adjacent to the row, and extract text boxes between the row target text box and the adjacent row target text box, wherein the column coordinates of the text box in the row target text box are smaller than the column coordinates of the text box in the adjacent row target text box;

[0094] The determining unit is used to determine the row target text box and the text box between the adjacent row target text box, and the row target text box as a subtable, and traverse all the row target text boxes to obtain M subtables.

[0095] Optionally, the division module 32 includes:

[0096] A calculation unit, used for calculating the center point coordinates of each target text in the row target text box;

[0097] The connection unit is used to connect the center point coordinates of the text box of each target text to obtain the fitting curve of the row target text box.

[0098] Optionally, the table row reconstruction device 30 further includes:

[0099] A row spacing calculation module is used to calculate the row spacing between adjacent column text boxes and obtain multiple row spacing values;

[0100] The judgment module is used to calculate the median of the row spacing values ​​according to multiple row spacing values. If the row spacing value is greater than the median, empty text boxes are filled between adjacent column text boxes corresponding to the row spacing value to obtain multiple filled column text boxes.

[0101] Optionally, the search module 34 includes:

[0102] A variance calculation unit, used to calculate the distance variance between the text box and the corresponding fitting curve in each candidate combination;

[0103] The sorting unit is used to sort the distance variances from small to large and select the candidate combinations corresponding to the first K distance variances as the optimal combination.

[0104] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0105] Figure 4This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. Figure 4 As shown, the terminal device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps of the table row reconstruction method for any of the above-mentioned form images are implemented.

[0106] The terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 4 The terminal device is merely an example and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0107] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or any conventional processor.

[0108] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the internal memory of the terminal device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the terminal device. In other embodiments, it can also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the terminal device. Furthermore, the memory can also include both the internal storage unit of the terminal device and an external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of computer programs. The memory can also be used to temporarily store data that has been output or is about to be output.

[0109] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0110] The present invention can implement all or part of the processes in the above-mentioned embodiment method, and can also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiment when executing it.

[0111] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0112] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0113] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment based on actual needs.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A table row reconstruction method for a form image, characterized in that: The table row reconstruction method includes: Obtaining a form image to be reconstructed, performing text recognition on the form image to be reconstructed to obtain N text contents in the form image to be reconstructed, constructing a text box enclosing the corresponding text contents, and filtering out target text and a text box corresponding to the target text from the N text contents, where N is an integer greater than 1; Determining, based on the coordinate information of the text box of the target text, a row target text box composed of multiple target text text boxes in the same row, obtaining row target text boxes corresponding to different rows, dividing the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to the different rows, and calculating, for any sub-table, a fitting curve of the row target text box based on the coordinate information of the corresponding row target text box in the sub-table, where M is an integer greater than 1; Extracting multiple column text boxes in the same column as the text box of the target text, constructing a directed graph corresponding to any column text box based on a preset spacing, determining candidate alignment text boxes corresponding to the column text box based on the directed graph, and traversing all column text boxes to obtain a candidate alignment text box corresponding to each column text box; For any column, determine the candidate combination consisting of the corresponding column text box and the candidate alignment text box, filter the candidate combination according to the preset filtering conditions, and obtain the K optimal combinations after filtering. According to the preset search algorithm and the K optimal combinations, search for the text box in the same row as the corresponding column text box to obtain the row text box in the row where the column text box is located. According to the text content in each text box in the row text box, obtain the reconstructed row text in the form image to be reconstructed, where K is an integer greater than 1.

2. The table row reconstruction method according to claim 1, wherein: The performing text recognition on the form image to be reconstructed to obtain N text contents in the form image to be reconstructed includes: Performing text detection on the form image to be reconstructed to obtain a text region representing text in the form image to be reconstructed; Perform text recognition on the text area to obtain the text content of the text in the text area.

3. The table row reconstruction method according to claim 1, wherein: The step of dividing the form image to be reconstructed into M sub-tables according to the row target text boxes corresponding to the different rows includes: For any row target text box, determine the adjacent row target text box adjacent to the row, extract the text box between the row target text box and the adjacent row target text box, and the column coordinates of the text box in the row target text box are smaller than the column coordinates of the text box in the adjacent row target text box; The text box between the row target text box and the adjacent row target text box, and the row target text box are determined as a subtable, and all row target text boxes are traversed to obtain M subtables.

4. The table row reconstruction method according to claim 1, wherein: The step of calculating the fitting curve of the row target text box according to the coordinate information of the row target text box corresponding to the subtable includes: Calculate the center coordinates of each target text in the target text box of the row; Connect the center point coordinates of the text box of each target text to obtain the fitting curve of the row target text box.

5. The table row reconstruction method according to claim 1, wherein: After extracting a plurality of text boxes in the same column as the text box of the target text according to the coordinate information of the text box of the target text, the method further includes: Calculate the row spacing between text boxes in adjacent columns to obtain multiple row spacing values; According to the multiple row spacing values, the median of the row spacing values ​​is calculated. If the row spacing value is greater than the median, empty text boxes are filled between adjacent column text boxes corresponding to the row spacing value to obtain multiple filled column text boxes.

6. The table row reconstruction method according to claim 1, wherein: The candidate combinations are screened according to the preset screening conditions to obtain K optimal combinations after screening, including: Calculate the distance variance between the text box and the corresponding fitting curve in each candidate combination; The distance variances are sorted from small to large, and candidate combinations corresponding to the first K distance variances are selected as the optimal combinations.

7. A table row reconstruction device for a form image, characterized in that: The table row reconstruction device comprises: an acquisition module, configured to acquire a form image to be reconstructed, perform text recognition on the form image to be reconstructed, obtain N text contents in the form image to be reconstructed, construct a text box enclosing the corresponding text contents, and filter out target text and a text box corresponding to the target text from the N text contents, where N is an integer greater than 1; a division module, configured to determine, based on the coordinate information of the text boxes of the target text, a row target text box composed of multiple target text text boxes in the same row, obtain row target text boxes corresponding to different rows, divide the form image to be reconstructed into M sub-tables based on the row target text boxes corresponding to the different rows, and for any sub-table, calculate a fitting curve for the row target text box based on the coordinate information of the corresponding row target text box in the sub-table, where M is an integer greater than 1; A construction module is configured to extract multiple column text boxes in the same column as the text box of the target text, construct a directed graph corresponding to any column text box based on a preset spacing, determine candidate alignment text boxes corresponding to the column text box based on the directed graph, and traverse all column text boxes to obtain a candidate alignment text box corresponding to each column text box; The search module is used to determine, for any column, a candidate combination consisting of a corresponding column text box and a candidate alignment text box, screen the candidate combinations according to preset screening conditions, and obtain K optimal combinations after screening. According to a preset search algorithm and the K optimal combinations, the module searches for a text box in the same row as the corresponding column text box to obtain a row text box in the row where the column text box is located, and obtains the reconstructed row text in the form image to be reconstructed based on the text content in each text box in the row text box, where K is an integer greater than 1.

8. The table row reconstruction device according to claim 7, wherein: The acquisition module includes: a detection unit, configured to perform text detection on the form image to be reconstructed, and obtain a text region representing text in the form image to be reconstructed; The recognition unit is used to perform text recognition on the text area to obtain the text content of the text in the text area.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the table row reconstruction method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the table row reconstruction method according to any one of claims 1 to 6 is implemented.

Images