Method for retrieving information in table picture, electronic device and storage medium

Abstract

This application provides a method, electronic device, and storage medium for retrieving information from table images. The method includes: using the table image as input to a text detection model to obtain the position information of text row candidate boxes output by the text detection model; identifying the strings contained within the text row candidate boxes using a text recognition model based on the position information of the text row candidate boxes; clustering the text content corresponding to each cell based on the position information of the text row candidate boxes corresponding to each string; calculating the matching degree between the received search statement and the text content corresponding to each cell based on the matching degree; and filtering out matching target text based on the matching degree. This solution only needs to identify the text content to complete the information retrieval from the table image, saving computing power, money, and time. It also solves the problems of inaccurate existing structured recognition methods, inability to recognize complex table types, and low universality.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method for retrieving information from table documents and images, as well as an electronic device and a computer-readable storage medium. Background Technology

[0002] With the continuous development of technology, various industries have higher requirements for work efficiency, leading to the rapid development of the RPA (Robotic Process Automation) industry. In the field of table entry retrieval, existing methods typically apply to PDF and Excel versions of table files, with limited research on entry retrieval using image tables (tables in image format).

[0003] A typical workflow is as follows: 1. The table image needs to be structured using OCR (Optical Character Recognition). 2. Fuzzy matching of key sentences is performed based on the structured recognition results. However, OCR structured recognition consumes a significant amount of computing power and time. Summary of the Invention

[0004] This application provides a method for retrieving information from table images to save computing power and time.

[0005] This application provides a method for retrieving information from a table image, including:

[0006] The table image is used as input to the text detection model to obtain the position information of the candidate boxes for text lines output by the text detection model.

[0007] Based on the position information of the text line candidate box, the string contained in the text line candidate box is identified by the text recognition model;

[0008] Based on the position information of the candidate text boxes corresponding to each string, clustering is used to obtain the text content corresponding to each cell;

[0009] Based on the received search query, calculate the matching degree between the search query and the text content corresponding to each cell, and filter out the matching target text based on the matching degree.

[0010] In one embodiment, before using the table image as input to the text detection model and obtaining the position information of the text line candidate boxes output by the text detection model, the method further includes:

[0011] Get the original image of the table document;

[0012] The original image of the table document is subjected to text orientation recognition, and the original image of the table document with text orientation not in the standard orientation is rotated to obtain a preprocessed image;

[0013] The table image is obtained from the preprocessed image.

[0014] In one embodiment, obtaining the table image from the preprocessed image includes:

[0015] Determine whether the aspect ratio of the preprocessed image is greater than a first preset value;

[0016] If the aspect ratio is greater than a preset value, multiple images of the table with an aspect ratio less than or equal to a first preset value and an overlap distance within a second preset range are cropped from the preprocessed image.

[0017] In one embodiment, before using the table image as input to the text detection model and obtaining the position information of the text line candidate boxes output by the text detection model, the method further includes:

[0018] Get the original image of the table document;

[0019] Determine whether the aspect ratio of the original image of the table document is greater than a first preset value;

[0020] If the aspect ratio is greater than a preset value, multiple images of the table are cropped from the original table document image, with an aspect ratio less than or equal to the first preset value and an overlap distance within the second preset range.

[0021] In one embodiment, the step of clustering to obtain the text content corresponding to each cell based on the position information of the text line candidate boxes corresponding to each string includes:

[0022] Based on the position information of the text line candidate boxes corresponding to each string, a clustering algorithm is used to divide all text line candidate boxes into multiple clusters.

[0023] The strings in the candidate text boxes within the same cluster are concatenated to obtain the text content contained in the same cell.

[0024] In one embodiment, the step of calculating the matching degree between the received search query and the text content corresponding to each cell, and filtering out matching target text based on the matching degree, includes:

[0025] For each piece of text content, perform word segmentation on the search query and the text content;

[0026] Calculate the similarity between the search query and the text content based on the phrases contained in the search query and the phrases contained in the text content;

[0027] Text content with a similarity greater than a threshold is selected as target text that matches the search query.

[0028] In one embodiment, the method further includes:

[0029] Highlight the candidate text box containing the target text.

[0030] This application also provides a method for retrieving information from a table image, including:

[0031] The table image is used as input to the text detection model to obtain the position information of the candidate boxes for text lines output by the text detection model.

[0032] Based on the position information of the text line candidate box, the string contained in the text line candidate box is identified by the text recognition model;

[0033] Based on the received search statement, calculate the matching degree between the search statement and the string contained in each text candidate box, and filter out the target string that matches the search statement;

[0034] Based on the position information of the candidate text boxes corresponding to each target string, the target strings are clustered to obtain the text content that matches the search statement.

[0035] On the other hand, embodiments of this application also provide a device for retrieving information from a table image, the device comprising:

[0036] The text detection module is used to take a table image as input to the text detection model and obtain the position information of the candidate boxes for text lines output by the text detection model.

[0037] The text recognition module is used to recognize the string contained in the text line candidate box by means of a text recognition model based on the position information of the text line candidate box;

[0038] The text clustering module is used to cluster the text content of each cell based on the position information of the candidate text lines corresponding to each string.

[0039] The statement matching module is used to calculate the matching degree between the received search statement and the text content corresponding to each cell, and to filter out the matching target text based on the matching degree.

[0040] This application embodiment also provides another device for retrieving information from a table image, the device comprising:

[0041] The text detection module is used to take a table image as input to the text detection model and obtain the position information of the candidate boxes for text lines output by the text detection model.

[0042] The text recognition module is used to recognize the string contained in the text line candidate box by means of a text recognition model based on the position information of the text line candidate box;

[0043] The text matching module is used to calculate the matching degree between the received search statement and the string contained in each text candidate box, and to filter out the target string that matches the search statement.

[0044] The text clustering module is used to cluster the target strings to obtain text content that matches the search statement based on the position information of the candidate text boxes corresponding to each target string.

[0045] This application also provides an electronic device, the electronic device comprising:

[0046] processor;

[0047] Memory used to store processor-executable instructions;

[0048] The processor is configured to perform a method for retrieving information from the aforementioned table image.

[0049] This application also provides a computer-readable storage medium storing a computer program that can be executed by a processor to complete the method for retrieving information from the above-mentioned table images.

[0050] The solution provided in this application uses a text detection model and a text recognition model to output the strings contained in the candidate text boxes. Then, based on the position information of the strings, the text content of the same cell is clustered. By calculating the match between the search statement and the text content of each cell, the matching target text is determined. There is no need to perform structured recognition on the table image. This application only needs to recognize the text content to complete the information retrieval of the table image, saving computing power, money and time. It also solves the problems of inaccuracy of existing structured recognition, inability to meet the recognition of complex types of tables, and low universality. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below.

[0052] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0053] Figure 2This is a flowchart illustrating a method for retrieving information from a table image provided in an embodiment of this application;

[0054] Figure 3 This is an illustration of a table image with text oriented vertically.

[0055] Figure 4 This is a schematic diagram of a table image with an aspect ratio greater than the first preset value;

[0056] Figure 5 yes Figure 2 Detailed flowchart of step S240 in the corresponding embodiment;

[0057] Figure 6 This is a schematic diagram of the overall process of the method for retrieving information from a table image provided in the embodiments of this application;

[0058] Figure 7 This is a flowchart illustrating a method for retrieving information from a table image according to another embodiment of this application;

[0059] Figure 8 This is a schematic diagram of a list of leased properties;

[0060] Figure 9 This is a schematic diagram of a financing business registration form;

[0061] Figure 10 This is a block diagram of an information retrieval device in a table image, as shown in an embodiment of this application;

[0062] Figure 11 This is a block diagram of a device for retrieving information from a table image, as shown in another embodiment of this application. Detailed Implementation

[0063] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0064] Similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0065] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 100 can be used to execute the information retrieval method for tabular images provided in an embodiment of this application. For example... Figure 1As shown, the electronic device 100 includes: one or more processors 102 and one or more memories 104 storing processor-executable instructions. The processors 102 are configured to execute a method for retrieving information from the tabular images provided in the following embodiments of this application.

[0066] The processor 102 may be a gateway, a smart terminal, or a device that includes a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data from other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.

[0067] The memory 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the method for retrieving information from the table image described below. Various application programs and various data may also be stored on the computer-readable storage medium, such as various data used and / or generated by the application programs.

[0068] In one embodiment, Figure 1 The illustrated electronic device 100 may also include an input device 106, an output device 108, and a data acquisition device 110, these components being interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device 100 may also have other components and structures as needed.

[0069] The input device 106 may be a device used by a user to input commands, and may include one or more of a keyboard, mouse, microphone, and touchscreen. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. The data acquisition device 110 may acquire images of objects and store the acquired images in the memory 104 for use by other components. For example, the data acquisition device 110 may be a camera.

[0070] In one embodiment, the components in the example electronic device 100 used to implement the method for retrieving information in the table images of the present application can be integrated or distributed. For example, the processor 102, memory 104, input device 106 and output device 108 can be integrated into one unit, while the data acquisition device 110 can be separated.

[0071] In one embodiment, the example electronic device 100 for implementing the method for retrieving information in a table image according to the embodiments of this application can be implemented as a smart terminal such as a smartphone, tablet computer, desktop computer, laptop computer, server, vehicle device, etc.

[0072] Figure 2 This is a flowchart illustrating a method for retrieving information from a table image provided in an embodiment of this application. For example... Figure 2 As shown, the method includes the following steps S210-S240.

[0073] Step S210: Use the table image as input to the text detection model to obtain the position information of the text line candidate boxes output by the text detection model.

[0074] A table image refers to a table in image format. Table images can be stored locally or received from an external device. The text detection model can be a model pre-trained using algorithms such as PixelLink or Faster R-CNN before step S210. A text line candidate box refers to the smallest bounding rectangle of a text line. Position information can be represented by the coordinates of the top-left and bottom-right vertices of the text line candidate boxes. After obtaining the table image, it can be input into the table text detection model to obtain the position information of the text line candidate boxes whose confidence level is greater than a threshold.

[0075] In one embodiment, the original image of the table document can be processed to obtain a table image. The original image of the table document refers to the original image containing the table image. Processing the original image of the table document may include orientation correction and cropping.

[0076] In one embodiment, the original image of the table document can be obtained; text direction recognition can be performed on the original image of the table document, and the original image of the table document with text direction not in the standard direction can be rotated to obtain a preprocessed image; the table image can be obtained from the preprocessed image.

[0077] In one embodiment, the text direction may include both vertical and horizontal directions, such as... Figure 3The text direction shown can be considered vertical. Assuming the standard direction is horizontal by default, if the original table document's text direction is vertical, then the original table document needs to be rotated to change the text direction to horizontal. For clarity, the rotated image can be called a pre-processed image.

[0078] The text orientation recognition algorithm can use a machine learning algorithm similar to the random forest classifier, and the training samples can be table sample images with known text orientations.

[0079] In another embodiment, based on the similarity of each pair of adjacent rows in the original table document image, adjacent rows with a similarity reaching a first preset similarity threshold are merged to obtain a similar row table; based on the similarity of each pair of adjacent columns in the original table document image, adjacent columns with a similarity reaching a second preset similarity threshold are merged to obtain a similar list table; the number of rows and columns in the original table document image, the number of rows in the similar row table, and the number of columns in the similar list table are input into a pre-trained machine learning model to obtain the text direction of the original table document image. The similarity calculation involves converting the text content of two adjacent rows or columns into feature vectors, and calculating the distance between these feature vectors to obtain the similarity between the two adjacent rows or columns.

[0080] In other embodiments, the text orientation may be 0°, 90°, 180°, and 270°. The original image of the table document can be input into a trained recognition model, which outputs probability values ​​corresponding to the four categories, representing the probability that the original image of the table document will be rotated clockwise by that angle. The highest probability is selected as the classification result. For example, if the probability value of 0° is the highest, it means the text orientation is in the standard direction and does not need to be rotated. If the probability value of 90° is the highest, it means the text orientation is not in the standard direction and needs to be rotated 90° clockwise, and so on.

[0081] As needed, the preprocessed image can be directly used as a table image. In other embodiments, it can be further determined whether the aspect ratio of the preprocessed image is greater than a first preset value; if the aspect ratio is greater than the preset value, multiple table images with aspect ratios less than or equal to the first preset value and overlapping distances within a second preset range are cropped from the preprocessed image.

[0082] The aspect ratio refers to the ratio of length to width. The first preset value can be 3. If the aspect ratio is greater than 3, it indicates that the preprocessed image is a long, narrow image, and the size is not quite right, so the preprocessed image can be cropped. For example... Figure 4The image shown is a preprocessed image with an aspect ratio greater than a first preset value. This preprocessed image can be cropped into three table images (see labels 41, 42, and 43). The cropping requirement is that the aspect ratio of each table image must be less than or equal to the first preset value, and the overlap distance between adjacent table images must be within a second preset range. The overlap distance (see label 40) refers to the width of the overlapping portion between two adjacent table images. The second preset range can be 50-60 pixels.

[0083] In other embodiments, the text orientation of the original table document image can be disregarded. After obtaining the original table document image, it can be directly determined whether the aspect ratio of the original table document image is greater than a first preset value. If the aspect ratio is greater than the preset value, multiple table images with aspect ratios less than or equal to the first preset value and overlapping distances within a second preset range can be cropped from the original table document image. Specific cropping methods are described above and will not be repeated here.

[0084] Step S220: Based on the position information of the text line candidate box, identify the string contained in the text line candidate box using a character recognition model.

[0085] Based on the location information of each text line candidate box, the region containing each candidate box is located, and the text recognition model identifies the string contained in that region of the image. The text recognition model is used to identify the specific content contained within the text line candidate box, which may include Chinese characters, letters, symbols, or numbers. The specific content contained within the text line candidate box can be referred to as a string. The text recognition model can be a model pre-trained using CTC (Connectionist Temporal Classification) and attention mechanism algorithms.

[0086] Step S230: Based on the position information of the candidate text lines corresponding to each string, cluster to obtain the text content corresponding to each cell.

[0087] The position information of each text line candidate box refers to the position information of the text line candidate box containing the string. The position information of the text line candidate box is determined by step S210 above. A cell may contain one or more text line candidate boxes; that is, a cell may contain one or more lines of strings. Text content refers to the text formed by concatenating strings within the same cell. Since the Euclidean distance between text line candidate boxes within the same cell is relatively small, while the Euclidean distance between text line candidate boxes in different cells is relatively large, text line candidate boxes with similar Euclidean distances can be grouped into the same class (cluster) based on their position information. The strings within text line candidate boxes grouped into the same class can be considered as the text content within the same cell.

[0088] In one embodiment, based on the position information of the text line candidate boxes corresponding to each string, all text line candidate boxes can be divided into multiple clusters using a clustering algorithm (e.g., K-means clustering). The strings within the text line candidate boxes of the same cluster are then concatenated to obtain the text content contained in the same cell.

[0089] The number of clusters can be determined based on the number of cells, which can be determined based on the number of rows and columns entered by the user, or obtained through the text detection model in step S210.

[0090] Specifically, step 1: Use the center point coordinates of the text line candidate boxes as samples, and randomly select K (number of cells) points from all samples as initial centroids (completely random); step 2: Calculate the Euclidean distance of each sample to each centroid, and assign the samples to the cluster corresponding to the nearest centroid; step 3: Calculate the mean of all samples in each cluster, and use the mean to update the centroid of the cluster; repeat steps 2 and 3 until one of the following conditions is met: the position change of the centroid is less than the specified threshold (default is 0.0001), reaching the maximum number of iterations.

[0091] Through the above clustering process, all text line candidate boxes can be divided into multiple clusters. The strings of text line candidate boxes within the same cluster can be concatenated in a top-to-bottom order to form the text content of a single cell.

[0092] Step S240: Based on the received search statement, calculate the matching degree between the search statement and the text content corresponding to each cell, and filter out the matching target text based on the matching degree.

[0093] The search query can be a query entered by the user or sent by an external device. The matching degree can be represented by the similarity between the search query and the text content. Text content with a similarity greater than a threshold can be considered as target text content that matches the search query.

[0094] In one embodiment, such as Figure 5 As shown, the above step S240 specifically includes: step S241-step S242.

[0095] Step S241: For each text content, perform word segmentation on the search statement and the text content.

[0096] Specifically, for the word segmentation operation, it can first be determined whether the retrieval statement contains proper nouns. If it contains proper nouns (a proper noun is the name of a specific entity, such as a person, an organization name, a location, or a professional term (such as aspirin in the medical field)), the proper noun can be first grouped into one word and removed from the retrieval statement, and then the remaining statement is segmented based on the n-gram (n-ary language model). N represents the number of characters in a word segment. For example, for "China Agricultural University", if n = 2 is set, the word segmentation candidates are "China", "Country Agriculture", "Agriculture", "Industry University", "University", etc. Since the probabilities of "Country Agriculture" and "Industry University" appearing in the language model library (a knowledge base that stores the probabilities of different word groups) are relatively low, then according to the probability, "China Agricultural University" will be segmented into "China", "Agriculture", and "University". Similarly, the word segmentation operation is performed on each text content.

[0097] Step S242: Calculate the similarity between the retrieval statement and the text content according to the word groups included in the retrieval statement and the word groups included in the text content.

[0098] Performing a word segmentation operation on the retrieval statement can obtain multiple word groups, and performing a word segmentation operation on the text content can also obtain multiple word groups. The similarity S between the retrieval statement and the text content can be calculated using the following formula where A represents the number of word groups included in the retrieval statement but not present in the text content; B represents the number of word groups included in the text content but not present in the retrieval statement; and C represents the number of identical word groups included in both.

[0099] Step S243: Screen out the text content with a similarity greater than the threshold as the target text that matches the retrieval statement.

[0100] For example, the threshold can be 0.4 - 0.5, and the text content with a similarity greater than the threshold can be considered as the target text that matches the retrieval statement. In one embodiment, the text line candidate box where the target text is located can be highlighted. For example, the text line candidate box where the target text is located can be highlighted.

[0101] Figure 6 It is the overall flow schematic diagram of the retrieval method for the information in the table picture provided by the embodiments of the present application. As Figure 6As shown in the figure, (1) First, preprocess the original table document picture. The preprocessing method may include orientation correction and cropping to obtain a table picture. (2) Perform text detection on the table picture through the above step S210 to obtain the position information of the candidate text line boxes. (3) Perform character recognition on the characters within the candidate text line boxes through the above step S220 to obtain the strings contained in each candidate text line box. (4) Through the above step S230, cluster the text content of each cell according to the position information of the candidate text line box corresponding to each string. For example, "56, 65, 78, 87 2 Unit 902" indicates that the text content of the cell with the upper left vertex coordinates (56, 65) and the lower right vertex coordinates (78, 87) is "2 Unit 902". (5) Perform fuzzy matching on the received retrieval statement and the text content of each cell through the above step S240. Then, highlight the candidate text line boxes where the matched text content is located.

[0102] It should be noted that assume a cell contains three lines of strings, and the character recognition results are "China Agricultural University", "School of Engineering", and "Room 203". Assume no clustering is performed, and the three lines of strings are matched separately. When the retrieval statement is "Classroom of the School of Science, Shanxi Agricultural University", through word segmentation operation, it can be split into "Shanxi", "Agriculture", "University", "Science", "School", and "Classroom". When calculating the matching degree between the retrieval statement and "School of Engineering", "School of Engineering" will be split into "Engineering" and "School of Engineering", so the matching degree between the retrieval statement and "School of Engineering" is relatively low, and "School of Engineering" is ignored by the algorithm. However, through the clustering method in the above embodiment, these three strings "China Agricultural University", "School of Engineering", and "Room 203" can be merged into a text content "Room 203, School of Engineering, China Agricultural University", directly perform word segmentation on this text content to obtain "China", "Agriculture", "University", "School of Engineering", "203", and "Classroom", and use the formula to calculate the matching degree between this text content and the retrieval statement. Among them, A represents the number of phrases contained in the retrieval statement and not present in the text content; B represents the number of phrases contained in the text content and not present in the retrieval statement; C represents the number of identical phrases contained in both. Assume the value of S is greater than the threshold, then the entire text content will be retrieved, thus avoiding the omission of strings and making the retrieval results of the table picture more accurate.

[0103] Figure 7 is a schematic flowchart of a method for retrieving information in a table picture provided by another embodiment of the present application. Different from the Figure 2 corresponding embodiment, Figure 7 in the corresponding embodiment, after completing text detection and character recognition, fuzzy matching is first performed, and then clustering is performed. Specifically, as Figure 7As shown, it includes the following steps S710-S740.

[0104] Step S710: Use the table image as input to the text detection model to obtain the position information of the text line candidate boxes output by the text detection model.

[0105] Step S720: Based on the position information of the text line candidate box, identify the string contained in the text line candidate box using a character recognition model.

[0106] The specific implementation process of steps S710 and S720 above can be referred to Figure 2 Steps S210-S220 of the corresponding embodiment.

[0107] Step S730: Based on the received search statement, calculate the matching degree between the search statement and the string contained in each text candidate box, and filter out the target string that matches the search statement.

[0108] Matching degree can be the similarity between the search query and the string. The target string refers to the string whose similarity to the search query is greater than a threshold; it is called the target string for differentiation. In one embodiment, the search query and the string can be converted into word vectors respectively, and the cosine similarity between the word vectors can be calculated as the matching degree between the search query and the string.

[0109] In one embodiment, step S730 may specifically include: performing word segmentation on the search statement and the string for each string; calculating the similarity between the search statement and the string based on the word groups contained in the search statement and the string; and selecting strings with similarity greater than a threshold as target strings that match the search statement.

[0110] The word segmentation process can be referenced above. First, proper nouns are divided into words, and then the remaining sentences are segmented using n-grams (n-gram language models). As mentioned above, the similarity S between the search statement and the string can be calculated based on... The similarity is calculated as follows. In this embodiment, A represents the number of phrases contained in the search query but not in the string; B represents the number of phrases contained in the string but not in the search query; and C represents the number of identical phrases contained in both. The similarity threshold can be 0.4-0.5. For example, the threshold can be 0.45. If the similarity between a string and the search query is greater than 0.45, then that string is the target string.

[0111] Step S740: Based on the position information of the text line candidate boxes corresponding to each target string, cluster the target strings to obtain text content that matches the search statement.

[0112] The position information of each target string's corresponding text line candidate box refers to the position information of the text line candidate box containing the target string, which can be obtained in step S710. The position information of the text line candidate box can be represented by the coordinates of the center point of the text line candidate box. Based on the position information of the text line candidate boxes corresponding to each target string, clustering algorithms such as k-means can be used to divide all target strings' corresponding text line candidate boxes into multiple clusters. The strings within the text line candidate boxes of the same cluster are concatenated to obtain the text content matching the search query.

[0113] The specific clustering process is described above. Through this process, all candidate text lines corresponding to the target string can be divided into multiple clusters. The strings of candidate text lines within the same cluster can be concatenated from top to bottom to form the text content of the same cell. This text content is the text content that matches the search query. This text content can be highlighted as needed.

[0114] The above embodiments reduce the number of samples during clustering by first filtering the target string through fuzzy matching and then performing clustering, thereby reducing the amount of computation, shortening the computation time, and improving computational efficiency.

[0115] Figure 8 This is a schematic diagram of a rental property list. Rental property lists are characterized by containing complex characters (including symbols, numbers, words, and letters) and a large amount of document information. Whether using structured recognition algorithms based on natural language processing or template matching based on manually applied rules (such as graph learning neural networks), the final results are unsatisfactory and consume significant computing power and time.

[0116] Figure 9 This is a schematic diagram of a financing business registration form. In forms like this one, which involve information registration, the key-value pair logic is quite complex, making structured recognition difficult. Performing structured recognition on the entire form solely for information retrieval is not a cost-effective approach.

[0117] for Figure 8 and Figure 9Such table images can be processed using the solution provided in this application embodiment. By using a text detection model and a text recognition model, the strings contained in the candidate text rows are output. Then, based on the position information of the strings, the text content of the same cell is clustered. By calculating the matching reading between the search statement and the text content of each cell, the matching target text is determined. There is no need to perform structured recognition on the table image (meaning that it is output in the form of a table). This application embodiment only needs to recognize the text content to complete the information retrieval of the table image, saving computing power, money and time. It also solves the problems of inaccuracy of existing structured recognition, inability to meet the recognition of complex types of tables, and low universality.

[0118] The following are embodiments of the apparatus described in this application, which can be used to execute the method embodiment for retrieving information from the table images described above. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiment for retrieving information from the table images described in this application.

[0119] Figure 10 This is a block diagram illustrating an information retrieval device from a table image, as shown in an embodiment of this application. Figure 10 As shown, the device includes:

[0120] The text detection module 1010 is used to take a table image as input to the text detection model and obtain the position information of the text line candidate boxes output by the text detection model.

[0121] The text recognition module 1020 is used to recognize the string contained in the text line candidate box by means of a text recognition model based on the position information of the text line candidate box;

[0122] The text clustering module 1030 is used to cluster the text content corresponding to each cell based on the position information of the candidate text lines corresponding to each string.

[0123] The statement matching module 1040 is used to calculate the matching degree between the received search statement and the text content corresponding to each cell, and to filter out the matching target text based on the matching degree.

[0124] Figure 11 This is a block diagram illustrating an information retrieval device from a table image, as shown in an embodiment of this application. Figure 11 As shown, the device includes:

[0125] The text detection module 1110 is used to take a table image as input to the text detection model and obtain the position information of the text line candidate boxes output by the text detection model.

[0126] The text recognition module 1120 is used to recognize the string contained in the text line candidate box by means of a text recognition model based on the position information of the text line candidate box;

[0127] The text matching module 1130 is used to calculate the matching degree between the received search statement and the string contained in each text candidate box, and to filter out the target string that matches the search statement.

[0128] The text clustering module 1140 is used to cluster the target strings to obtain text content that matches the search statement based on the position information of the text line candidate boxes corresponding to each target string.

[0129] The specific implementation process of the functions and roles of each module in the above-mentioned device can be found in the corresponding steps of the information retrieval method in the table and image above, and will not be repeated here.

[0130] The apparatuses and methods disclosed in the several embodiments provided in this application can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0131] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0132] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A method of retrieving information in a tabular picture, characterized by, include: The table image is used as input to the text detection model to obtain the position information of the candidate boxes for text lines output by the text detection model. Based on the position information of the text line candidate box, the string contained in the text line candidate box is identified by the text recognition model; Based on the position information of the candidate text boxes corresponding to each string, clustering is used to obtain the text content corresponding to each cell; Based on the received search query, calculate the matching degree between the search query and the text content corresponding to each cell, and filter out the matching target text based on the matching degree; The step of clustering to obtain the text content corresponding to each cell based on the position information of the candidate text lines corresponding to each string includes: Based on the position information of the text line candidate boxes corresponding to each string, a clustering algorithm is used to divide all text line candidate boxes into multiple clusters. Concatenate the strings in the candidate text boxes within the same cluster to obtain the text content contained in the same cell; The process involves dividing all text line candidate boxes into multiple clusters based on their position information, using a clustering algorithm. This includes: using the center point coordinates of the text line candidate boxes as samples, randomly selecting K points from all samples as initial centroids; calculating the Euclidean distance from each sample to each centroid, and assigning the sample to the cluster corresponding to the nearest centroid; calculating the mean of all samples within each cluster, and using this mean to update the centroid of the cluster; repeating the above steps until the position change of the centroid is less than a specified threshold or the maximum number of iterations is reached.

2. The method of claim 1, wherein, Before using the table image as input to the text detection model and obtaining the position information of the text line candidate boxes output by the text detection model, the method further includes: Get the original image of the table document; The original image of the table document is subjected to text orientation recognition, and the original image of the table document with text orientation not in the standard orientation is rotated to obtain a preprocessed image; The table image is obtained from the preprocessed image.

3. The method of claim 2, wherein, Obtaining the table image from the preprocessed image includes: Determine whether the aspect ratio of the preprocessed image is greater than a first preset value; If the aspect ratio is greater than a preset value, multiple images of the table with an aspect ratio less than or equal to a first preset value and an overlap distance within a second preset range are cropped from the preprocessed image.

4. The method of claim 1, wherein, Before using the table image as input to the text detection model and obtaining the position information of the text line candidate boxes output by the text detection model, the method further includes: Get the original image of the table document; Determine whether the aspect ratio of the original image of the table document is greater than a first preset value; If the aspect ratio is greater than a preset value, multiple images of the table are cropped from the original table document image, with an aspect ratio less than or equal to the first preset value and an overlap distance within the second preset range.

5. The method of claim 1, wherein, The step of calculating the matching degree between the received search query and the text content corresponding to each cell, and filtering out matching target text based on the matching degree, includes: For each piece of text content, perform word segmentation on the search query and the text content; Calculate the similarity between the search query and the text content based on the phrases contained in the search query and the phrases contained in the text content; Text content with a similarity greater than a threshold is selected as target text that matches the search query.

6. The method of claim 1, wherein, The method further includes: Highlight the candidate text box containing the target text.

7. A method of retrieving information in a tabular picture, characterized by, include: The table image is used as input to the text detection model to obtain the position information of the candidate boxes for text lines output by the text detection model. Based on the position information of the text line candidate box, the string contained in the text line candidate box is identified by the text recognition model; Based on the received search statement, calculate the matching degree between the search statement and the string contained in each text candidate box, and filter out the target string that matches the search statement; Based on the position information of the candidate text boxes corresponding to each target string, the target strings are clustered to obtain text content that matches the search statement; Based on the position information of the candidate text lines corresponding to each target string, the target strings are clustered to obtain text content that matches the search query, including: Based on the position information of the text line candidate boxes corresponding to each target string, a clustering algorithm is used to divide all text line candidate boxes into multiple clusters. Concatenate the strings in the candidate text boxes within the same cluster to obtain the text content contained in the same cell; The process involves dividing all text line candidate boxes into multiple clusters based on their position information, using a clustering algorithm. This includes: using the center point coordinates of the text line candidate boxes as samples, randomly selecting K points from all samples as initial centroids; calculating the Euclidean distance from each sample to each centroid, and assigning the sample to the cluster corresponding to the nearest centroid; calculating the mean of all samples within each cluster, and using this mean to update the centroid of the cluster; repeating the above steps until the position change of the centroid is less than a specified threshold or the maximum number of iterations is reached.

8. An electronic device, comprising: The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to execute the method for retrieving information from a table image as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that can be executed by a processor to perform the method for retrieving information from a table image as described in any one of claims 1-7.

Images