Data extraction method and device, computing equipment and storage medium

Abstract

The invention provides a data extraction method and device, computing equipment and a storage medium, and belongs to the technical field of image processing. According to the data extraction method provided by the invention, the range of the borderless table in the to-be-recognized image is determined based on the anchor point field in the regular template, then the rows and columns of the field frames are determined based on the position relationship between the field frames, and the data are extracted according to the rows and columns of the field frames. In the frameless table, the overlapping range of the data belonging to the same column in the transverse direction is relatively large, and the overlapping range of the data belonging to the same row in the longitudinal direction is relatively large, so that the method provided by the invention can determine which field frames belong to the same column and which field frames belong to the same row according to the position relationship between the field frames. Compared with data extraction according to fixed data position information, the data extraction method provided by the invention can avoid data extraction errors caused by data position deviation such as data line feed and data dislocation, and improves the accuracy of data reading.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a data extraction method, apparatus, computing device, and storage medium. Background Art

[0002] In business activities, borderless tables are widely used to record data. Driven by the digital transformation, more and more enterprises and institutions adopt automated recognition technology to extract data from borderless tables to improve work efficiency.

[0003] Currently, the data extraction method usually determines the specific coordinate positions of the data in the borderless table based on a pre-established image recognition template, intercepts the image corresponding to the specific coordinate positions, and then identifies the image as a string through an optical character recognition system.

[0004] The above data extraction method extracts data according to the fixed data position information in the image recognition template. In real life, the positions of the data in the image may deviate and do not exactly match the data position information in the image recognition template. Therefore, the above data extraction method cannot avoid data extraction errors caused by deviations in data positions such as data line breaks and data misalignment, resulting in a low accuracy rate of data extraction. Summary of the Invention

[0005] Embodiments of this application provide a data extraction method, apparatus, computing device, and storage medium, which can avoid data extraction errors caused by deviations in data positions such as data line breaks and data misalignment, and improve the accuracy rate of data extraction. The technical solution is as follows.

[0006] In a first aspect, a data extraction method is provided. The method includes: determining a plurality of anchor field boxes from a plurality of field boxes of an image to be recognized based on the anchor fields indicated by a regular template, where each field box corresponds to the position of a string in the image to be recognized, and the plurality of anchor field boxes match the anchor fields; determining a plurality of first field boxes located within the borderless table of the image to be recognized from the plurality of field boxes based on the plurality of anchor field boxes; determining the columns of the plurality of first field boxes and determining the rows of the plurality of first field boxes based on the positional relationship between the plurality of first field boxes and the plurality of anchor field boxes; and extracting the data in the plurality of first field boxes based on the rows and columns of the plurality of first field boxes.

[0007] The data extraction method provided by the embodiments of the present application determines the range of the borderless table in the image to be recognized based on the anchor fields in the regular template, then determines the rows and columns of the field boxes based on the positional relationship between the field boxes, and extracts data according to the rows and columns of the field boxes. Since in a borderless table, the data belonging to the same column has a large overlapping range horizontally, and the data belonging to the same row has a large overlapping range vertically, therefore, the data extraction method provided by the embodiments of the present application can determine which field boxes belong to the same column and which field boxes belong to the same row according to the positional relationship between the field boxes. Compared with extracting data based on fixed data position information, the data extraction method provided by the embodiments of the present application can avoid data extraction errors caused by deviations in data positions such as data line breaks and data misalignment, and improve the accuracy of data reading.

[0008] In some embodiments, the above-mentioned determining, from multiple field boxes, multiple first field boxes located within the borderless table of the image to be recognized based on multiple anchor field boxes includes: determining the upper and lower boundaries of the borderless table based on the upper and lower boundaries of the multiple anchor field boxes; and determining multiple first field boxes located within the borderless table of the image to be recognized based on the upper and lower boundaries of the borderless table. Since only the relevant information of a small number of anchor fields in the image is included in this regular template, the structure of this regular template is simple, the production cost is low, the applicable image range is wider, and only a very small number of fixed fields in the image to be recognized are needed to realize the range recognition of the borderless table.

[0009] In some embodiments, the above-mentioned determining multiple first field boxes located within the borderless table of the image to be recognized based on the upper and lower boundaries of the borderless table includes: determining the field boxes other than the target field box within the upper and lower boundaries of the borderless table as the first field box, where the target field box satisfies a form condition; the form condition is any one of the following: the string of the field box includes a preset string, or the closest distance between the field box and multiple other field boxes in a preset direction is greater than or equal to a distance threshold, or the format of the string of the field box is different from the preset format of the borderless table. Through the above form conditions, the field boxes that do not belong to the borderless table can be removed, and the accuracy of data extraction can be improved.

[0010] In some embodiments, determining the columns of the multiple first field boxes includes: for any first field box, based on the left and right boundaries of the first field box and the multiple anchor field boxes, obtaining the overlap ratio between the first field box and any anchor field box, where the overlap ratio is the proportion of the overlapping area between the first field box and the anchor field box in the union of the left and right boundaries of the first field box and the anchor field box; using the columns of the anchor field boxes with an overlap ratio greater than or equal to the ratio threshold as the columns of the first field box. Determining the columns of the first field box based on the size of the overlapping area between the first field box and the anchor field box in the left and right directions can more accurately determine the columns of the misaligned data and improve the accuracy of data reading.

[0011] In some embodiments, determining the columns of the multiple first field boxes includes: for any anchor field box, if there are multiple second field boxes in the column of the anchor field box, using the smallest left boundary among the multiple second field boxes as the left boundary of the anchor field box, where the second field box is a first field box with a left boundary smaller than that of the anchor field box; if there are multiple third field boxes in the column of the anchor field box, using the largest right boundary among the multiple third field boxes as the right boundary of the anchor field box, where the third field box is a first field box with a right boundary larger than that of the anchor field box; determining the columns of the multiple first field boxes based on the left and right boundaries of the anchor field box. Refreshing the left and right boundaries of the anchor field box can more accurately determine the columns of the data with more severe misalignment and improve the accuracy of data reading.

[0012] In some embodiments, determining the columns of the multiple first field boxes includes: for any first field box, if the overlap ratio between the first field box and at least two anchor field boxes is greater than or equal to the ratio threshold, then determining the column of the anchor field box with the largest overlap ratio among the at least two anchor field boxes as the column of the first field box. This step has a higher accuracy in selecting the column with a larger overlap ratio for the case where the overlapping areas between the misaligned data and multiple table headers are all large.

[0013] In some embodiments, determining the columns of the multiple first field boxes includes: for any first field box, if the overlap ratio between the first field box and any anchor field box is less than the ratio threshold, then obtaining the distances between the first field box and the multiple anchor field boxes; determining the column of the anchor field box closest to the first field box as the column of the first field box. This step has a higher accuracy in selecting the closest column for the case where the overlapping areas between the misaligned data and any table header are all small.

[0014] In some embodiments, the above-mentioned determining the rows of the multiple first field boxes includes: based on the multiple anchor field boxes and the multiple first field boxes, determining multiple straight lines, the multiple straight lines being parallel to the connecting lines of the center points of the multiple anchor field boxes and passing through the reference anchor point, the reference anchor point being the center point of the field box in the preset column, and the preset column being the column corresponding to a column of data in the borderless table where data line breaks do not occur; based on the multiple straight lines, determining the rows of the multiple first field boxes. Determining the row direction with the anchor field box as the reference and then dividing the rows in the borderless table with the center point of the first field box where data line breaks do not occur as the reference can effectively avoid the situation of taking line-breaking data as a separate row and improve the accuracy of data reading.

[0015] In some embodiments, the above-mentioned determining the rows of the multiple first field boxes based on the multiple straight lines includes: for any first field box, if the first field box is penetrated by a straight line, determining the row of the straight line as the row of the first field box; if the first field box is not penetrated by a straight line, determining the row of the straight line closest to the first field box as the row of the first field box. The straight lines identify the rows in the borderless table, and determining the row of the first field box according to the distance between the first field box and these straight lines can more accurately find the rows of the line-breaking data and improve the accuracy of data reading.

[0016] In some embodiments, before the above-mentioned determining the multiple first field boxes located within the borderless table of the to-be-recognized image from the multiple field boxes based on the multiple anchor field boxes, the method further includes: adjusting the positions of the multiple field boxes of the to-be-recognized image based on the positions of the anchor fields in the regular template and the multiple anchor field boxes. Adjusting the to-be-recognized image based on the anchor field boxes can improve the accuracy of recognition.

[0017] In some embodiments, the above-mentioned regular template includes information of a specified anchor field, and the specified anchor field includes the header field of the borderless table in the to-be-recognized image.

[0018] In a second aspect, a data extraction device is provided for performing the above-mentioned data extraction method. Specifically, the data extraction device includes functional modules for performing the data extraction method provided in the first aspect or any optional manner of the first aspect above.

[0019] In a third aspect, a computing device or a computing device cluster is provided. The computing device includes a processor, and the processor is used to execute program code so that the computing device or the computing device cluster performs operations to implement the operations performed by the above-mentioned data extraction method.

[0020] In a fourth aspect, a computer-readable storage medium is provided. At least one program code is stored in the storage medium, and the program code is read by the processor to enable the computing device to perform the operations performed by the above-mentioned data extraction method.

[0021] In a fifth aspect, a computer program product or a computer program is provided. The computer program product or the computer program includes program code stored in a computer-readable storage medium. A processor of a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code to cause the computing device to execute the method provided in the first aspect or various alternative implementations of the first aspect.

[0022] Based on the implementations provided in the above aspects of this application, further combinations can be made to provide more implementations. Description of the Drawings

[0023] Figure 1 is a schematic structural diagram of a data extraction system provided by an embodiment of this application;

[0024] Figure 2 is a schematic structural diagram of a computing device provided by an embodiment of this application;

[0025] Figure 3 is a schematic structural diagram of a computing device cluster provided by an embodiment of this application;

[0026] Figure 4 is a flowchart of a data extraction method provided by an embodiment of this application;

[0027] Figure 5 is an example diagram of an image to be recognized provided by an embodiment of this application;

[0028] Figure 6 is a flowchart of another data extraction method provided by an embodiment of this application;

[0029] Figure 7 is a processing flowchart of data extraction provided by an embodiment of this application;

[0030] Figure 8 is a flowchart of a method for determining the range of a borderless table provided by an embodiment of this application;

[0031] Figure 9 is a flowchart of a method for determining columns provided by an embodiment of this application;

[0032] Figure 10 is a flowchart of a method for determining rows provided by an embodiment of this application;

[0033] Figure 11 is a schematic structural diagram of a data extraction device provided by an embodiment of this application. Detailed Embodiments

[0034] To make the objectives, technical solutions and advantages of this application more clear, the following will further describe in detail the embodiments of this application with reference to the accompanying drawings.

[0035] Figure 1 FIG. 4 is a schematic structural diagram of a data extraction system provided according to an embodiment of this application, which is used to apply the data extraction method provided by the embodiment of this application. As Figure 1 shown, the data extraction system includes a terminal 101 and a server 102, and the terminal 101 is connected to the server 102. Figure 1 The number of the shown terminals and servers is only for illustration, and the embodiment of this application does not limit the number of terminals and servers in the data extraction system.

[0036] The terminal 101 is used to obtain an image to be recognized and send the image to be recognized to the server 102. The terminal 101 includes, but is not limited to: scanners, smart phones, smart TVs, TV boxes, keyboards, mice, game pads, smart wearable devices, tablet computers, portable personal computers, mobile Internet devices, etc. The embodiment of this application does not limit the type of the terminal 101. The server 102 is used to perform data extraction on the image to be recognized according to the data extraction method provided by the embodiment of this application. The server 102 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), big data and artificial intelligence platforms.

[0037] In the embodiment of this application, the terminal 101 obtains an image to be recognized and sends the image to be recognized to the server 102. The server 102 performs data extraction on the image to be recognized and returns the data extraction result to the terminal 101. In some embodiments, the terminal 101 also has a data extraction function and can independently complete the process of obtaining an image to be recognized and performing data extraction on the image to be recognized.

[0038] Figure 2 FIG. 5 is a schematic structural diagram of a computing device provided by the embodiment of this application. The computing device 200 may be the server 102 in the above data extraction system, or may be the terminal 101 with a data extraction function in the above data extraction system. The embodiment of this application does not limit this. It should be understood that the computing device introduced below can implement any function in any of the following methods. Generally, the computing device 200 includes: a processor 201 and a memory 202.

[0039] The processor 201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 201 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 201 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the central processing unit (CPU); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 201 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 201 may further include an artificial intelligence (AI) processor, and the AI processor is used to process computational operations related to machine learning.

[0040] The memory 202 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 202 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 202 is used to store at least one program, and the at least one program is used to be executed by the processor 201 to implement the data extraction method provided in the method embodiments of the present application.

[0041] In some embodiments, the computing device 200 may further optionally include: a peripheral device interface 203 and at least one peripheral device. The processor 201, the memory 202, and the peripheral device interface 203 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 203 through a bus, signal lines, or a circuit board.

[0042] In the embodiments of the present application, the computing device 200 can be configured as a terminal or a server, and the terminal or the server is used as the execution subject to implement the technical solutions provided in the method embodiments of the present application.

[0043] In some embodiments, the computing device 200 may be a portable mobile terminal, such as: a smart phone, a tablet computer, a Moving Picture Experts Group Audio Layer IV (MP4) player, a laptop computer, or a desktop computer. The computing device 200 may also be referred to by other names such as user equipment, portable terminal, laptop terminal, desktop terminal, etc.

[0044] In some embodiments, the computing device 200 may be an independent physical server, or may be implemented as a cluster of computing devices as Figure 3 shown, that is, a server cluster or a distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Taking the computing device as a cloud server as an example, the computing device may also be referred to as a cloud platform (i.e., the abbreviation of the cloud computing platform), which refers to a service based on hardware resources and software resources, providing computing, network, and storage capabilities. Through the network "cloud", huge data calculations are processed remotely and then returned to the user, with characteristics such as large scale, distribution, virtualization, high availability, scalability, on-demand service, and security. The cloud platform can achieve the rapid allocation and release of configurable computing resources with a relatively small management cost or a relatively low interaction complexity between the user and the service provider.

[0045] The image to be recognized may be an electronic bill, a bill image, various documents with borderless tables, pictures, tables, or any image including a borderless table, such as a document borderless table, a Portable Document Format (PDF) borderless table, a picture borderless table, etc. Taking a bill as an example, in order to make the layout of the bill cleaner and easier to read, various bills (such as value-added tax invoices, receipts, and expense reports, etc.) use borderless tables extensively to arrange and organize data, for clearly displaying different items and data categories. The data extraction method provided by the embodiments of the present application is directed to the data extraction of any image containing a borderless table.

[0046] Figure 4 is a flowchart of a data extraction method provided according to an embodiment of the present application. As Figure 4 shown, the following takes the server executing the above data extraction method as an example for description. The method includes the following steps:

[0047] 401. The server determines multiple anchor field boxes from multiple field boxes of the image to be recognized based on the anchor fields indicated by the regular template. Each field box corresponds to the position of a string in the image to be recognized, and the multiple anchor field boxes match the anchor fields.

[0048] In the embodiments of the present application, the server matches the anchor fields in the regular template with the fields in the field boxes of the image to be recognized one by one to obtain multiple anchor field boxes.

[0049] Among them, the anchor field is a partial fixed field in the standard image, including the fields corresponding to the table headers in different regions of the image. The field box is used to identify each field recognized by the server from the image based on a pre-trained recognition model. The field box indicates the coordinates and text content of the corresponding field. The coordinates can be expressed as the vertex coordinates or the center coordinates of the field box, etc., which are not limited in the embodiments of the present application. The regular template is a template used to indicate multiple anchor fields in the image to be recognized. The regular template includes information about multiple specified anchor fields in the standard image. The information of the anchor field includes the regular expression of the anchor field, the field coordinates, the field arrangement direction (horizontal or vertical), and the region to which the field belongs (upper left / lower left / upper right / lower right). The regular template can be used to identify the anchor field boxes in the image to be recognized, so as to adjust the image to be recognized based on the anchor field boxes and divide the region of the image to be recognized to locate the borderless table. Among them, the standard image refers to an image placed upright, that is, an image in which the rows and columns are parallel to the horizontal and vertical coordinates of the image respectively. The regular expression, the field arrangement direction, and the region to which the field belongs are all used to find a field box in the image to be recognized that matches the information of the anchor field, and this field box is used as the anchor field box. Among them, successfully matching the information of the anchor field means that the field in the field box satisfies the regular expression of the anchor field, the arrangement direction of the field in the field box is the same as the field arrangement direction of the anchor field, and the coordinates of the field box in the image to be recognized are in the region to which the anchor field belongs. The field coordinates are the coordinates of the anchor field in the standard image and are used to adjust the coordinates of the field box in the image to be recognized.

[0050] In some embodiments, the process of obtaining a field box that successfully matches the information of the anchor field is as follows: find a field box that satisfies the regular expression of the anchor field from the image to be recognized through regular matching. If the field arrangement direction and the region to which the field belongs of this field box are both the same as the field arrangement direction and the region to which the field belongs of the anchor field, then this field box is used as the anchor field box. Take Figure 5Taking the image to be recognized as an example, in the corresponding regular template, the field arrangement direction of the anchor field "Specification Model" is horizontal, and the field belongs to the upper left area. For the anchor field "Specification Model", the server matches the regular expression of "Specification Model" in the regular template with multiple field boxes of the image to be recognized one by one, and finds the field box including "Specification Model". If the "Specification Model" in this field box is arranged horizontally and this field box is located in the upper left of the image to be recognized, then this field box is the anchor field box that successfully matches the anchor field "Specification Model".

[0051] In the embodiments of the present application, the anchor fields are divided into perspective transformation anchor fields, table header anchor fields, and lower boundary anchor fields. Among them, the perspective transformation anchor field can be used for perspective transformation to adjust the coordinates of the field box in the image to be recognized, and the lower boundary anchor field box can be used to indicate the lower boundary of the borderless table. The table header anchor field is the table header field of the borderless table, which can be used to indicate the table header of the borderless table to determine the upper boundary of the borderless table, and can also divide the columns of the borderless table. Based on this, in the regular template, the information of the table header anchor field also includes the label corresponding to this table header anchor field, and each label corresponds to an array, which is used to store the data in the column to which the table header anchor field belongs.

[0052] Taking Figure 5 the image to be recognized shown as an example, Figure 5 As shown is a bill image. In Figure 5 the corresponding regular template, "Purchaser", "Password Area", "Seller", "Remarks", "Project Name" above Area A, "Specification Model", "Unit", etc., and "Total" below Area A in the figure are all anchor fields. Among them, "Purchaser", "Password Area", "Seller", and "Remarks" can be used for perspective transformation, that is, these fields belong to the above-mentioned perspective transformation anchor fields. The fields such as "Project Name", "Specification Model", and "Unit" above Area A are the table headers of the borderless table, which can be used to indicate the upper boundary of the borderless table, that is, these fields belong to the above-mentioned table header anchor fields. "Total" below Area A can be used to indicate the lower boundary of the borderless table, that is, these fields belong to the above-mentioned lower boundary anchor fields.

[0053] It should be noted that the anchor fields of the image to be recognized shown above Figure 5 are only examples. In some embodiments, for different types of images, different anchor fields can be selected. For example, the designer can select any four fixed fields from the image as the perspective transformation anchor fields, and select any fixed field closest to the lower boundary of the borderless table as the lower boundary anchor field.

[0054] The regular template provided by the embodiment of the present application has a relatively high level of abstraction and only includes the relevant information of a small number of anchor point fields in the image. Therefore, the structure of the regular template is simple, the production cost is low, and the applicable range is wider. Only a very small number of fixed fields in the image to be recognized are required to extract the data in the entire borderless table of the image to be recognized. Taking bills as an example, the data extraction method and highly abstract regular template used in this process can process the borderless tables in various types of bills, and can not only extract the fields in ordinary "horizontal" bills, but also extract the fields in "vertical" roll tickets, and can also extract the data in self-constructed bills.

[0055] In the embodiment of the present application, the above regular template is designed based on a standard image. For different types of images, the designer summarizes the fixed fields in this type of image and writes the information of these fixed fields in the standard image into the regular template to obtain multiple regular templates for different types of images. For example, for different types of bills, there may be different anchor point fields between their regular templates, or, for different types of bills, their regular templates include different anchor point fields, and the coordinates of the anchor point fields are also different.

[0056] 402. The server determines, based on multiple anchor point field frames, multiple first field frames located within the borderless table of the image to be recognized from multiple field frames.

[0057] Among them, the first field frame is the field frame corresponding to the data to be extracted in the borderless table. In the embodiment of the present application, the anchor point field frames that match the information of the header anchor point field and the lower boundary anchor point field among the multiple anchor point field frames can be used to indicate the upper and lower boundaries of the borderless table. Therefore, based on these anchor point field frames, it can be determined which field frames are within the upper and lower boundaries of the borderless table.

[0058] 403. The server determines the columns of the multiple first field frames and determines the rows of the multiple first field frames based on the positional relationship between the multiple first field frames and the multiple anchor point field frames.

[0059] In the embodiment of the present application, the anchor point field frames that match the information of the header anchor point field among the multiple anchor point field frames can divide the columns in the borderless table and determine the row direction in the borderless table. The positional relationship between the field frames includes the size of the overlapping range of the field frames in the horizontal and vertical directions, which can reflect the subordinate relationship between the field frames. Therefore, based on the positional relationship between the anchor point field frames that match the information of the header anchor point field and the first field frames, the rows and columns to which the first field frames belong can be determined. For example, Figure 5 if the overlapping range of "Project Name" and "Optical Instrument * Lens" in the horizontal direction is large, then "Optical Instrument * Lens" belongs to the column of "Project Name".

[0060] 404. The server extracts data from multiple first field boxes based on the rows and columns of the multiple first field boxes.

[0061] In the embodiment of the present application, the server extracts data from the borderless table in the form of an array based on the rows and columns to which the multiple first field boxes belong. That is, the data in the first field box is filled into the corresponding position of the corresponding array. Specifically, if the first field box belongs to the column of which header anchor field, the data in the first field box is filled into the array of the label corresponding to the header anchor field. If the first field box belongs to the nth row, the data in the first field box is filled into the nth position of the corresponding array.

[0062] The data extraction method provided by the embodiment of the present application determines the range of the borderless table in the image to be recognized based on the anchor fields in the regular template, then determines the rows and columns of the field boxes based on the positional relationship between the field boxes, and extracts data according to the rows and columns of the field boxes. Since in a borderless table, the data belonging to the same column has a large overlapping range horizontally, and the data belonging to the same row has a large overlapping range vertically, therefore, the data extraction method provided by the embodiment of the present application can determine which field boxes belong to the same column and which field boxes belong to the same row according to the positional relationship between the field boxes. Compared with extracting data according to fixed data position information, the data extraction method provided by the embodiment of the present application can avoid data extraction errors caused by data line breaks and data misalignment and other deviations in data positions, and improve the accuracy of data reading.

[0063] Figure 6 is a flowchart of a data extraction method provided by an embodiment of the present application. Figure 7 is a processing flowchart of data extraction provided by an embodiment of the present application. As Figure 7 shown, the data extraction method includes: determining the range of the borderless table, performing column clustering on the borderless table, and performing row merging on the borderless table. Among them, determining the range of the borderless table includes steps 601 to 605, performing column clustering on the borderless table includes step 606, and performing row merging on the borderless table includes step 607. The following combines Figure 6 and Figure 7 , taking the terminal obtaining the image to be recognized and the server executing the above data extraction method on the image obtained by the terminal as an example for illustration. The method includes the following steps:

[0064] 600. The terminal obtains the image to be recognized and sends a data extraction request to the server. The data extraction request carries the image to be recognized.

[0065] Among them, the image to be recognized is an image containing a borderless table, which can be an original image or an image obtained through preprocessing. The preprocessing can be cropping after edge detection of the original image, or image enhancement and image filtering of the original image, etc. The embodiments of the present application do not limit the method for obtaining the image to be recognized.

[0066] 601. The server obtains multiple field boxes of the image to be recognized based on the image to be recognized input by the client, and each field box corresponds to the position of a string in the image to be recognized.

[0067] In the embodiments of the present application, the server receives the image to be recognized input by the client, and calls a pre-trained recognition model to process the image to be recognized, and obtains the coordinates of each field box in the image to be recognized and the data in the field box. Each data is a string, and each field box corresponds to a string in the image to be recognized and the coordinates of the string. Among them, the recognition model includes a first sub-model and a second sub-model. The first sub-model is a positioning model for obtaining the coordinates of each field box in the image to be recognized, and is trained with a large number of images and the coordinates of multiple field boxes in the images as the training set. The second sub-model is a character recognition model for recognizing the data in each field box.

[0068] In some embodiments, the server uses the optical character recognition (OCR) method to perform OCR recognition on the image to be recognized, and obtains the data in each field box of the image to be recognized.

[0069] 602. The server obtains the regular template corresponding to the image to be recognized based on the multiple field boxes of the image to be recognized.

[0070] In the embodiments of the present application, the server stores multiple regular templates, and different regular templates correspond to different types of images.

[0071] In step 602, for any regular template, the server matches the information of multiple first anchor fields in the regular template with each field box in the image to be recognized based on the information of the multiple first anchor fields in the regular template. If there is a field box in the image to be recognized that successfully matches the information of each first anchor field in the regular template, the regular template is obtained as the regular template corresponding to the image to be recognized. If there is no field box in the image to be recognized that successfully matches the information of any first anchor field in the regular template, the information of the multiple first anchor fields in the next regular template is used to match each field box in the image to be recognized until the regular template corresponding to the image to be recognized is obtained. Among them, the above first anchor fields are perspective transformation anchor fields in the image to be recognized, and the matching process is the same as the matching process in step 401 above, which will not be elaborated in this embodiment of the present application.

[0072] Take Figure 5 the image to be recognized in as an example. If there is a regular template whose perspective transformation anchor fields are "purchaser", "password area", "seller", and "remark". After the server matches the information of these perspective transformation anchor fields with all the field boxes in the image to be recognized one by one, it is found that the information of these four perspective transformation anchor fields all have successfully matched anchor field boxes in the multiple field boxes of the image to be recognized, then this regular template is used as the regular template corresponding to the image to be recognized.

[0073] In this embodiment of the present application, the perspective transformation anchor field is used as the first anchor field to obtain the regular template corresponding to the image to be recognized. For any type of image, the number of its perspective transformation anchor fields is usually small. Using the perspective transformation anchor as the first anchor field to obtain the regular template has a relatively high success rate in obtaining the regular template corresponding to the image to be recognized. Take Figure 5Taking the bill in as an example, there are only four perspective transformation anchor fields in the regular template corresponding to the bill. If the information of these four perspective transformation anchor fields all has successfully matched field boxes in the image to be recognized, the regular template can be used as the regular template corresponding to the image to be recognized. In addition, the perspective transformation anchor fields are usually fixed fields unique to this type of image, and in this type of image, the header anchor fields or lower bound anchor fields of different images may be slightly different. Therefore, for different types of images, it is very likely that their perspective transformation anchor fields are different. Based on the perspective transformation anchor fields, the accuracy of obtaining the regular template corresponding to the image to be recognized is relatively high. For different images in a certain type of image, since the header anchor fields and lower bound anchor fields in the regular template are the union of the header anchor fields and lower bound anchor fields of different images in this type of image, after the server obtains the regular template corresponding to this type of image, for different images in this type of image, the server can all find the field boxes in the image that successfully match the header anchor fields and lower bound anchor fields based on this regular template. That is to say, the regular template of this type of image is applicable to different images in this type of image.

[0074] In some embodiments, the first anchor field may also be the header anchor field or all anchor fields in the regular template, etc., and the embodiments of the present application do not limit this.

[0075] 603. The server determines multiple anchor field boxes from multiple field boxes of the image to be recognized based on the anchor fields indicated by the regular template, and the multiple anchor field boxes match the anchor fields.

[0076] In the embodiments of the present application, for any anchor field in the regular template, the server obtains a field box that successfully matches the information of the anchor field from multiple field boxes of the image to be recognized, and uses this field box as the anchor field box. This matching process is the same as the matching process in step 401 above, and the embodiments of the present application will not elaborate here.

[0077] 604. The server adjusts the position coordinates of multiple field boxes of the image to be recognized based on the position coordinates of the anchor fields in the regular template and multiple anchor field boxes.

[0078] In the embodiments of the present application, the server performs perspective transformation on the image to be recognized based on the position coordinates of the perspective transformation anchor fields in the regular template and multiple anchor field boxes that successfully match the perspective transformation anchor fields in the image to be recognized, and obtains the position coordinates of multiple field boxes in the adjusted image to be recognized. Among them, perspective transformation refers to obtaining the mapping matrix between the standard image and the image to be recognized based on the position coordinates of the perspective transformation anchor fields and the position coordinates of the anchor field boxes that successfully match the perspective transformation anchor fields in the image to be recognized, and adjusting the position coordinates of each field box in the image to be recognized based on this mapping matrix.

[0079] Using the regular template based on spatial information provided in the embodiments of the present application to perform perspective transformation on the image to be recognized can adjust the position coordinates of the field boxes in the image to be recognized, reduce the influence of rigid factors such as the terminal photographing angle and image rotation on data extraction, and enhance the robustness of borderless table recognition.

[0080] It should be noted that the above step 604 is an optional step. For the image to be recognized with rotation distortion, adjusting the position coordinates of the field boxes in the image to be recognized can more accurately determine the rows and columns to which the field boxes belong and improve the accuracy of data extraction. However, this adjustment process is not a necessary step. For an image with a small degree of distortion, the server may not adjust the position coordinates of the field boxes in the image.

[0081] 605. The server determines a plurality of first field boxes located within the borderless table of the image to be recognized from the plurality of field boxes based on a plurality of anchor field boxes.

[0082] In the embodiments of the present application, the anchor field boxes that match successfully with the header anchor field and the lower bound anchor field in the regular template are used to determine the range of the borderless table. Since the header anchor field is a fixed field in the header of the borderless table and is located at the uppermost part of the borderless table, and the lower bound anchor field is the fixed field closest to the lower border in the borderless table, therefore, based on the field boxes that match successfully with the header anchor field and the lower bound anchor field in the regular template, the upper and lower boundaries of the borderless table can be obtained, thereby determining the range of the borderless table. As Figure 8 shown, this step 605 includes steps 6051 to 6052.

[0083] 6051. The server determines the upper and lower boundaries of the borderless table based on the upper and lower boundaries of the plurality of anchor field boxes.

[0084] In the embodiments of the present application, the server determines the upper and lower boundaries of the borderless table based on the upper and lower boundaries of the anchor field boxes that match successfully with the header anchor field and the lower bound anchor field in the regular template. That is, the upper boundary or the lower boundary of the header anchor field box is used as the upper boundary of the borderless table, and the upper boundary or the lower boundary of the lower bound anchor field box is used as the lower boundary of the borderless table.

[0085] The embodiments of the present application are described by taking the example that there is only one anchor field box that matches the header anchor field in the regular template, or there are multiple anchor field boxes that match the header anchor field and the upper or lower boundaries of these anchor field boxes are the same. In some embodiments, the upper or lower boundaries of the above-mentioned multiple anchor field boxes are different. If the upper boundary of the borderless table is determined based on the upper boundaries of the above-mentioned multiple anchor field boxes, the largest upper boundary among the upper boundaries of the multiple anchor field boxes can be used as the upper boundary of the borderless table, or the smallest upper boundary among the upper boundaries of the above-mentioned multiple anchor field boxes can be used as the upper boundary of the borderless table, or the average value of the upper boundaries of the above-mentioned multiple anchor field boxes can be used as the upper boundary of the borderless table. The embodiments of the present application do not limit this. If the upper boundary of the borderless table is determined based on the lower boundaries of the above-mentioned multiple anchor field boxes, this process is the same as the process of determining the upper boundary of the borderless table based on the upper boundaries of the above-mentioned multiple anchor field boxes, and the embodiments of the present application do not elaborate on this.

[0086] The embodiments of the present application are described by taking the example that the upper or lower boundaries of multiple anchor field boxes that match the lower-bound anchor field are the same. In some embodiments, the upper or lower boundaries of the above-mentioned multiple anchor field boxes are different. The process of determining the lower boundary of the borderless table is the same as the process of determining the upper boundary of the borderless table described above, and the embodiments of the present application do not elaborate on this.

[0087] 6052. The server determines multiple first field boxes located within the borderless table of the to-be-recognized image based on the upper and lower boundaries of the borderless table.

[0088] Among them, the first field box is the field box corresponding to the data belonging to the borderless table in the to-be-recognized image. There may not only be data belonging to the borderless table within the upper and lower boundaries of the borderless table, but also data that does not belong to the borderless table. For example, Figure 5 in the to-be-recognized image of, there is data such as "invoice copy" on the left and right sides of the borderless table that does not belong to the borderless table. In addition, if the upper and lower boundaries of the borderless table are inaccurate, it is also possible to divide the data above or below the borderless table into the range of the borderless table. Therefore, after determining the upper and lower boundaries of the borderless table, based on these upper and lower boundaries, the field boxes within the upper and lower boundaries are screened out, and then these field boxes are verified one by one. The field boxes other than the field boxes corresponding to the data that does not belong to the borderless table are used as the first field boxes. The specific process is as follows: The field boxes other than the target field box within the upper and lower boundaries of the borderless table are determined as the first field boxes. The target field box satisfies the form condition, and the form condition is any of the following: the string in the field box includes a preset string, or the closest distance between the field box and multiple other field boxes in a preset direction is greater than or equal to the distance threshold, or the format of the string in the field box is different from the preset format of the borderless table.

[0089] In the above formal conditions, that the preset string is included in the string of the field box means that the server has pre-recorded the strings (preset strings) that often appear around the borderless table. If the upper and lower boundaries of the borderless table determined by the server are inaccurate and the string above the borderless table is included in the scope of the borderless table, then by matching the field boxes within the upper and lower boundaries with the preset strings one by one, the field box including the preset string can be found. That the closest distance between a field box and multiple other field boxes in the preset direction is greater than or equal to the distance threshold means that if the upper and lower boundaries of the borderless table determined by the server are inaccurate and the data outside the borderless table is also included in the scope of the borderless table, and at this time, it is impossible to make a judgment based on the preset string, then the distances between the field boxes within the upper and lower boundaries in the vertical or horizontal direction and the field boxes other than this field box can be obtained. If the smallest distance is still greater than or equal to the distance threshold, it indicates that this field box does not belong to the borderless table. That the format of the string of the field box is different from the preset format of the borderless table means that the arrangement direction of the fields in this field box is different from the preset format in the borderless table.

[0090] Taking Figure 5 the to-be-recognized image in Figure 5 as an example, if the upper boundary of the borderless table determined by the server is inaccurate and the "bank name and account number" above the borderless table is also included in the scope of the borderless table, then by matching the field boxes within the upper and lower boundaries with the preset strings one by one, the field box corresponding to the mis-included "bank name and account number" can be found. Still taking Figure 5 the to-be-recognized image in Figure 5 as an example, if the lower boundary of the borderless table determined by the server is inaccurate and the value corresponding to "total" below the borderless table is also included in the scope of the borderless table, and at this time, it is impossible to make a judgment based on the preset string, then the distances between the field boxes within the upper and lower boundaries in the vertical direction and the field boxes other than this field box can be obtained. If the smallest distance is still greater than or equal to the distance threshold, it indicates that this field box does not belong to the borderless table. Still taking

[0091] the to-be-recognized image in Figure 7 as an example, in this to-be-recognized image, the field box corresponding to "invoice copy" is within the upper and lower boundaries of the borderless table but does not belong to the borderless table.

[0092] 606. The server determines the columns of multiple first field boxes based on the positional relationship between multiple first field boxes and multiple anchor field boxes.

[0093] In the embodiments of the present application, the server performs column clustering on the borderless table based on the anchor field boxes that match the table header anchor field to determine the first field boxes in the column corresponding to the table header, that is, to determine the columns of multiple first field boxes. During the process of determining the columns of multiple first field boxes, the columns of multiple first field boxes can be determined through one round of determination process, or can be determined through multiple rounds of determination process. During the multiple rounds of determination process, the boundaries of the anchor field boxes that match the table header anchor field can be dynamically refreshed. Taking the i-th round of determination process and the (i + 1)-th round of determination process as an example, the left and right boundaries of the anchor field box in the (i + 1)-th round are adjusted according to the first field boxes in the same column as the anchor field box determined in the i-th round until the left and right boundaries of the anchor field box no longer change. As Figure 9 shown, step 606 includes steps 6061 to 6066.

[0094] 6061. For any first field box, the server obtains the overlapping ratio between the first field box and any anchor field box based on the left and right boundaries of the first field box and multiple anchor field boxes. The overlapping ratio is the ratio of the overlapping area between the first field box and the anchor field box in the union of the left and right boundaries of the first field box and the anchor field box.

[0095] In the embodiments of the present application, the server obtains the left and right boundaries of multiple anchor field boxes that match the table header anchor field and multiple first field boxes, that is, the coordinates of the left and right boundaries of the field boxes on the horizontal coordinate axis. For any first field box, during the process of obtaining the overlapping ratio between the first field box and any anchor field box, the server subtracts the larger left boundary from the smaller right boundary of the first field box and any anchor field box to obtain the size of the overlapping range in the horizontal direction between the first field box and the anchor field box, and divides the size of the overlapping range by the size of the union of the left and right boundaries of the first field box and the anchor field box to obtain the overlapping ratio between the first field box and the anchor field box. For example, if the left boundary of a first field box is 3 and the right boundary is 8, and the left boundary of an anchor field box is 4 and the right boundary is 9, then the overlapping ratio between the first field box and the anchor field box is (8 - 4) / (9 - 3) = 0.67.

[0096] The above step 6061 is described by taking the ratio of the overlapping area of the first field box and the anchor field box as the first field, in the union of the left and right boundaries of the first field box and the anchor field box, as an example. In some embodiments, the above overlapping ratio can also be the ratio of the overlapping area of the first field box and the anchor field box between the left and right boundaries of the first field box, or the ratio of the overlapping area of the first field box and the anchor field box between the left and right boundaries of the anchor field box. The embodiments of the present application do not limit this.

[0097] 6062. The server takes the columns of the anchor field boxes with an overlapping ratio greater than or equal to the ratio threshold as the columns of the first field box.

[0098] In the embodiments of the present application, if the overlapping ratio between the first field box and any anchor field box is greater than or equal to the ratio threshold, the server classifies the first field box into the column where the anchor field box is located. The above steps 6061 to 6062 are a round of determination processes. Through the above steps 6061 and 6062, the columns of some first field boxes in the borderless table can be initially determined. However, since there may be data with serious misalignment and offset in the to-be-recognized image, the first field boxes corresponding to these data may have an overlapping ratio greater than or equal to the ratio threshold with at least two anchor field boxes, or may have an overlapping ratio less than the ratio threshold with the anchor field boxes. The above steps 6061 and 6062 cannot determine the columns to which these data belong. For the above two situations, steps 6063 and 6064 are respectively used to determine the columns of the field boxes corresponding to these data.

[0099] 6063. For any first field box, if the overlapping ratio between the first field box and at least two anchor field boxes is greater than or equal to the ratio threshold, the server determines the column of the anchor field box with the largest overlapping ratio among the at least two anchor field boxes as the column of the first field box.

[0100] In the embodiments of the present application, there may be some first field boxes corresponding to misaligned data with an overlapping ratio greater than or equal to the ratio threshold with at least two anchor fields. Then the server corrects the misaligned data based on the overlapping ratio. Specifically, the server determines the column of the anchor field box with the largest overlapping ratio among the at least two anchor field boxes corresponding to the first field box as the column of the first field box.

[0101] 6064. For any first field box, if the overlapping ratio between the first field box and any anchor field box is less than the ratio threshold, the server obtains the distances between the first field box and multiple anchor field boxes, and determines the column of the anchor field box closest to the first field box as the column of the first field box.

[0102] In the embodiments of the present application, there may be a situation where the overlapping ratio of the first field box corresponding to some misaligned data with any anchor field box is less than the ratio threshold. Then, the server finds the table header for the data corresponding to the first field box based on the distance between the field boxes, that is, determines the column of the first field box. Specifically, the server obtains the distances between the first field box and multiple anchor field boxes, and determines the column of the anchor field box with the smallest distance among them as the column of the first field box. The distance between the first field box and the anchor field box may be the straight-line distance between the center points of the first field box and the anchor field box, or the distance between the center points of the first field box and the anchor field box on the horizontal coordinate axis. The embodiments of the present application do not limit how to calculate the distance between the first field box and the anchor field box.

[0103] The above steps 6061 to 6064 are all described by taking one round of determination process as an example. In some embodiments, if after one round of determination process, the preset condition is not met, the server performs multiple rounds of determination process. The preset condition may be that the number of first field boxes for which the columns have been determined is less than the number threshold. Taking the (i + 1)-th round of determination process as an example, the multiple rounds of determination process include steps 6065 to 6066.

[0104] 6065. For any anchor field box, the server adjusts the left and right boundaries of the anchor field box based on the first field boxes in the column of the anchor field box.

[0105] This step 6065 is a process of dynamically refreshing the boundaries of the anchor field boxes that match successfully with the table header anchor field to solve the data offset problem. In this process, for any of the above anchor field boxes, the server takes the left boundary of the field box with the smallest left boundary in the column of the anchor field box determined in the i-th round as the left boundary of the anchor field box, and takes the right boundary of the field box with the largest right boundary in the column of the anchor field box as the right boundary of the anchor field box. Specifically as follows: For any anchor field box, if there are multiple second field boxes in the column of the anchor field box, take the smallest left boundary among the multiple second field boxes as the left boundary of the anchor field box. The second field box is a first field box with a left boundary smaller than that of the anchor field box; if there are multiple third field boxes in the column of the anchor field box, take the largest right boundary among the multiple third field boxes as the right boundary of the anchor field box. The third field box is a first field box with a right boundary larger than that of the anchor field box. The dynamic boundary adjustment algorithm in this step 6065 can assist in the column clustering of the borderless table and reduce the impact caused by misalignment in ticket printing.

[0106] 6066. The server determines the columns of multiple first field boxes based on the adjusted left and right boundaries of the anchor field boxes.

[0107] In the embodiments of the present application, after the determination process of the i-th round is completed, there may still be some first field boxes in the borderless table that have not determined columns. Then, the server repeats the above steps 6061 to 6062 based on the left and right boundaries of the anchor field boxes adjusted in the above step 6065 to determine the columns of this part of the first field boxes. After multiple rounds of determination processes are completed, steps 6063 to 6064 are executed.

[0108] In some embodiments, after completing the above steps 601 to 606, the server performs semantic parsing on multiple first field boxes, so as to eliminate invalid first field boxes, avoid data extraction errors caused by errors in the above steps 601 to 606, and improve the accuracy of data extraction. The invalid first field boxes may be field boxes that do not belong to the borderless table, etc., and the embodiments of the present application do not make restrictive definitions on this. Specifically, the server uses a pre-prepared regular expression to match the data in multiple first field boxes, finds the first field boxes that meet the regular expression, and uses the first field boxes as invalid first field boxes. Among them, the pre-prepared regular expression may be a regular expression corresponding to the data that often goes wrong summarized according to the image type.

[0109] 607. The server determines the rows of multiple first field boxes based on the positional relationship between the multiple first field boxes and the multiple anchor field boxes.

[0110] In the embodiments of the present application, the server performs row merging on the borderless table based on multiple anchor field boxes that match successfully with the header anchor fields and multiple first field boxes to determine the rows of the multiple first field boxes. As Figure 10 shown, this step 607 includes steps 6071 to 6073.

[0111] 6071. The server determines a reference line based on multiple anchor field boxes.

[0112] In the embodiments of the present application, the server connects the center points of multiple anchor field boxes that match successfully with the header anchor fields to obtain a reference line. Taking the horizontal direction of the image to be recognized as the x-axis and the vertical direction as the y-axis, in this coordinate system, the reference line is represented by y = kx + b0, where k represents the slope of the reference line and b0 represents the intercept of the reference line on the y-axis.

[0113] The embodiments of the present application take the center points of multiple anchor field boxes being on a straight line as an example for illustration. In some embodiments, if the center points of multiple anchor field boxes are not on a straight line, the server calls a pre-trained fitting model to fit a straight line based on the center points of multiple anchor field boxes as the reference line.

[0114] 6072. The server determines multiple lines based on a reference line and multiple first field frames. These multiple lines are parallel to the reference line and pass through the reference anchor points, where the reference anchor points are the center points of the field frames in a preset column, and the preset column is the column corresponding to a column of data in the borderless table where data wrapping does not occur.

[0115] This step 6072 includes steps 6072A and 6072B.

[0116] 6072A. The server determines multiple reference anchor points based on multiple first field frames.

[0117] In the embodiments of this application, after the server determines the columns of multiple first field frames, it determines multiple field frames belonging to the preset column from the multiple first field frames in the borderless table, and takes the center points of these multiple field frames as the reference anchor points. Taking the Figure 5 to-be-recognized image in as an example, among the multiple columns of the borderless table, the data in the column with the table header "Amount" does not wrap, so the "Amount" column is taken as the preset column, the multiple first field frames belonging to the "Amount" column are taken as the field frames for positioning, and the center points of these multiple first field frames are taken as the reference anchor points.

[0118] Taking the center points of multiple field frames corresponding to a column of data where data wrapping does not occur as the reference anchor points can avoid taking the points in the field frames corresponding to wrapped data as the reference anchor points, thereby avoiding taking the wrapped data as a separate row of data and improving the accuracy of data extraction.

[0119] In some embodiments, the server can select any point within the first field frame as the reference anchor point, and the embodiments of this application do not limit this.

[0120] 6072B. The server determines multiple lines based on the reference line and multiple reference anchor points.

[0121] In the embodiments of this application, the server determines multiple lines based on multiple anchor field frames and multiple first field frames. The multiple lines are parallel to the connection lines of the center points of the multiple anchor field frames and pass through the reference anchor points. That is, the server determines multiple lines based on the reference line and multiple reference anchor points, and these lines are parallel to the reference line and pass through the reference anchor points. Taking the coordinate system in step 6071 above as an example, the reference line is represented by y = kx + b0, the reference anchor point coordinates are (x, y), then the multiple lines can be represented by y = kx + b1, y = kx + b2, y = kx + b3, etc. The slopes of these lines are the same as the slope of the reference line, indicating that these lines are parallel to the reference line, and the intercepts of these lines on the y-axis are obtained based on the coordinates of the reference anchor points.

[0122] The above steps 6071 to 6072 are a possible implementation of obtaining a straight line passing through the first field box in the row direction for the first field box that has been separated.

[0123] 6073. The server determines the rows of multiple first field boxes based on multiple straight lines.

[0124] In the embodiments of the present application, for any first field box, if the first field box is passed through by a straight line, the row of the straight line is determined as the row of the first field box; if the first field box is not passed through by a straight line, the row of the straight line closest to the first field box is determined as the row of the first field box. Wherein, the first field box being passed through by a straight line means that the intercept of the straight line on the y-axis is within the upper and lower boundaries of the first field box. Specifically, if the first field box is not passed through by a straight line, it indicates that the data in the field box may be obtained by line break of the data in a certain first field box. Then the server performs a nearest match on the first field box that is not passed through by a straight line, calculates the distances between the first field box and multiple straight lines, sorts these distances, determines the row of the straight line with the closest distance as the row of the first field box, and takes the data in the first field box as the line break data of the data in the first field box in the same column in this row, and merges the line break data.

[0125] In some embodiments, for the first field box that is not passed through by a straight line, the server calculates the distance between the first field box and the first field boxes in the same column, and takes the data in the first field box as the line break data of the data in the first field box with the closest distance.

[0126] 608. The server extracts the data in multiple first field boxes based on the rows and columns of the multiple first field boxes.

[0127] In the embodiments of the present application, the server extracts the data in the borderless table based on the rows and columns of the multiple first field boxes and outputs the final modeled result. That is, for any first field box, the server fills the data in the first field box into the corresponding position of the corresponding array. The corresponding array is determined by the column of the first field box, and the corresponding position of the first field box in the array is determined by the row of the first field box. Taking Figure 5 the to-be-recognized image in as an example, "Optical Instrument * Lens" in the image belongs to the "Project Name" column and is in the second row of the borderless table. Then when extracting the data, "Optical Instrument * Lens" is filled into the second position of the array corresponding to "Project Name".

[0128] The innovative data extraction method provided by the embodiments of the present application determines the range of the borderless table in the image to be recognized based on the anchor fields in the regular template, then clusters the field boxes by calculating the overlap ratio, and uses line fitting to separate the field boxes into rows to determine the rows and columns of the field boxes, and extracts data according to the rows and columns of the field boxes. In a borderless table, the data belonging to the same column has a large overlap range horizontally, and the possibility that the line break data and the data that does not break are passed through by the same horizontal line is small. Therefore, the data extraction method provided by the embodiments of the present application can determine which field boxes belong to the same column and which field boxes belong to the same row according to the positional relationship between the field boxes. Compared with extracting data according to fixed data position information, the data extraction method provided by the embodiments of the present application can avoid data extraction errors caused by data position deviations such as data line breaks and data misalignments, and improve the accuracy of data reading.

[0129] The method of the embodiments of the present application is introduced above. The device of the embodiments of the present application is introduced below. It should be understood that the device introduced below has any functions of the computing device in the above method. As described above in combination with Figures 1 to 10 The data extraction method provided by the embodiments of the present application is described in detail. Based on the same inventive concept, the data extraction device provided by the embodiments of the present application will be described below in combination with Figure 11 It should be understood that the technical features described in the method embodiments also apply to the following device embodiments.

[0130] See Figure 11 , the embodiments of the present application provide a data extraction device, which includes:

[0131] The anchor field box determination module 1101 is used to determine a plurality of anchor field boxes from a plurality of field boxes in the image to be recognized based on the anchor fields indicated by the regular template. Each field box corresponds to the position of a string in the image to be recognized, and the plurality of anchor field boxes match the anchor fields;

[0132] The first field box determination module 1102 is used to determine a plurality of first field boxes located within the borderless table in the image to be recognized from the plurality of field boxes based on the plurality of anchor field boxes;

[0133] The row and column determination module 1103 is used to determine the columns of the plurality of first field boxes and determine the rows of the plurality of first field boxes based on the positional relationship between the plurality of first field boxes and the plurality of anchor field boxes;

[0134] The extraction module 1104 is used to extract the data in the plurality of first field boxes based on the rows and columns of the plurality of first field boxes.

[0135] In some embodiments, the above first field box determination module 1102 includes:

[0136] A table boundary determination unit, configured to determine the upper and lower boundaries of a borderless table based on the upper and lower boundaries of a plurality of anchor field frames;

[0137] A first field frame determination unit, configured to determine a plurality of first field frames located within the borderless table of the image to be recognized based on the upper and lower boundaries of the borderless table.

[0138] In some embodiments, the above first field frame determination unit is configured to: determine the field frames other than the target field frame within the upper and lower boundaries of the borderless table as the first field frames, where the target field frame meets a form condition; the form condition is any one of the following: the string in the field frame includes a preset string, or, the closest distance between the field frame and a plurality of other field frames in a preset direction is greater than or equal to a distance threshold, or, the format of the string in the field frame is different from the preset format of the borderless table.

[0139] In some embodiments, the above row and column determination module 1103 is configured to: for any first field frame, based on the left and right boundaries of the first field frame and a plurality of anchor field frames, obtain the overlap ratio between the first field frame and any anchor field frame, where the overlap ratio is the proportion of the overlapping area between the first field frame and the anchor field frame in the union of the left and right boundaries of the first field frame and the anchor field frame; determine the column of the anchor field frame with an overlap ratio greater than or equal to a ratio threshold as the column of the first field frame.

[0140] In some embodiments, the above row and column determination module 1103 is configured to: for any anchor field frame, if there are multiple second field frames in the column of the anchor field frame, take the smallest left boundary of the multiple second field frames as the left boundary of the anchor field frame, where the second field frame is a first field frame smaller than the left boundary of the anchor field frame; if there are multiple third field frames in the column of the anchor field frame, take the largest right boundary of the multiple third field frames as the right boundary of the anchor field frame, where the third field frame is a first field frame larger than the right boundary of the anchor field frame; determine the columns of the plurality of first field frames based on the left and right boundaries of the anchor field frame.

[0141] In some embodiments, the above row and column determination module 1103 is configured to: for any first field frame, if the overlap ratio between the first field frame and at least two anchor field frames is greater than or equal to a ratio threshold, then determine the column of the anchor field frame with the largest overlap ratio among the at least two anchor field frames as the column of the first field frame.

[0142] In some embodiments, the above row and column determination module 1103 is configured to: for any first field frame, if the overlap ratio between the first field frame and any anchor field frame is less than a ratio threshold, then obtain the distance between the first field frame and a plurality of anchor field frames; determine the column of the anchor field frame closest to the first field frame as the column of the first field frame.

[0143] In some embodiments, the above row and column determination module 1103 includes:

[0144] A straight line determination unit, configured to determine multiple straight lines based on multiple anchor field frames and multiple first field frames. The multiple straight lines are parallel to the connection lines of the center points of the multiple anchor field frames and pass through a reference anchor point. The reference anchor point is the center point of the field frame in a preset column, and the preset column is the column corresponding to a column of data in the borderless table where data wrapping does not occur;

[0145] A row determination unit, configured to determine the rows of the multiple first field frames based on the multiple straight lines.

[0146] In some embodiments, the above row determination unit is configured to: for any first field frame, if the first field frame is penetrated by a straight line, determine the row of the straight line as the row of the first field frame; if the first field frame is not penetrated by a straight line, determine the row of the straight line closest to the first field frame as the row of the first field frame.

[0147] In some embodiments, the above data extraction device further includes:

[0148] An image adjustment module, configured to adjust the positions of multiple field frames of the image to be recognized based on the positions of the anchor fields in the regular template and the multiple anchor field frames.

[0149] In some embodiments, the above regular template includes information of specified anchor fields, and the specified anchor fields include the header fields of the borderless table in the image to be recognized.

[0150] It should be understood that the above data extraction device corresponds to the computing device in the above method embodiment. Each module in the device and the above other operations and / or functions respectively implement various steps and methods implemented by the computing device in the method embodiment. For specific details, reference may be made to the above method embodiment. For the sake of brevity, it will not be elaborated here.

[0151] In an exemplary embodiment, a computer-readable storage medium is further provided, such as a memory including program code. The above program code can be executed by a processor in a computing device to complete the data extraction method in the above embodiment. For example, the computer-readable storage medium is a non-temporary computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0152] The embodiments of the present application also provide a computer program product or a computer program. The computer program product or the computer program includes program codes. The computer instructions are stored in a computer-readable storage medium. A processor in a computing device reads the program codes from the computer-readable storage medium, and the processor executes the program codes to enable the computing device to execute the above data extraction method.

[0153] In addition, an embodiment of the present application also provides a device, which may specifically be a chip, a component or a module. The device may include a processor and a memory connected to each other. The memory is used to store computer execution instructions. When the device runs, the processor may execute the computer execution instructions stored in the memory to enable the chip to execute the data extraction method in each of the above method embodiments.

[0154] Among them, the device, equipment, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be elaborated here.

[0155] Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, it belongs to the same concept as the data extraction method embodiment provided in the above embodiment, and the specific implementation process can be seen in the method embodiment, which will not be elaborated here.

[0156] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of the device or unit may be in an electrical, mechanical or other form.

[0157] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0159] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0160] In the description of the present application, unless otherwise specified, " / " means "or". For example, A / B can represent A or B. The "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, "at least one" means one or more, and "a plurality" means two or more. The words such as "first" and "second" do not limit the quantity and execution order, and the words such as "first" and "second" do not necessarily limit to be different.

[0161] In the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0162] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions. For example, the sensitive words involved in the present application are obtained under full authorization.

[0163] Any combination of the above optional technical solutions can form an optional embodiment of the present disclosure, which will not be elaborated one by one here.

[0164] The above are only the optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A data extraction method, characterized in that, The method includes: Based on the anchor fields indicated by the regular template, determining multiple anchor field boxes from multiple field boxes of the image to be recognized, each field box corresponding to the position of a string in the image to be recognized, and the multiple anchor field boxes matching the anchor fields; Based on the multiple anchor field boxes, determining multiple first field boxes located within the borderless table of the image to be recognized from the multiple field boxes; Based on the positional relationship between the multiple first field boxes and the multiple anchor field boxes, determining the columns of the multiple first field boxes and determining the rows of the multiple first field boxes; Based on the rows and columns of the multiple first field boxes, extracting the data in the multiple first field boxes.

2. The method according to claim 1, wherein The determining, based on the multiple anchor field boxes, of multiple first field boxes located within the borderless table of the image to be recognized from the multiple field boxes includes: Based on the upper and lower boundaries of the multiple anchor field boxes, determining the upper and lower boundaries of the borderless table; Based on the upper and lower boundaries of the borderless table, determining multiple first field boxes located within the borderless table of the image to be recognized.

3. The method according to claim 2, wherein The determining, based on the upper and lower boundaries of the borderless table, of multiple first field boxes located within the borderless table of the image to be recognized includes: Determining the field boxes other than the target field box within the upper and lower boundaries of the borderless table as the first field boxes, where the target field box meets the form condition; The form condition is any one of the following: the string in the field box includes a preset string, or, the shortest distance between the field box and multiple other field boxes in a preset direction is greater than or equal to a distance threshold, or, the format of the string in the field box is different from the preset format of the borderless table.

4. The method according to claim 1, characterized in that The determining of the columns of the multiple first field boxes includes: For any first field box, based on the left and right boundaries of the first field box and the multiple anchor field boxes, obtaining the overlapping ratio between the first field box and any one of the anchor field boxes, where the overlapping ratio is the proportion of the overlapping area between the first field box and the anchor field box in the union of the left and right boundaries of the first field box and the anchor field box; Taking the columns of the anchor field boxes with an overlapping ratio greater than or equal to the ratio threshold as the columns of the first field box.

5. The method according to claim 4, wherein The determining of the columns of the multiple first field boxes includes: For any anchor field box, if there are multiple second field boxes in the column of the anchor field box, taking the smallest left boundary of the multiple second field boxes as the left boundary of the anchor field box, where the second field box is a first field box with a left boundary smaller than that of the anchor field box; If there are multiple third field boxes in the column of the anchor field box, taking the largest right boundary of the multiple third field boxes as the right boundary of the anchor field box, where the third field box is a first field box with a right boundary larger than that of the anchor field box; Based on the left and right boundaries of the anchor field box, determining the columns of the multiple first field boxes.

6. The method according to claim 4, wherein The determining of the columns of the multiple first field boxes includes: For any first field box, if the overlapping ratio of the first field box with at least two anchor field boxes is greater than or equal to the ratio threshold, then determine the column of the anchor field box with the largest overlapping ratio among the at least two anchor field boxes as the column of the first field box.

7. The method according to claim 4, wherein The determining the columns of the multiple first field boxes includes: For any first field box, if the overlapping ratio of the first field box with any of the anchor field boxes is less than the ratio threshold, then obtain the distances between the first field box and the multiple anchor field boxes; Determine the column of the anchor field box closest to the first field box as the column of the first field box.

8. The method according to claim 1, wherein The determining the rows of the multiple first field boxes includes: Based on the multiple anchor field boxes and the multiple first field boxes, determine multiple straight lines, the multiple straight lines are parallel to the connecting lines of the center points of the multiple anchor field boxes and pass through the reference anchor point, the reference anchor point is the center point of the field box in the preset column, and the preset column is the column corresponding to a column of data in the borderless table where data line breaks do not occur; Based on the multiple straight lines, determine the rows of the multiple first field boxes.

9. The method according to claim 8, wherein The based on the multiple straight lines, determining the rows of the multiple first field boxes includes: For any first field box, if the first field box is passed through by a straight line, then determine the row of the straight line as the row of the first field box; If the first field box is not passed through by a straight line, then determine the row of the straight line closest to the first field box as the row of the first field box.

10. The method according to claim 1, wherein Before the determining, based on the multiple anchor field boxes, multiple first field boxes located within the borderless table of the to-be-recognized image from the multiple field boxes, the method further includes: Based on the positions of the anchor fields in the regular template and the multiple anchor field boxes, adjust the positions of the multiple field boxes of the to-be-recognized image.

11. The method according to claim 1, wherein The regular template includes information of specified anchor fields, and the specified anchor fields include the header fields of the borderless table in the to-be-recognized image.

12. A data extraction device, characterized in that, The device includes: An anchor field box determination module, configured to determine multiple anchor field boxes from multiple field boxes of a to-be-recognized image based on the anchor fields indicated by a regular template, each field box corresponding to the position of a string in the to-be-recognized image, and the multiple anchor field boxes match the anchor fields; A first field box determination module, configured to determine multiple first field boxes located within the borderless table of the to-be-recognized image from the multiple field boxes based on the multiple anchor field boxes; A row and column determination module, configured to determine the columns of the multiple first field boxes and determine the rows of the multiple first field boxes based on the positional relationship between the multiple first field boxes and the multiple anchor field boxes; An extraction module, configured to extract data in the multiple first field boxes based on the rows and columns of the multiple first field boxes.

13. A computing device, characterized in that, The computing device includes a processor, and the processor is configured to execute program code to cause the computing device to execute the method according to any one of claims 1 to 11.

14. A cluster of computing devices, characterized in that, Comprising a plurality of computing devices, including a processor for executing program code such that the cluster of computing devices performs the method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, At least one program code is stored in the storage medium and is read by a processor to cause a computing device to perform the method according to any one of claims 1 to 11.

Images