Image recognition method and system, data processing method

Abstract

This application discloses an image recognition method and system, and a data processing method. The image recognition method includes: acquiring a table image; processing the table image to obtain corresponding table data, wherein the table data includes: text information within cells of the table image and structural relationships between cells; processing the table data using a table recognition model to obtain keywords and keyword structure types in the table image, wherein the table recognition model is used to acquire the table data, input the table data into an encoding network to obtain vectors corresponding to the table data, input the vectors into a feature extraction network to obtain features of the table data, and input the features of the table data into a classification network to obtain keywords and keyword structure types. This application solves the technical problem of low accuracy in recognizing keywords in table images in related technologies.

Description

Technical Field

[0001] The present application relates to the field of image recognition, and more specifically, to an image recognition method and system, and a data processing method. Background Art

[0002] Tables are a data visualization method that can intuitively show how data is organized. Currently, OCR (Optical Character Recognition) is commonly used to recognize table images. However, the text blocks output by this method lack the original data structure information, resulting in low recognition accuracy.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image recognition method and system, and a data processing method to at least solve the technical problem of low recognition accuracy of keywords in table images in the related art.

[0005] According to one aspect of an embodiment of the present application, there is provided an image recognition method, comprising: acquiring a table image; processing the table image to obtain table data corresponding to the table image, wherein the table data comprises: text information within cells in the table image and structural relationships between cells; processing the table data using a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to acquire the table data, input the table data into an encoding network to obtain a vector corresponding to the table data, input the vector into a feature extraction network to obtain features of the table data, input the features of the table data into a classification network to obtain keywords and structural types of keywords.

[0006] According to another aspect of an embodiment of the present application, an image recognition device is also provided, including: an acquisition module for acquiring a table image; a first processing module for processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; a second processing module for processing the table data using a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to acquire the table data, input the table data into an encoding network to obtain a vector corresponding to the table data, input the vector into a feature extraction network to obtain features of the table data, input the features of the table data into a classification network to obtain keywords and structural types of keywords.

[0007] According to one aspect of an embodiment of the present application, a data processing method is provided, including: obtaining an image to be processed, wherein the image to be processed includes text data and format data; generating a target graph corresponding to the image to be processed, wherein the target graph includes a plurality of nodes, the nodes are connected by edges, the nodes are used to represent the text data, and the edges are used to represent the format data; and displaying the target graph.

[0008] According to another aspect of an embodiment of the present application, a storage medium is further provided, the storage medium including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned image recognition method and data processing method.

[0009] According to another aspect of an embodiment of the present application, a computing device is also provided, including: a memory and a processor, the memory is used to store programs, and the processor is used to run the programs, wherein the above-mentioned image recognition method and data processing method are executed when the programs are run.

[0010] According to another aspect of an embodiment of the present application, there is also provided an image recognition system, including: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: obtaining a table image; processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; processing the table data using a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to obtain the table data, input the table data into an encoding network to obtain a vector corresponding to the table data, input the vector into a feature extraction network to obtain features of the table data, and input the features of the table data into a classification network to obtain keywords and structural types of keywords.

[0011] In the embodiment of the present application, after obtaining the table image, the table image is first processed to obtain the corresponding table data, and the table data is further processed using the table recognition model to obtain the keywords and the structural type of the keywords, thereby achieving the purpose of table image recognition. It is easy to notice that the table data can be processed using the table recognition model to obtain the structural type of the keywords in the table, so that the recognition of the text block in the table image contains the original data structure information, achieving the technical effect of improving the accuracy of keyword recognition and improving the robustness of table image recognition, thereby solving the technical problem of low recognition accuracy of keywords in table images in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0013] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image recognition method according to an embodiment of the present application;

[0014] Figure 2 is a flowchart of an image recognition method according to an embodiment of the present application;

[0015] Figure 3 is a schematic diagram of an optional table image according to an embodiment of the present application;

[0016] Figure 4 is a schematic diagram of an optional table recognition model according to an embodiment of the present application;

[0017] Figure 5 is a schematic diagram of an image recognition device according to an embodiment of the present application;

[0018] Figure 6 is a flow chart of a data processing method according to an embodiment of the present application;

[0019] Figure 7 is a schematic diagram of an optional interactive interface according to an embodiment of the present application; and

[0020] Figure 8 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0024] OCR: It may refer to the optical character recognition through image processing and pattern recognition technology.

[0025] Key relationship analysis: Analyze whether the key is a simple key, list key, or composite key.

[0026] Table comprehension: Analyzing how data is organized in a table.

[0027] Graph: It is used to describe data that does not have a regular spatial structure. It can be a collection of nodes connected by a series of edges. Nodes can be represented by circles and edges can be represented by lines between circle keys.

[0028] Graph convolutional layer: may refer to a convolutional layer applicable to graph data.

[0029] Graph Attention Layer: The fixed normalization operation in graph convolution can be replaced by an attention mechanism.

[0030] Multi-Layer Perceptron: Multi-Layer Perceptron, MLP, can be a feed-forward artificial neural network model that maps multiple input data sets to a single output data set.

[0031] Currently, OCR is commonly used to recognize table images, but the text blocks output by this method lack the original data structure information. In view of the above problem, a variety of solutions have been proposed in the prior art to analyze the structure of the key. However, the above solutions can only process tables with a single logical structure and require the logical structure type to be known in advance.

[0032] In order to solve the above problems, a graph is constructed for the table data recognized by OCR, and the table recognition model based on graph convolution is processed, so that the structural type in the table can be learned to achieve robust processing of complex tables.

[0033] Example 1

[0034] According to an embodiment of the present application, an image recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image recognition method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0036] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image recognition method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned image recognition method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0040] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0041] Under the above operating environment, this application provides Figure 2 The image recognition method shown. Figure 2 is a flow chart of an image recognition method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0042] Step S202, obtaining a table image;

[0043] The table image in the above steps can be a blank table image without filling in data, or a table image filled in data. In the embodiment of the present application, a blank table image is taken as an example for description.

[0044] Step S204, processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells;

[0045] The text information in the above steps may be text in a cell, wherein for a blank cell, the text information may be blank information. The structural relationship may represent the positional relationship between two cells in a table, including but not limited to: adjacent relationship, same row relationship, same column relationship.

[0046] In an optional embodiment, the cells in the table can be determined by identifying the horizontal and vertical lines in the image, and the text in each cell can be further identified by OCR. In addition, the structural relationship between the cells can be determined by the position information of the cells in the table. The position information of the cells can be determined in the following manner: first, the size of the table is determined based on the horizontal and vertical lines in the image, that is, the number of rows and columns of the table is determined, and then the position information of the cells is determined by the number of rows and columns occupied by the cells in the table.

[0047] Optionally, the position information can be represented by a 1*4 matrix, where the first element indicates the starting row number of the cell, the second element indicates the ending row number of the cell, the third element indicates the starting column number of the cell, and the fourth element indicates the ending column number of the cell. Figure 3 By identifying the horizontal and vertical lines in the table image shown, it can be identified that the table has 3 rows and 3 columns, containing a total of 6 cells. The position information of the first cell is [0 1 0 3], the position information of the second cell is [1 2 0 1], the position information of the third cell is [1 2 1 2], the position information of the fourth cell is [1 2 2 3], the position information of the fifth cell is [2 3 0 1], and the position information of the sixth cell is [2 3 1 3].

[0048] Step S206, using the table recognition model to process the table data to obtain the keywords and the structural types of the keywords in the table image, wherein the table recognition model is used to obtain the table data, input the table data into the encoding network, obtain the vector corresponding to the table data, input the vector into the feature extraction network to obtain the features of the table data, input the features of the table data into the classification network, and obtain the keywords and the structural types of the keywords.

[0049] The structure types in the above steps may include but are not limited to: simple type, list type and composite type, where a simple type means that the key is a key in a key-value pair, a list type means that the key is a key in a list key-value, and a composite type means that the key is a key in a composite key-value. Figure 3 In the table shown, the structure type of the Key "Student Name" is a simple type, the structure type of the Key "Relationship" is a list type, and the structure type of the Key "Home Visit Situation" is a composite type.

[0050] In an optional embodiment, in order to be able to identify the structural type of keywords in a table image, the table structure can be abstracted into nodes and edges in a graph model. On this basis, the nodes and edges can be classified through a table recognition model, so that the structural type of the keyword can be accurately obtained.

[0051] Through the solution provided by the above embodiment of the present application, after obtaining the table image, the table image is first processed to obtain the corresponding table data, and the table data is further processed using the table recognition model to obtain the keywords and the structural type of the keywords, thereby achieving the purpose of table image recognition. It is easy to notice that the table data can be processed using the table recognition model to obtain the structural type of the keywords in the table, so that the recognition of the text block in the table image contains the original data structure information, achieving the technical effect of improving the accuracy of keyword recognition and improving the robustness of table image recognition, thereby solving the technical problem of low recognition accuracy of keywords in table images in the related art.

[0052] In the above embodiment of the present application, the table data is processed by a table recognition model to obtain the keywords and the structural types of the keywords in the table image, including: generating a target graph based on the table data, wherein the target graph contains multiple nodes, any two nodes are connected by edges, the nodes are used to represent the text information in the cells, and the edges are used to represent the structural relationship between the cells; the nodes and edges in the target graph are input into the encoding layer of the table recognition model to obtain the first vector of the nodes and the second vector of the edges; the first vector and the second vector are input into the graph volume base layer and the graph attention layer of the table recognition model to obtain the node features and the edge features; the node features and the edge features are input into the output layer of the table recognition model to obtain the keywords and the structural types of the keywords.

[0053] The target graph in the above steps may be a data structure consisting of multiple nodes and edges between the nodes, which is different from an image. In order to obtain the direction between two cells, the target graph may be a directed graph.

[0054] Optional, the first vector is t i , the second vector is r ij , graph convolution can be expressed as the following formula:

[0055] h ij =g(t i , r ij , t j )=MLP([t i ||r ij ||t j ]),

[0056] Among them, MLP represents multi-layer perceptron, and || represents vector connection operation.

[0057] Graph attention can be expressed as the following formula:

[0058] t′ i =σ(∑ j∈{1，…，n} α ij h ij ),

[0059]

[0060] Among them, σ represents the activation function. The node feature after encoding at each layer is t′ in the above formula i , edge feature r′ ij It is expressed as follows:

[0061] r′ ij =MLP(h ij ).

[0062] For example, Figure 4 The table recognition model shown in the figure is used as an example to illustrate. Figure 4 For the table image shown in , the cells can be regarded as nodes of the graph. The node features are the vector representations of the text in the cells after the encoder. In the graph, there will be edges between any two nodes. The edge features are the 01 vector representations composed of the direction, adjacent and same-row and same-column relationships between two cells. Then, the node and edge vectors are processed by 5 layers of graph convolution and graph attention to obtain the processed node features and edge features. The output layer is further used to extract the Key and Key structure types in the table image.

[0063] It should be noted that the classification result of the node can also be achieved by searching the Key library, and then the KV classification is used as input for processing using graph convolution and graph attention to obtain the structure type of the Key.

[0064] In the above embodiment of the present application, the node features and edge features are input into the output layer of the table recognition model to obtain keywords and the structural types of keywords, including: using a multilayer perceptron and a normalized exponential function to process the node features and the edge features to obtain the classification results of the nodes and the classification results of the edges; based on the classification results of the nodes and the classification results of the edges, the keywords and the structural types of the keywords are obtained.

[0065] Optionally, the classification result of the node includes one of the following: keyword, data and key-value combination, which respectively indicate whether the cell contains a key, a value, or a KV cell. The classification result of the edge includes one of the following: parallel, belong, complex belong and none. The first three types of relationships respectively indicate a common key structure, a list KV structure and a complex KV structure.

[0066] For example, Figure 4 The table recognition model shown in the figure is used as an example to illustrate that the node features and edge features after 5 layers of graph convolution and graph attention processing are processed through MLP and softmax to output the classification results of nodes and edges. Then, based on the above classification results, the Key and Key structure types in the original table image can be obtained. Figure 4 As shown in the figure, nodes classified as Value are represented by hollow circles, nodes classified as Key are represented by solid circles, and nodes classified as KV are represented by rings. The edges of parallel relationships are represented by solid lines, the edges of subordinate relationships are represented by dotted lines, the edges of composite relationships are represented by dotted lines, and the edges without relationships are represented by double dotted lines.

[0067] In the above embodiment of the present application, the method also includes the following steps: generating training data; and training the initial model using the training data to obtain a table recognition model.

[0068] In an optional embodiment, since the table data available on the Internet is limited, in order to ensure the recognition accuracy of the table recognition model, a large amount of table data can be generated as training data as needed.

[0069] In the above embodiment of the present application, generating training data includes: dividing the table into multiple areas; determining the basic logical structure corresponding to each area, wherein the basic logical structure includes one of the following: key-value pairs, list key values, and compound key values; generating training data based on the basic logical structure corresponding to each area.

[0070] In an optional embodiment, the table data can be expressed as follows: Figure 3 The table 2D layout and the hierarchical method of filling the basic logical structure are shown in the figure. The basic logical structures of the table include KV pairs, list KVs and compound KVs. First, the 2D layout of the table can be generated by randomly dividing it with horizontal and vertical lines, and then a basic logical structure is randomly selected for each area. Finally, the corpus is randomly filled based on the basic logical structure, and the basic logical structure is filled to generate training data.

[0071] In the above embodiment of the present application, generating training data based on the basic logical structure corresponding to each region includes: filling keyword corpus and data corpus in each region based on the basic logical structure corresponding to each region; filling the basic logical structure corresponding to each region in each region.

[0072] In an optional embodiment, after randomly selecting a basic logical structure, an instance of the basic logical structure may be randomly generated by a parameterized method, and randomly filled with KV corpus, and finally filled with the basic logical structure.

[0073] In the above-mentioned embodiment of the present application, processing the table image to obtain the table data corresponding to the table image includes: processing the table image to obtain the image of the cell and the structural relationship between the cells; processing the image of the cell to obtain the text information in the cell.

[0074] In an optional embodiment, the image of the cell and the structural relationship between the cells can be obtained through table restoration technology, which is specifically implemented as follows: by identifying the horizontal and vertical lines in the image, the cells in the table are determined, and the structural relationship between the cells is determined through the position information of the cells in the table. Further, text recognition is performed on the cell image through OCR to obtain the text in each cell.

[0075] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0077] Example 2

[0078] According to an embodiment of the present application, an image recognition device for implementing the above-mentioned image recognition method is also provided. Figure 5 As shown, the device 500 includes: an acquisition module 502 , a first processing module 504 and a second processing module 506 .

[0079] Among them, the acquisition module 502 is used to acquire a table image; the first processing module 504 is used to process the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; the second processing module 506 is used to process the table data using a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to acquire the table data, input the table data into the encoding network to obtain a vector corresponding to the table data, input the vector into the feature extraction network to obtain features of the table data, input the features of the table data into the classification network to obtain keywords and structural types of keywords.

[0080] It should be noted that the acquisition module 502, the first processing module 504 and the second processing module 506 correspond to steps S202 to S206 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0081] In the above embodiment of the present application, the second processing module includes: a generating unit, a first processing unit, a second processing unit and a third processing unit.

[0082] Among them, the generation unit is used to generate a target graph based on the table data, wherein the target graph contains multiple nodes, any two nodes are connected by edges, the nodes are used to represent the text information in the cells, and the edges are used to represent the structural relationship between the cells; the first processing unit is used to input the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the node and the second vector of the edge; the second processing unit is used to input the first vector and the second vector into the graph volume base layer and the graph attention layer of the table recognition model to obtain node features and edge features; the third processing unit is used to input the node features and edge features into the output layer of the table recognition model to obtain keywords and keyword structure types.

[0083] In the above embodiment of the present application, the third processing unit includes: a first sub-processing unit and a second sub-processing unit.

[0084] Among them, the first sub-processing unit is used to process the node features and edge features using a multilayer perceptron and a normalized exponential function to obtain the classification results of the nodes and the classification results of the edges; the second sub-processing unit is used to obtain the keywords and the structural types of the keywords based on the classification results of the nodes and the classification results of the edges.

[0085] In the above embodiments of the present application, the device further includes: a generation module and a training module.

[0086] Among them, the generation module is used to generate training data; the training module is used to train the initial model using the training data to obtain a table recognition model.

[0087] In the above embodiment of the present application, the generation module includes: a division unit, used to divide the table into multiple areas; a determination unit, used to determine the basic logical structure corresponding to each area, wherein the basic logical structure includes one of the following: key-value pairs, list key values, and compound key values; a generation unit, used to generate training data based on the basic logical structure corresponding to each area.

[0088] In the above embodiment of the present application, the generating unit includes: a first sub-filling unit and a second sub-filling unit.

[0089] The first sub-filling unit is used to fill keyword corpus and data corpus in each region based on the basic logical structure corresponding to each region; the second sub-filling unit is used to fill the basic logical structure corresponding to each region in each region.

[0090] In the above embodiment of the present application, the first processing module includes: a fourth processing unit and a fifth processing unit.

[0091] The fourth processing unit is used to process the table image to obtain the image of the cell and the structural relationship between the cells; the fifth processing unit is used to process the image of the cell to obtain the text information in the cell.

[0092] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0093] Example 3

[0094] According to an embodiment of the present application, an image recognition system is also provided, including:

[0095] Processor; and

[0096] The memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: obtaining a table image; processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; processing the table data using a table recognition model to obtain keywords in the table image and structural types of the keywords, wherein the table recognition model is used to obtain the table data, input the table data into an encoding network to obtain a vector corresponding to the table data, input the vector into a feature extraction network to obtain features of the table data, and input the features of the table data into a classification network to obtain keywords and structural types of the keywords.

[0097] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0098] Example 4

[0099] According to an embodiment of the present application, a data processing method is also provided.

[0100] Figure 6 is a flow chart of a data processing method according to an embodiment of the present application. Figure 6 As shown, the method comprises the following steps:

[0101] Step S602, obtaining an image to be processed, wherein the image to be processed includes text data and format data;

[0102] In different application fields, the image to be processed in the above steps is different. For example, in the application field of table data processing, the image to be processed can be a blank table image without data filled in, or a table image filled with data. In the embodiment of the present application, a blank table image is used as an example for explanation. The text data can represent the text in the cells of the image to be processed, and the format data can represent the positional relationship between the cells in the image to be processed, including but not limited to: adjacent relationship, same row relationship, and same column relationship.

[0103] For another example, in the application field of road monitoring, the image to be processed may be an image of a vehicle license plate captured by a surveillance camera, the text data may include Chinese, English, and numbers in the license plate, and the format data may represent the positional relationship between each character in the license plate and other characters, including but not limited to: adjacent relationship, peer relationship, etc.

[0104] In an optional embodiment, if Figure 7 As shown, the user can upload the image to be processed by clicking the "Upload Image" button or directly dragging the image file into the dotted box.

[0105] Step S604, generating a target graph corresponding to the image to be processed, wherein the target graph includes a plurality of nodes, the nodes are connected by edges, the nodes are used to represent text data, and the edges are used to represent format data;

[0106] The target graph in the above steps may be a data structure consisting of multiple nodes and edges between the nodes, which is different from an image. In order to obtain the direction between two cells, the target graph may be a directed graph.

[0107] Step S606, display the target image.

[0108] In an optional embodiment, if Figure 7 As shown, in order to facilitate the user to view the target image corresponding to the image to be processed, an interactive interface can be provided to the user, and the generated target image can be displayed in the display area of ​​the interactive interface.

[0109] In the above embodiment of the present application, the method further includes: receiving user input and adjusting the relationship between nodes in the target graph.

[0110] In an optional embodiment, the user can delete nodes in the target graph as needed and adjust the relationship between nodes. The above adjustment may include: deleting edges between nodes, adding edges between nodes, modifying arrow directions, etc. In order to facilitate users to adjust the target graph, such as Figure 7 As shown, the user can directly operate the target graph in the display area. For example, the user can select the node or edge to be adjusted by clicking, and then achieve the adjustment purpose by selecting the corresponding operation or performing the corresponding gesture operation.

[0111] In the above embodiment of the present application, the method further includes: based on the target graph, identifying the image to be processed to obtain a recognition result, wherein the recognition result includes: keywords in the image to be processed, and the structural type of the keywords.

[0112] In the application field of table data processing, the structure types in the above steps may include but are not limited to: simple type, list type and composite type, where a simple type means that the key is the key in a key-value pair, a list type means that the key is the key in a list key-value, and a composite type means that the key is the key in a composite key-value. Figure 3 In the table shown, the structure type of the Key "Student Name" is a simple type, the structure type of the Key "Relationship" is a list type, and the structure type of the Key "Home Visit Situation" is a composite type.

[0113] In the application field of road monitoring, the keywords in the above steps can be each word, English letter or number in the license plate, and the structure type can be the position type between the keywords.

[0114] In the above embodiment of the present application, based on the target graph, the image to be processed is recognized, and the recognition result obtained includes: inputting the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the node and the second vector of the edge; inputting the first vector and the second vector into the graph convolution layer and the graph attention layer of the table recognition model to obtain the node features and the edge features; inputting the node features and the edge features into the output layer of the table recognition model to obtain the recognition result.

[0115] In the above embodiment of the present application, the node features and edge features are input into the output layer of the table recognition model to obtain keywords and the structural types of keywords, including: using a multilayer perceptron and a normalized exponential function to process the node features and the edge features to obtain the classification results of the nodes and the classification results of the edges; based on the classification results of the nodes and the classification results of the edges, the keywords and the structural types of the keywords are obtained.

[0116] In the above embodiment of the present application, the method also includes the following steps: generating training data; and training the initial model using the training data to obtain a table recognition model.

[0117] In the above embodiment of the present application, generating training data includes: dividing the table into multiple areas; determining the basic logical structure corresponding to each area, wherein the basic logical structure includes one of the following: key-value pairs, list key values, and compound key values; generating training data based on the basic logical structure corresponding to each area.

[0118] In the above embodiment of the present application, generating training data based on the basic logical structure corresponding to each region includes: filling keyword corpus and data corpus in each region based on the basic logical structure corresponding to each region; filling the basic logical structure corresponding to each region in each region.

[0119] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0120] Example 5

[0121] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0122] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0123] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image recognition method: obtaining a table image; processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; using a table recognition model to process the table data to obtain keywords and structural types of keywords in the table image, wherein the table recognition model includes: a coding layer, a graph convolution layer, a graph attention layer and an output layer connected in sequence.

[0124] Optionally, Figure 8 is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 8 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 802 and a memory 804.

[0125] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image recognition method and device in the embodiment of the present application, and program instructions corresponding to the data processing method. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image recognition method and data processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain a table image; process the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; use a table recognition model to process the table data to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to obtain the table data, input the table data into the encoding network to obtain a vector corresponding to the table data, input the vector into the feature extraction network to obtain features of the table data, input the features of the table data into the classification network to obtain keywords and structural types of keywords.

[0127] Optionally, the processor may also execute the program code of the following steps: based on the table data, generate a target graph, wherein the target graph contains multiple nodes, any two nodes are connected by edges, the nodes are used to represent the text information in the cells, and the edges are used to represent the structural relationship between the cells; input the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the nodes and the second vector of the edges; input the first vector and the second vector into the graph volume base layer and the graph attention layer of the table recognition model to obtain node features and edge features; input the node features and edge features into the output layer of the table recognition model to obtain keywords and keyword structure types.

[0128] Optionally, the processor may also execute the following program code: using a multilayer perceptron and a normalized exponential function to process node features and edge features to obtain node classification results and edge classification results; based on the node classification results and edge classification results, obtaining keywords and keyword structure types.

[0129] Optionally, the processor may also execute program codes of the following steps: generating training data; and training an initial model using the training data to obtain a table recognition model.

[0130] Optionally, the processor may also execute program code of the following steps: dividing the table into multiple regions; determining a basic logical structure corresponding to each region, wherein the basic logical structure includes one of the following: a key-value pair, a list key value, and a compound key value; and generating training data based on the basic logical structure corresponding to each region.

[0131] Optionally, the processor may also execute program code of the following steps: based on the basic logical structure corresponding to each region, filling keyword corpus and data corpus in each region; filling the basic logical structure corresponding to each region in each region.

[0132] Optionally, the processor may also execute program codes of the following steps: processing the table image to obtain the image of the cell and the structural relationship between the cells; processing the cell image to obtain the text information in the cell.

[0133] The embodiment of the present application provides an image recognition solution. The table data is processed by a table recognition model to obtain the structural type of the keywords in the table, so that the recognition of the text block in the table image contains the original data structure information, achieving the technical effect of improving the accuracy of keyword recognition and the robustness of table image recognition, thereby solving the technical problem of low recognition accuracy of keywords in table images in the related art.

[0134] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the image to be processed, wherein the image to be processed includes text data and format data; generate a target graph corresponding to the image to be processed, wherein the target graph includes multiple nodes, and the nodes are connected by edges, the nodes are used to represent the text data, and the edges are used to represent the format data; display the target graph.

[0135] Optionally, the processor may also execute program code of the following steps: receiving user input, and adjusting the relationship between nodes in the target graph.

[0136] Optionally, the processor may also execute the program code of the following steps: based on the target graph, recognize the image to be processed to obtain a recognition result, wherein the recognition result includes: keywords in the image to be processed, and the structural type of the keywords.

[0137] Optionally, the processor may also execute the following program code steps: input the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the nodes and the second vector of the edges; input the first vector and the second vector into the graph convolution layer and the graph attention layer of the table recognition model to obtain the node features and the edge features; input the node features and the edge features into the output layer of the table recognition model to obtain the recognition results.

[0138] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 8 It does not limit the structure of the above electronic device. For example, the computer terminal A may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations are shown.

[0139] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0140] Example 6

[0141] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image recognition method and the data processing method provided in the above embodiment.

[0142] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0143] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps: acquiring a table image; processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information within cells in the table image and structural relationships between cells; processing the table data using a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to acquire the table data, input the table data into an encoding network to obtain a vector corresponding to the table data, input the vector into a feature extraction network to obtain features of the table data, input the features of the table data into a classification network to obtain keywords and structural types of keywords.

[0144] Optionally, the storage medium is also configured to store program codes for executing the following steps: generating a target graph based on the table data, wherein the target graph comprises a plurality of nodes, any two nodes are connected by edges, the nodes are used to represent the text information in the cells, and the edges are used to represent the structural relationship between the cells; inputting the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the nodes and the second vector of the edges; inputting the first vector and the second vector into the graph volume base layer and the graph attention layer of the table recognition model to obtain node features and edge features; inputting the node features and edge features into the output layer of the table recognition model to obtain keywords and the structural types of the keywords.

[0145] Optionally, the storage medium is also configured to store program codes for executing the following steps: processing node features and edge features using a multilayer perceptron and a normalized exponential function to obtain node classification results and edge classification results; and obtaining keywords and keyword structure types based on the node classification results and edge classification results.

[0146] Optionally, the processor may also execute program codes of the following steps: generating training data; and training an initial model using the training data to obtain a table recognition model.

[0147] Optionally, the storage medium is also configured to store program codes for executing the following steps: dividing the table into multiple regions; determining the basic logical structure corresponding to each region, wherein the basic logical structure includes one of the following: key-value pairs, list key values, and compound key values; and generating training data based on the basic logical structure corresponding to each region.

[0148] Optionally, the storage medium is also configured to store program codes for executing the following steps: based on the basic logical structure corresponding to each region, filling keyword corpus and data corpus in each region; filling the basic logical structure corresponding to each region in each region.

[0149] Optionally, the storage medium is also configured to store program codes for executing the following steps: processing the table image to obtain the image of the cell and the structural relationship between the cells; processing the image of the cell to obtain the text information in the cell.

[0150] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: obtaining an image to be processed, wherein the image to be processed includes text data and format data; generating a target graph corresponding to the image to be processed, wherein the target graph includes multiple nodes, the nodes are connected by edges, the nodes are used to represent the text data, and the edges are used to represent the format data; and displaying the target graph.

[0151] Optionally, the storage medium is further configured to store program codes for executing the following steps: receiving user input, and adjusting the relationship between nodes in the target graph.

[0152] Optionally, the storage medium is further configured to store program codes for executing the following steps: based on the target graph, identifying the image to be processed to obtain a recognition result, wherein the recognition result includes: keywords in the image to be processed, and the structural type of the keywords.

[0153] Optionally, the storage medium is also configured to store program codes for executing the following steps: inputting the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the nodes and the second vector of the edges; inputting the first vector and the second vector into the graph convolution layer and the graph attention layer of the table recognition model to obtain node features and edge features; inputting the node features and edge features into the output layer of the table recognition model to obtain recognition results.

[0154] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0155] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0160] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: include: Acquire an image to be processed, wherein the image to be processed includes text data and format data; Generate a target graph corresponding to the image to be processed, wherein the target graph includes a plurality of nodes, the nodes are connected by edges, the nodes are used to represent the text data, and the edges are used to represent the format data; displaying the target graph; Based on the target graph, the image to be processed is identified to obtain a recognition result, wherein the recognition result is a recognition result obtained by identifying node features and edge features, the node features are obtained based on vectors corresponding to the nodes of the target graph, and the edge features are obtained based on vectors corresponding to the edges of the target graph, and the recognition result includes: keywords in the image to be processed, and the structural type of the keywords.

2. The method according to claim 1, further comprising: Receive user input and adjust the relationship between nodes in the target graph.

3. The method according to claim 1, wherein: Based on the target image, the image to be processed is identified, and the identification result obtained includes: Inputting the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the node and the second vector of the edge; Inputting the first vector and the second vector into the graph convolution layer and the graph attention layer of the table recognition model to obtain node features and edge features; The node features and the edge features are input into the output layer of the table recognition model to obtain the recognition result.

4. The method according to claim 3, wherein: The method further comprises: Generate training data; The initial model is trained using the training data to obtain the table recognition model.

5. The method according to claim 4, wherein: Generating training data includes: Divide the table into multiple regions; Determine a basic logical structure corresponding to each region, wherein the basic logical structure includes one of the following: a key-value pair, a list key-value, and a composite key-value; The training data is generated based on the basic logical structure corresponding to each area.

6. An image recognition method, comprising: Get the table image; Processing the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; The table data is processed by a table recognition model to obtain keywords and structural types of keywords in the table image, wherein the table recognition model is used to obtain the table data, input the nodes and edges in the target graph corresponding to the table data into the encoding network, obtain the first vector of the node and the second vector of the edge corresponding to the table data, input the first vector and the second vector into the feature extraction network to obtain the node features and edge features of the table data, input the node features and the edge features of the table data into the classification network to obtain the keywords and structural types of the keywords.

7. The method according to claim 6, wherein: The table data is processed using a table recognition model to obtain keywords and keyword structure types in the table image, including: Based on the table data, a target graph is generated, wherein the target graph includes a plurality of nodes, any two nodes are connected by the edge, the node is used to represent the text information in the cell, and the edge is used to represent the structural relationship between the cells; Inputting the nodes and edges in the target graph into the encoding layer of the table recognition model to obtain the first vector of the node and the second vector of the edge; Inputting the first vector and the second vector into the graph convolution layer and the graph attention layer of the table recognition model to obtain node features and edge features; The node features and the edge features are input into the output layer of the table recognition model to obtain the keywords and the structural types of the keywords.

8. The method according to claim 7, wherein: The node features and the edge features are input into the output layer of the table recognition model to obtain the keywords and the structural types of the keywords, including: Processing the node features and the edge features using a multilayer perceptron and a normalized exponential function to obtain a classification result of the node and a classification result of the edge; Based on the classification results of the nodes and the classification results of the edges, the keywords and the structural types of the keywords are obtained.

9. The method according to claim 8, wherein: The classification result of the node includes one of the following: keyword, data and key value combination, and the classification result of the edge includes one of the following: parallel relationship, subordinate relationship, compound subordinate relationship and no relationship.

10. The method according to claim 6, wherein: The method further comprises: Generate training data; The initial model is trained using the training data to obtain the table recognition model.

11. The method according to claim 10, wherein: Generating training data includes: Divide the table into multiple regions; Determine a basic logical structure corresponding to each region, wherein the basic logical structure includes one of the following: a key-value pair, a list key-value, and a composite key-value; The training data is generated based on the basic logical structure corresponding to each area.

12. The method according to claim 11, wherein: Based on the basic logical structure corresponding to each area, generating the training data includes: Based on the basic logical structure corresponding to each region, fill the keyword corpus and data corpus in each region; A basic logical structure corresponding to each area is filled in each area.

13. The method according to claim 6, wherein: Processing the table image to obtain table data corresponding to the table image includes: Processing the table image to obtain the image of the cell and the structural relationship between the cells; The image of the cell is processed to obtain text information in the cell.

14. A storage medium comprising a stored program, wherein: When the program is running, the device where the storage medium is located is controlled to execute the data processing method described in any one of claims 1 to 5, or the image recognition method described in any one of claims 6 to 13.

15. A computing device comprising: A memory and a processor, the memory being used to store a program, the processor being used to run the program, wherein when the program is run, the data processing method described in any one of claims 1 to 5, or the image recognition method described in any one of claims 6 to 13 is executed.

16. An image recognition system, comprising: processor; as well as A memory, connected to the processor, is used to provide instructions for the processor to process the following processing steps: acquiring a table image; Process the table image to obtain table data corresponding to the table image, wherein the table data includes: text information in cells in the table image and structural relationships between cells; use a table recognition model to process the table data to obtain keywords in the table image and structural types of keywords, wherein the table recognition model is used to obtain the table data, input the nodes and edges in the target graph corresponding to the table data into an encoding network, obtain a first vector of the node and a second vector of the edge corresponding to the table data, input the first vector and the second vector into a feature extraction network, obtain node features and edge features of the table data, input the node features and the edge features of the table data into a classification network, and obtain the keywords and structural types of the keywords.

Images