Table identification method, identification model training method and related products

Abstract

The invention discloses a table identification method, an identification model training method and a related product. The table identification method and the identification model training method are used for rapidly and accurately identifying a table from a table picture. The table recognition method comprises the steps of performing feature extraction on a table picture containing a table to obtain a first visual feature, and performing feature extraction on a first cue word used for describing the table picture to obtain a first text feature; performing enhancement processing on the first visual feature based on the first text feature to obtain a second visual feature; table information of the table is recognized from the table picture through a recognition model based on the second visual feature, and the table information comprises position information of cells in the table and texts in the cells; and generating the table based on the table information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a table recognition method, a recognition model training method, and related products. Background Art

[0002] Documents are an important way to store information, and tables are the most common and intuitive form of information organization in documents. Tables contain a lot of structured information, and obtaining this structured information helps build a huge database for data storage and management. In actual scenarios, documents that can be obtained are usually in the form of images, which requires the use of table recognition technology to identify tables from documents in the form of images.

[0003] In computing platforms with large computing loads, such as the Haisuan platform, there are a large number of table images that need to be processed. These table images have significant characteristics such as dense text and dense cells, and require high running time. Therefore, how to quickly and accurately identify tables from table images has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a table recognition method, a recognition model training method and related products for quickly and accurately recognizing a table from a table image.

[0005] In order to achieve the above objectives, the present application embodiment adopts the following technical solutions: In a first aspect, an embodiment of the present application provides a table recognition method, comprising: Extracting features of a table image containing a table to obtain a first visual feature, and extracting features of a first prompt word used to describe the table image to obtain a first text feature; Performing enhancement processing on the first visual feature based on the first text feature to obtain a second visual feature; Recognize, by using a recognition model and based on the second visual feature, table information of the table from the table image, the table information including: position information of cells in the table and text in the cells; Based on the table information, the table is generated.

[0006] In a second aspect, an embodiment of the present application provides a method for training a recognition model, comprising: Extracting features of a sample table image containing a sample table to obtain a first sample visual feature, and extracting features of a second prompt word used to describe the sample table image to obtain a first sample text feature; Performing enhancement processing on the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; Using a recognition model, based on the second sample visual features, identifying table information of the sample table from the table image, the table information including: position information of cells in the sample table and text in the cells; Based on the table information of the sample table and the reference table information, the parameters of the recognition model are adjusted.

[0007] In a third aspect, an embodiment of the present application provides a table recognition device, including: A first extraction module is used to extract features from a table image containing a table to obtain a first visual feature, and to extract features from a first prompt word used to describe the table image to obtain a first text feature; A first enhancement module, configured to enhance the first visual feature based on the first text feature to obtain a second visual feature; A first recognition module, configured to recognize table information of the table from the table image by using a recognition model based on the second visual feature, the table information including: position information of cells in the table and text in the cells; The first generating module is used to generate the table based on the table information.

[0008] In a fourth aspect, an embodiment of the present application provides a training device for a recognition model, comprising: A second extraction module is used to extract features from a sample table image containing a sample table to obtain a first sample visual feature, and to extract features from a second prompt word used to describe the sample table image to obtain a first sample text feature; A second enhancement module, configured to enhance the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; A second recognition module is used to recognize table information of the sample table from the table image through a recognition model based on the second sample visual features, where the table information includes: position information of cells in the sample table and text in the cells; The first adjustment module is used to adjust the parameters of the recognition model based on the table information of the sample table and the reference table information.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, including: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the table recognition method provided in the first aspect; or, the processor is configured to execute the instructions to implement the recognition model training method provided in the second aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the table recognition method provided in the first aspect; or, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the recognition model training method provided in the second aspect.

[0011] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the table recognition method provided in the first aspect or the recognition model training method provided in the second aspect.

[0012] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: First, feature extraction is performed on a table image containing a table to obtain a first visual feature, and a first prompt word is used to describe the table image. By extracting features from the first prompt word, a first text feature is obtained. Then, based on the first text feature, the first visual feature is enhanced so that the first text feature contains implicit display attention to text and cells. This type of display attention can guide the first visual feature, so that the first visual feature is more focused on the text area and cell area in the table image, and a second visual feature is obtained to express the features in a more targeted manner. Second, further, through the recognition model, based on the second visual feature, the position information of the cells in the table and the table information such as the text in the cells can be more accurately identified from the table image. Based on these table information, the corresponding table can be accurately generated.

[0013] It can be seen that the table recognition method proposed in the embodiment of the present application does not need to rely on additional OCR technology, which not only avoids the error introduced by OCR, but also makes comprehensive use of image and text information to make the recognition model more robust. In addition, the serial sequence prediction scheme is abandoned, which not only reduces the difficulty of annotation, but also makes it so that in actual applications, the time consumption does not become uncontrollable due to the increase in the number of cells, thereby improving the efficiency of table recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a table recognition method provided for one embodiment of the present application; Figure 2A schematic diagram of a first visual feature extraction process provided for an embodiment of the present application; Figure 3 A schematic diagram of a process for acquiring a second visual feature provided by an embodiment of the present application; Figure 4 A flowchart of a method for training a recognition model provided for one embodiment of the present application; Figure 5 A schematic diagram of a first sample text feature extraction process provided for an embodiment of the present application; Figure 6 A schematic diagram of a training process of a recognition model provided for one embodiment of the present application; Figure 7 A schematic diagram of the structure of a table recognition device provided in one embodiment of the present application; Figure 8 A schematic diagram of the structure of a training device for a recognition model provided in one embodiment of the present application; Fig. 9 A schematic diagram of the structure of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0016] The terms "first", "second", etc. in this specification and claims are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in this specification and claims means at least one of the connected objects, and the character " / " generally means that the objects associated with each other are in an "or" relationship.

[0017] In related technologies, table recognition is usually converted into a sequence recognition problem, and the table results are output through the transformer architecture (such as TableMaster). Specifically, the table image is encoded and sequentially decoded in sequence through the transformer architecture, and each symbol in the sequence (such as,,, …,) is output. At the same time, optical character recognition (OCR) technology is needed to obtain the text content contained in the table image, and further embed the text content into the table cells in a fill-in-the-blank manner to obtain the final table structure.

[0018] However, the above solution has the following defects: First, it is necessary to produce labeled data for model training, but producing labeled data is difficult and requires a high level of manpower.

[0019] Second, serial decoding becomes time-consuming and uncontrollable as the length of the sequence increases, which not only leads to low table recognition efficiency but also limits the practical application of the above scheme.

[0020] Third, a separate OCR technology is used to process the text content in the image, so the core transformer architecture does not utilize this information, especially when the table in the table image is a wireless table, which limits the scope of application of the above solution. In addition, OCR outputs some adjacent text lines as one instance, which will cause matching errors when filling text content into cells, affecting recognition accuracy.

[0021] In view of this, an embodiment of the present application proposes a table recognition method based on multimodal technology and prompt word guidance technology. First, feature extraction is performed on a table image containing a table to obtain a first visual feature, and a first prompt word is used to describe the table image. By extracting features from the first prompt word, a first text feature is obtained; then, the first visual feature is enhanced based on the first text feature, so that the first text feature implies a display focus on text and cells. This type of display focus can guide the first visual feature, so that the first visual feature is more focused on the text area and cell area in the table image, and a second visual feature is obtained to express the features in a more targeted manner; second; further, through the recognition model, based on the second visual feature, the position information of the cells in the table and the table information such as the text in the cells can be more accurately identified from the table image, and based on these table information, the corresponding table can be accurately generated.

[0022] It can be seen that the table recognition method proposed in the embodiment of the present application does not need to rely on additional OCR technology, which not only avoids the error introduced by OCR, but also makes comprehensive use of image and text information to make the recognition model more robust. In addition, the serial sequence prediction scheme is abandoned, which not only reduces the difficulty of annotation, but also makes it so that in actual applications, the time consumption does not become uncontrollable due to the increase in the number of cells, thereby improving the efficiency of table recognition. The embodiment of the present application also proposes a training method for a recognition model. First, feature extraction is performed on a sample table image containing a sample table to obtain a first sample visual feature, and a second prompt word is used to describe the sample table image. By extracting features from the second prompt word, a first sample text feature is obtained. Then, based on the first sample text feature, the first sample visual feature is enhanced so that the first sample text feature contains implicit display attention to text and cells. This type of display attention can guide the first sample visual feature, so that the first sample visual feature is more focused on the text area and cell area in the table image, and the second sample visual feature is obtained to express the features in a more targeted manner. Second, further, through the recognition model, based on the second sample visual feature, the position information of the cells in the sample table and the table information such as the text in the cells are identified from the sample table image. Based on these table information and reference table information, the parameters of the recognition model are adjusted so that the recognition model can comprehensively learn and utilize the image and text information from the second visual feature accurately, thereby having the ability to accurately recognize the table and having better robustness. In addition, compared with the serial sequence prediction scheme, it not only reduces the difficulty of labeling, but also ensures that in practical applications, the time consumption does not become uncontrollable due to the increase in the number of cells, thereby improving the model training efficiency and table recognition efficiency.

[0023] It should be understood that the table recognition method and the training method of the recognition model provided in the embodiments of the present application can be performed by an electronic device. The so-called electronic device here can include terminal devices, such as smart phones, tablet computers, laptops, desktop computers, intelligent voice interaction devices, smart home appliances, smart watches, vehicle terminals, aircraft, etc.; or, the electronic device can also include a server, such as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0024] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0025] Please refer to Figure 1 , is a flowchart of a table recognition method provided in an embodiment of the present application, the method comprising the following steps: S102, extracting features from a table image containing a table to obtain a first visual feature.

[0026] As an example, Figure 2 As shown in Figure 1, the first visual feature can be obtained by extracting features from the table image through the visual encoder. The first visual feature is a high-dimensional abstraction, which is expressed as ,in, Indicates the first visual feature, Indicates the height of the first visual feature; Indicates the width of the first visual feature; Representing multiple sub-features of the first visual feature, the multiple sub-features describe the visual characteristics of the table image from different dimensions. As an example, the sub-features of multiple dimensions can imply the color channels of the table image and various statistical values ​​in a mathematical sense, such as the mean and variance.

[0027] Among them, the visual encoder can be a part of the recognition model or an encoder independent of the recognition model.

[0028] S104: Characterize the first prompt word used to describe the table image to extract a first text feature.

[0029] The first prompt word can have various appropriate forms, which are not limited in the embodiments of the present application. As an example, the first prompt word includes: a first prompt word unit for describing the cell of the table, and a second prompt word unit for describing the text in the cell. For example, the first prompt word unit can be "cell" or "the image of cell...", etc., and the second prompt word unit can be "text" or "the image of text...", etc. In this way, the first prompt word clearly and finely describes the text information and cell information contained in the table in a progressive description manner, which helps the recognition model to display the use of these table information for table recognition, thereby improving recognition accuracy.

[0030] In the embodiment of the present application, the first text feature can be extracted from the first prompt word in various appropriate ways.

[0031] In one embodiment, the above S104 includes the following steps: performing feature extraction on the first prompt word unit and the second prompt word unit after splicing to obtain a first text feature.

[0032] For example, the first prompt word unit and the second prompt word unit are concatenated and input into a text encoder, and after encoding, a first text feature with a dimension of 2*d is obtained.

[0033] Among them, the text encoder can be a part of the recognition model, or it can be an encoder independent of the recognition model. In a more preferred solution, the text encoder and the visual encoder are both part of the recognition model. In this way, it can be ensured that the text encoder matches the visual encoder so that the text modality and the visual modality are aligned, which is a prerequisite for utilizing the excellent text perception ability of the recognition model.

[0034] S106: Enhance the first visual feature based on the first text feature to obtain a second visual feature.

[0035] The first visual feature is enhanced based on the first text feature, so that the first text feature contains implicit display attention to text and cells. This type of display attention can guide the first visual feature, making the first visual feature more focused on the text area and cell area in the table image, and obtaining the second visual feature for more targeted feature expression.

[0036] In the embodiment of the present application, the first visual feature can be enhanced in various appropriate ways.

[0037] In one embodiment, the above S106 includes the following steps: S161, determining a correlation coefficient between the first text feature and the first visual feature.

[0038] Specifically, first, the first visual feature is mapped to the feature space to which the first text feature belongs to obtain a first mapping feature to ensure the consistency of the two in size. In practical applications, mapping the first visual feature can be achieved through various feature mapping technologies, such as through a multi-layer perceptron (MLP) in a recognition model, which is not limited in the embodiments of the present application.

[0039] Then, the inner product between the first mapping feature and the first text feature is determined, and the product between the length of the first visual feature and the length of the first text feature is determined, and the correlation coefficient between the first visual feature and the first text feature is further determined based on the ratio of the inner product to the product.

[0040] For example, the correlation coefficient between the first visual feature and the first text feature can be determined by the following formula (1): (1) in, represents the correlation coefficient, Indicates the first visual feature, represents the first text feature, represents a multi-layer perceptron, represents the first mapping feature, Indicates the length of the first visual feature, Indicates the length of the first text feature.

[0041] S162: Based on the correlation coefficient and the first visual feature, the first text feature is enhanced to obtain a second text feature.

[0042] Specifically, the first visual feature is mapped to the feature space to which the first text feature belongs to obtain a first mapping feature; based on the correlation coefficient, the first mapping feature and the first text feature are fused to obtain a second text feature.

[0043] Exemplarily, the second text feature can be expressed as ,in, represents the second text feature, represents the correlation coefficient, Indicates the first visual feature, represents the first mapping feature, Represents the first text feature.

[0044] It is understandable that since the first text feature only considers the text information in the table image and does not interact with the image information of the table image itself, this results in the first text feature being the same for different table images. Once the table image to be predicted is inconsistent with the sample table image used for training, there is a risk of reduced performance of the recognition model. To this end, based on the correlation coefficient and the first visual feature, the first text feature is enhanced, and the resulting second text feature not only implies the text information in the table image, but also considers the interaction between the text information and the image information of the table image itself, and has an adaptive characteristic, that is, the same text prompt can generate an adaptive second text feature according to the different forms of sputum smears, thereby effectively avoiding the above technical problems and further improving the recognition accuracy of the table image.

[0045] S163, using a cross attention mechanism to fuse the second text feature and the first visual feature to obtain a second visual feature.

[0046] Attention Mechanism plays an important role in the field of deep learning, especially in many tasks such as Natural Language Processing (NLP) and computer vision. Its core idea is to enable the model to focus on certain specific parts of the input data, just like humans selectively focus on important parts when processing information. Cross-Attention Mechanism is a variant of the attention mechanism, which focuses on the information interaction between different inputs. This interaction can better integrate data information from different modalities (such as text and pictures) or different sources.

[0047] Specifically, if Figure 3 As shown, the above S163 includes the following steps: using the first visual feature as the query vector of the cross-attention mechanism, using the representation vector of the second text feature as the key vector and value vector of the cross-attention mechanism, performing attention operation on the query vector, the key vector and the value vector to obtain a third visual feature; fusing the third visual feature with the first visual feature to obtain a fourth visual feature; mapping the representation vector of the second text feature to the feature space to which the fourth visual feature belongs to obtain a second mapping feature; fusing the second mapping feature with the fourth visual feature to obtain a second visual feature.

[0048] Exemplarily, the fourth visual feature can be expressed as ,in, Indicates the third visual feature, represents the query vector, Indicates the first visual feature, represents the key vector, represents a value vector, Represents the second text feature The representation vector of , whose dimension is [2(n+1)*d].

[0049] The second visual feature can be expressed as ,in, Represents the second mapping feature.

[0050] In the embodiment of the present application, since the second text feature explicitly focuses on the text information and cell information in the table image, by using a cross-attention mechanism to fuse the features of the two modalities of vision and text (i.e., the second text feature and the first visual feature), the obtained second visual feature can be more focused on the text area where the text is located and the cell area where the cell is located in the table image, providing reliable data support for the subsequent accurate identification of the table information in the table image.

[0051] In another embodiment, an attention mechanism is used to fuse the first text feature and the first visual feature to obtain a second visual feature.

[0052] The specific method of fusing the first text feature and the first visual feature is similar to the specific method of fusing the second text feature and the first visual feature described above, and will not be described in detail.

[0053] The above describes some implementation methods of the above S106. Of course, it should be understood that the above S106 can also be implemented in other ways, which are not limited in the present embodiment of the application.

[0054] S108, identifying the table information of the table from the table image through the recognition model based on the second visual feature.

[0055] The table information includes: the position information of the cells in the table and the text in the cells. In addition, the table information may also include the text area in the cells.

[0056] The recognition model may adopt any model with image recognition function, such as a contrastive language-image pre-training (CLIP) model, etc., which is not limited in the embodiments of the present application.

[0057] S110, generating a table based on the table information.

[0058] Specifically, based on the cell position information, the row and column numbers of the cells are determined; according to the row and column numbers of the cells, the identified cells are sorted to obtain the row and column information of the table in the table image; further, for each cell, if the corresponding text is identified from the cell, the text is filled into the cell, thereby realizing table recognition of the table image and obtaining the corresponding table.

[0059] One or more embodiments of the present application provide a table recognition method, first, extracting features from a table image containing a table to obtain a first visual feature, and using a first prompt word to describe the table image, and extracting features from the first prompt word to obtain a first text feature; then, enhancing the first visual feature based on the first text feature, so that the first text feature implicitly contains a display focus on text and cells, and such display focus can guide the first visual feature, so that the first visual feature is more focused on the text area and cell area in the table image, and a second visual feature is obtained, so as to express the features in a more targeted manner; second; further, through the recognition model, based on the second visual feature, the position information of the cells in the table and the table information such as the text in the cells can be more accurately identified from the table image, and based on such table information, the corresponding table can be accurately generated.

[0060] It can be seen that the table recognition method proposed in the embodiment of the present application does not need to rely on additional OCR technology, which not only avoids the error introduced by OCR, but also makes comprehensive use of image and text information to make the recognition model more robust. In addition, the serial sequence prediction scheme is abandoned, which not only reduces the difficulty of annotation, but also makes it so that in actual applications, the time consumption does not become uncontrollable due to the increase in the number of cells, thereby improving the efficiency of table recognition.

[0061] Please refer to Figure 4 , is a flow chart of a method for training a recognition model provided by an embodiment of the present application, the method comprising the following steps: S402: extract features from a sample table image containing a sample table to obtain a first sample visual feature.

[0062] The specific implementation of the above S402 is Figure 1 The specific implementation of S102 in the illustrated embodiment is similar and will not be described in detail.

[0063] S404: extract features of the second prompt word used to describe the sample table image to obtain first sample text features.

[0064] The second prompt word can be in various appropriate forms, which is not limited in the embodiments of the present application. As an example, the second prompt word includes: a first prompt word unit for describing the cells of the sample table, and a second prompt word unit for describing the text in the cell. For example, the first prompt word unit can be "cell" or "the image of cell...", etc., and the second prompt word unit can be "text" or "the image of text...", etc. In this way, the second prompt word describes the text information and cell information contained in the sample table in a progressive description manner, which helps the recognition model to display the use of these table information for table recognition, thereby improving recognition accuracy.

[0065] In the embodiment of the present application, the first sample text feature can be extracted from the second prompt word in various appropriate ways.

[0066] In one implementation, the above S404 includes the following steps: directly performing feature extraction on the second prompt word to obtain a second text feature.

[0067] In another embodiment, considering that the first prompt word unit and the second prompt word unit are too simple and have limited information, and the second prompt word has a greater impact on the final recognition effect, that is, the quality of the second prompt word seriously affects the accuracy of subsequent tasks. For example, if the training stage is only in the following format "the image of text", it may not be able to adapt to the type of "this is a image of text", although they express the same meaning. In addition, it is also considered that the change of the prompt word is usually accompanied by fine-tuning of the model.

[0068] Based on the above two considerations, the embodiment of the present application adopts an adaptive method to enable the recognition model to adapt to different second prompt words. For example, for the recognition model, the image of text will be split into a sequence such as [the, image, of, text], and each word in the sequence will be represented as a feature vector. Since it is impossible to enumerate various descriptions in advance, a learnable parameter can be added before each prompt word unit so that the recognition model can learn this description during the training process, so as to adapt to the second prompt words of various expressions.

[0069] Specifically, the parameters of the recognition model include a first learnable parameter and a second learnable parameter, the dimensions of the two are the same, the first learnable parameter is used for the recognition model to learn different expressions of the first prompt word unit, and the second learnable parameter is used for the recognition model to learn different expressions of the second prompt word unit. Accordingly, the above S404 includes the following steps: adding the first learnable parameter before the first prompt word unit, and adding the second learnable parameter before the second prompt word unit to obtain a third prompt word; extracting features of the third prompt word to obtain features of the first sample text.

[0070] For example, Figure 5 As shown, the second prompt word includes a first sentence and a second sentence. The first sentence includes a first prompt word unit, followed by the cell information of the described sample table; the second sentence includes a second prompt word unit, followed by the text information in the described cell.

[0071] In this case, the first prompt word unit in the first sentence is encoded using the text encoder to obtain a vector t1 of dimension d; the first learnable parameter is encoded using the text encoder to obtain a vector R1 of size n*d (n represents the number, d represents the dimension); the vector R1 is concatenated before the vector t1 to obtain a vector t1' of dimension (n+1)*d, which is the text feature of the first sentence.

[0072] Similarly, the text encoder encodes the second prompt word unit in the second sentence to obtain a vector t2 of dimension d; the text encoder is used to encode the second learnable parameter to obtain a vector R2 of size n*d (n represents the number, d represents the dimension); the vector R2 is concatenated before the vector t2 to obtain a vector t2' of dimension (n+1)*d, which is the text feature of the second sentence.

[0073] Furthermore, the text features of the first sentence and the text features of the second sentence are concatenated to obtain the first sample text features, which is expressed as T=[2(n+1)*d].

[0074] S406: Enhance the first sample visual feature based on the first sample text feature to obtain a second sample visual feature.

[0075] The specific implementation of S406 is the same as Figure 1 The specific implementation of S106 in the illustrated embodiment is similar and will not be described in detail.

[0076] S408, identifying the table information of the sample table from the table image through the recognition model based on the second sample visual feature.

[0077] The specific implementation of S408 above is Figure 1 The specific implementation of S108 in the illustrated embodiment is similar and will not be described in detail.

[0078] The table information includes: the position information of the cells in the sample table and the text in the cells. In addition, the table information may also include: the position information of the text area in the cells.

[0079] S410, adjusting parameters of the recognition model based on the table information of the sample table and the reference table information.

[0080] Secondly, the reference table information refers to the actual table information of the sample table, which can be obtained through manual annotation. The reference table information may include: reference position information of cells in the sample table, reference text in cells, reference position information of text areas in cells, etc.

[0081] The difference between the table information output by the recognition model and the reference table information of the sample table can reflect the recognition accuracy of the recognition model. By adjusting the parameters of the recognition model based on this, the performance of the recognition model can be continuously optimized to ensure that the recognition model can recognize the table contained in any table image.

[0082] In one embodiment, the above-mentioned S410 includes the following steps: determining a first loss based on the position information and reference position information of the cell, the position information and reference position information of the text area in the cell; determining a second loss based on the text in the cell and the reference text; determining an overlapping area between the cell and the text area based on the position information of the cell and the position information of the text area in the cell, and determining a third loss based on the area ratio of the overlapping area to the text area; adjusting the parameters of the recognition model based on the first loss, the second loss and the third loss.

[0083] The first loss is used to represent the recognition accuracy of the recognition model for the location information of the cell and the location information of the text area. As an example, the first loss may be L1 loss.

[0084] The second loss is used to represent the recognition accuracy of the recognition model for the text in the cell. As an example, the second loss can adopt the cross entropy loss.

[0085] The third loss is used to represent the degree of match between the position of the cell identified by the recognition model and the position of the text area. As an example, considering that the text area is usually located in the cell, the third loss can be determined by the following formula (2): (2) in, represents the third loss, Represents the position coordinates of the cell identified by the recognition model, Represents the position coordinates of the text area in the cell recognized by the recognition model. Represents the overlapping area between the cell and the text area. Indicates the area of ​​the text area.

[0086] Furthermore, the weighted sum of the first loss, the second loss and the third loss is taken as the total loss of the recognition model, and based on the total loss of the recognition model, the back propagation algorithm, the gradient descent algorithm, etc. are used to adjust the parameters of the recognition model.

[0087] It can be understood that when calculating the loss of the recognition model, not only the loss caused by recognizing the position of the cell and the position of the text area, the loss caused by recognizing the text content in the cell, but also the loss caused by the matching degree between the position of the cell and the position of the text area is considered. The parameters of the recognition model are adjusted based on these three types of losses, so that the recognition model can accurately identify the cells in the table image and accurately identify the text content contained in the cells.

[0088] In another embodiment, the above S410 includes the following steps: determining a first loss based on the position information and reference position information of the cell, the position information and reference position information of the text area in the cell; determining a second loss based on the text in the cell and the reference text; determining the weighted sum of the first loss and the second loss as the total loss of the recognition model, and adjusting the parameters of the recognition model based on the total loss of the recognition model.

[0089] The above describes some implementation methods of the above S410. Of course, it should be understood that the above S410 can also be implemented in other ways, and the present application embodiment does not limit this.

[0090] The above steps S402~S410 are only a training process for the recognition model. In practical applications, the recognition model can be trained multiple times until the training stop condition is met, and the recognition model obtained can be used to implement the table recognition method provided in the embodiment of the present application. Among them, the training stop condition can be set according to actual needs, such as the number of training times reaches a threshold or the total loss of the recognition model converges, etc., which is not limited in the embodiment of the present application.

[0091] In practical applications, the recognition model may have various appropriate structures, and the above steps S402 to S408 may be implemented by corresponding modules in the recognition model. Figure 6 As shown, the recognition model is a multimodal model, which includes: a text encoder, a visual encoder, an embedding network, an attention network, and a prediction network.

[0092] Specifically, after the sample table image and the second prompt word are input into the recognition model, the text encoder extracts the features of the second prompt word to obtain the first sample text features; the visual encoder extracts the features of the sample table image to obtain the first sample text features.

[0093] Then, the recognition model determines the correlation coefficient between the first sample text feature and the first sample visual feature. The embedding network enhances the first sample text feature based on the correlation coefficient and the first sample visual feature to obtain the second sample text feature.

[0094] Furthermore, the attention network adopts a cross-attention mechanism to fuse the second sample text features and the first sample visual features to obtain the second sample visual features.

[0095] Furthermore, the prediction network identifies table information of the sample table from the sample table image based on the second sample visual feature, and the table information includes: position information of cells in the sample table, text information in the cells, position information of text areas in the cells, etc.

[0096] Finally, the first loss is determined based on the cell's position information and reference position information, the text area's position information and reference position information within the cell; the second loss is determined based on the text within the cell and the reference text; the third loss is determined based on the cell's position information and the text area's position information within the cell; and the parameters of the recognition model are adjusted based on the first loss, the second loss and the third loss.

[0097] Among them, the parameters of the recognition model may include, for example, but are not limited to: parameters of each module in the recognition model, a first learnable parameter added before the first prompt word unit, a second learnable parameter added before the second prompt word unit, etc.

[0098] One or more embodiments of the present application provide a method for training a recognition model. First, feature extraction is performed on a sample table image containing a sample table to obtain a first sample visual feature, and a second prompt word is used to describe the sample table image. By extracting features from the second prompt word, a first sample text feature is obtained. Then, based on the first sample text feature, the first sample visual feature is enhanced so that the first sample text feature contains implicit display attention to text and cells. This type of display attention can guide the first sample visual feature, so that the first sample visual feature is more focused on the text area and cell area in the table image, and the second sample visual feature is obtained to express the features in a more targeted manner. Second, further, through the recognition model, based on the second sample visual feature, the position information of the cells in the sample table and the table information such as the text in the cells are identified from the sample table image. Based on these table information and reference table information, the parameters of the recognition model are adjusted so that the recognition model can comprehensively learn and utilize the image and text information from the second visual feature accurately, thereby having the ability to accurately recognize the table and having better robustness. In addition, compared with the serial sequence prediction scheme, it not only reduces the difficulty of labeling, but also ensures that in practical applications, the time consumption does not become uncontrollable due to the increase in the number of cells, thereby improving the model training efficiency and table recognition efficiency.

[0099] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0100] Based on the same inventive concept, the present application also provides a table recognition device. Figure 7 , is a structural diagram of a table recognition device 700 provided in one embodiment of the present application, the device 700 includes: A first extraction module 710 is used to extract features from a table image containing a table to obtain a first visual feature, and to extract features from a first prompt word used to describe the table image to obtain a first text feature; A first enhancement module 720, configured to enhance the first visual feature based on the first text feature to obtain a second visual feature; A first recognition module 730 is configured to recognize table information of the table from the table image based on the second visual feature by using a recognition model, wherein the table information includes: position information of cells in the table and text in the cells; The first generating module 740 is configured to generate the table based on the table information.

[0101] In another embodiment, the first enhancement module is used to: determining a correlation coefficient between the first text feature and the first visual feature; Based on the correlation coefficient and the first visual feature, the first text feature is enhanced to obtain a second text feature; The second text feature and the first visual feature are fused by using a cross attention mechanism to obtain a second visual feature.

[0102] In another embodiment, the second text feature is obtained by: Mapping the first visual feature to a feature space to which the first text feature belongs to obtain a first mapping feature; Based on the correlation coefficient, the first mapping feature and the first text feature are fused to obtain a second text feature.

[0103] In another embodiment, the second visual feature is obtained by: Using the first visual feature as a query vector of the cross attention mechanism, using the representation vector of the second text feature as a key vector and a value vector of the cross attention mechanism, performing an attention operation on the query vector, the key vector and the value vector, to obtain a third visual feature; fusing the third visual feature and the first visual feature to obtain a fourth visual feature; Mapping the representation vector to a feature space to which the fourth visual feature belongs to obtain a second mapping feature; The second mapping feature and the fourth visual feature are fused to obtain a second visual feature.

[0104] Obviously, the table recognition device 700 provided in the embodiment of the present application can be used as the above Figure 1 The execution subject of the table recognition method shown in the figure can thus realize the table recognition device in Figure 1 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0105] Based on the same inventive concept, the present application also provides a training device for a recognition model. Figure 8 , is a schematic diagram of a structure of a recognition model training device 800 provided in one embodiment of the present application, the device 800 includes: A second extraction module 810 is used to extract features from a sample table image containing a sample table to obtain a first sample visual feature, and to extract features from a second prompt word used to describe the sample table image to obtain a first sample text feature; A second enhancement module 820, configured to enhance the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; A second recognition module 830 is used to recognize table information of the sample table from the table image based on the second sample visual features by using a recognition model, wherein the table information includes: position information of cells in the sample table and text in the cells; The first adjustment module 840 is used to adjust the parameters of the recognition model based on the table information of the sample table and the reference table information.

[0106] In another embodiment, the table information further includes: location information of the text area within the cell; The reference table information includes: reference position information of the cell, reference text in the cell, and reference position information of a text area in the cell.

[0107] The first adjustment module is used for: Determine a first loss based on the position information and reference position information of the cell, and the position information and reference position information of the text area in the cell; Determining a second loss based on the text in the cell and the reference text; Determining an overlapped area between the cell and the text area based on the position information of the cell and the position information of the text area in the cell, and determining a third loss based on an area ratio of the overlapped area to the text area; Based on the first loss, the second loss, and the third loss, parameters of the recognition model are adjusted.

[0108] In another embodiment, the parameter includes a first learnable parameter and a second learnable parameter. The second prompt word includes: a first prompt word unit for describing a cell in the sample table, and a second prompt word unit for describing a text in the cell.

[0109] The second extraction module is used for: Adding the first learnable parameter before the first prompt word unit, and adding the second learnable parameter before the second prompt word unit, to obtain a third prompt word; Feature extraction is performed on the third prompt word to obtain a first sample text feature.

[0110] Obviously, the training device 800 for the recognition model provided in the embodiment of the present application can be used as the above Figure 4 The execution subject of the training method of the recognition model shown in the figure can realize the training device of the recognition model in Figure 4 The functions realized are as follows. Since the principles are the same, they will not be described in detail.

[0111] Fig. 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Fig. 9 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0112] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0113] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0114] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a table recognition device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Extracting features of a table image containing a table to obtain a first visual feature, and extracting features of a first prompt word used to describe the table image to obtain a first text feature; Performing enhancement processing on the first visual feature based on the first text feature to obtain a second visual feature; Recognize, by using a recognition model and based on the second visual feature, table information of the table from the table image, the table information including: position information of cells in the table and text in the cells; Based on the table information, the table is generated.

[0115] Alternatively, the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a training device for the recognition model at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations: Extracting features of a sample table image containing a sample table to obtain a first sample visual feature, and extracting features of a second prompt word used to describe the sample table image to obtain a first sample text feature; Performing enhancement processing on the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; Using a recognition model, based on the second sample visual features, identifying table information of the sample table from the table image, the table information including: position information of cells in the sample table and text in the cells; Based on the table information of the sample table and the reference table information, the parameters of the recognition model are adjusted.

[0116] The above application Figure 1 The table recognition device disclosed in the embodiment shown or the above-mentioned application Figure 4The method performed by the training device of the recognition model disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0117] The electronic device may also perform Figure 1 The method and the table recognition device are implemented in Figures 1 to 5 Alternatively, the electronic device may also perform the functions of the embodiments shown. Figure 4 The method and the training device of the recognition model are implemented in Figure 4 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0118] Of course, in addition to software implementation, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0119] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1The method of the embodiment shown is specifically used to perform the following operations: Extracting features of a table image containing a table to obtain a first visual feature, and extracting features of a first prompt word used to describe the table image to obtain a first text feature; Performing enhancement processing on the first visual feature based on the first text feature to obtain a second visual feature; Recognize, by using a recognition model and based on the second visual feature, table information of the table from the table image, the table information including: position information of cells in the table and text in the cells; Based on the table information, the table is generated.

[0120] Alternatively, when the instruction is executed by an electronic device including multiple application programs, the electronic device can execute Figure 4 The method of the embodiment shown is specifically used to perform the following operations: Extracting features of a sample table image containing a sample table to obtain a first sample visual feature, and extracting features of a second prompt word used to describe the sample table image to obtain a first sample text feature; Performing enhancement processing on the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; Using a recognition model, based on the second sample visual features, identifying table information of the sample table from the table image, the table information including: position information of cells in the sample table and text in the cells; Based on the table information of the sample table and the reference table information, the parameters of the recognition model are adjusted.

[0121] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps in the table recognition method or recognition model training provided in the embodiment of the present application.

[0122] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0123] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0124] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0126] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A table recognition method, characterized in that: include: Extracting features of a table image containing a table to obtain a first visual feature, and extracting features of a first prompt word used to describe the table image to obtain a first text feature; Performing enhancement processing on the first visual feature based on the first text feature to obtain a second visual feature; Recognize, by using a recognition model and based on the second visual feature, table information of the table from the table image, the table information including: position information of cells in the table and text in the cells; Based on the table information, the table is generated.

2. The method according to claim 1, characterized in that The step of enhancing the first visual feature based on the first text feature to obtain a second visual feature includes: determining a correlation coefficient between the first text feature and the first visual feature; Based on the correlation coefficient and the first visual feature, the first text feature is enhanced to obtain a second text feature; The second text feature and the first visual feature are fused by using a cross attention mechanism to obtain a second visual feature.

3. The method according to claim 2, characterized in that The step of enhancing the first text feature based on the correlation coefficient and the first visual feature to obtain a second text feature includes: Mapping the first visual feature to a feature space to which the first text feature belongs to obtain a first mapping feature; Based on the correlation coefficient, the first mapping feature and the first text feature are fused to obtain a second text feature.

4. The method according to claim 2, characterized in that: The cross attention mechanism is adopted to fuse the second text feature and the first visual feature to obtain the second visual feature, including: Using the first visual feature as a query vector of the cross attention mechanism, using the representation vector of the second text feature as a key vector and a value vector of the cross attention mechanism, performing an attention operation on the query vector, the key vector and the value vector, to obtain a third visual feature; fusing the third visual feature and the first visual feature to obtain a fourth visual feature; Mapping the representation vector to a feature space to which the fourth visual feature belongs to obtain a second mapping feature; The second mapping feature and the fourth visual feature are fused to obtain a second visual feature.

5. A method for training a recognition model, characterized in that: include: Extracting features of a sample table image containing a sample table to obtain a first sample visual feature, and extracting features of a second prompt word used to describe the sample table image to obtain a first sample text feature; Performing enhancement processing on the first sample visual feature based on the first sample text feature to obtain a second sample visual feature; Using a recognition model, based on the second sample visual features, identifying table information of the sample table from the table image, the table information including: position information of cells in the sample table and text in the cells; Based on the table information of the sample table and the reference table information, the parameters of the recognition model are adjusted.

6. The method according to claim 5, characterized in that The table information also includes: location information of the text area in the cell; The reference table information includes: reference position information of the cell, reference text in the cell, and reference position information of the text area in the cell; The adjusting the parameters of the recognition model based on the table information of the sample table and the reference table information includes: Determine a first loss based on the position information and reference position information of the cell, and the position information and reference position information of the text area in the cell; Determining a second loss based on the text in the cell and the reference text; Determining an overlapped area between the cell and the text area based on the position information of the cell and the position information of the text area in the cell, and determining a third loss based on an area ratio of the overlapped area to the text area; Based on the first loss, the second loss, and the third loss, parameters of the recognition model are adjusted.

7. The method according to claim 5, characterized in that The parameters include a first learnable parameter and a second learnable parameter; The second prompt word includes: a first prompt word unit for describing a cell in the sample table, and a second prompt word unit for describing a text in the cell; The step of extracting features of the second prompt word used to describe the sample table image to obtain first sample text features includes: Adding the first learnable parameter before the first prompt word unit, and adding the second learnable parameter before the second prompt word unit, to obtain a third prompt word; Feature extraction is performed on the third prompt word to obtain a first sample text feature.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the table recognition method as described in any one of claims 1 to 4; or, the processor is configured to execute the instructions to implement the recognition model training method as described in any one of claims 5 to 7.

9. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the table recognition method as described in any one of claims 1 to 4; or, when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the recognition model as described in any one of claims 5 to 7.

10. A computer program product, characterized in that The computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to execute part or all of the steps in the table recognition method according to any one of claims 1 to 4 or the recognition model training method according to any one of claims 5 to 7.

Images