Table picture matching method, system, medium and device

Abstract

The invention provides a table picture matching method, system, medium and device. The table picture matching method comprises the following steps: acquiring a first table picture and a second table picture to be compared; performing feature extraction based on the first table picture to obtain a first matching feature; performing feature extraction based on the second table picture to obtain a second matching feature; calculating the pairwise similarity of the first matching feature and the second matching feature; and determining the matching degree of the first table picture and the second table picture based on the pairwise similarity. According to the method, on the basis of extracting the visual features of the table pictures, the text features and the title features are extracted at the same time, so that the matching precision of different table pictures can be effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of natural language processing technology and relates to a matching method, and in particular to a table image matching method, system, medium and device. Background Art

[0002] In actual application scenarios, it is often necessary to compare the matching degree of two table images. Table image matching mainly refers to the subtask of image matching, which is actually image retrieval. That is, when a user inputs an image and wants to find images with similar content, the image matching technology is used to find images with similar content and return them to the user; a more specific application is image search.

[0003] With the development of deep learning technology, especially the application of convolutional neural networks (CNN) in the field of image recognition, the matching technology of table images has been significantly improved. Existing algorithms can accurately identify the text, lines and cell structures in tables, and then realize the automatic extraction and matching of tables. This makes the matching of table images applicable in many scenarios, such as extracting table data from scanned documents or photos, and comparing and merging information between different tables.

[0004] At present, one of the core problems of table image matching methods is that the matching accuracy is not high enough. Although deep learning algorithms have made great progress in image recognition, faced with complex table structures and variable image quality, algorithms often have difficulty accurately extracting and matching key information in tables. In particular, when there are blurred, distorted, overlapping or irregularly formatted text, lines and cells in the table, the matching accuracy will further decrease, resulting in incorrect data extraction and matching results.

[0005] Therefore, in view of the above reasons, especially in the prior art, the matching of table images is mainly completed by extracting visual features of the table images. However, this method has the problem of low matching accuracy.

[0006] Therefore, how to improve the accuracy of table image matching is a problem that needs to be solved urgently. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a table image matching method, system, medium and device for solving the problem of low image matching accuracy due to only using visual features for matching in the existing image matching technology.

[0008] To achieve the above-mentioned purpose and other related purposes, in a first aspect, the present application provides a table image matching method, comprising the following steps: obtaining a first table image and a second table image to be compared; performing feature extraction based on the first table image to obtain a first matching feature; performing feature extraction based on the second table image to obtain a second matching feature; calculating the pairwise similarity between the first matching feature and the second matching feature; and determining the degree of matching between the first table image and the second table image based on the pairwise similarity.

[0009] In an implementation of the first aspect, performing feature extraction based on the first table image to obtain a first matching feature includes the following steps: performing feature extraction on the first table image to obtain a first title feature, a first visual feature, and a first text feature, respectively; taking the first title feature as a K vector, the first visual feature as a V vector, and the first text feature as a Q vector, and obtaining the first matching feature based on a cross-attention mechanism.

[0010] In an implementation manner of the first aspect, a first visual feature of the first table image is extracted based on a VIT model.

[0011] In an implementation of the first aspect, performing feature extraction on the first table image to obtain a first text feature includes the following steps: performing text recognition on the first table image to obtain a text row position and a text row content; extracting all the text, vertical text, and horizontal text in the first table image according to the text row position; extracting text semantic features of the all the text, the vertical text, and the horizontal text based on a BERT model to obtain a first semantic feature of all the text, a first semantic feature of the vertical text, and a first semantic feature of the horizontal text; and calculating an average of the first semantic feature of all the text, the first semantic feature of the vertical text, and the first semantic feature of the horizontal text to obtain a first text feature.

[0012] In an implementation of the first aspect, feature extraction is performed on the first table image to obtain a first title feature, including the following steps: in combination with text prompts, the vertical text and the horizontal text in the text row position are input into a large language model to obtain column titles and row titles; a CLIP-based text encoder is used to extract semantic features of the column titles and the row titles to obtain column title semantic features and row title semantic features; an average value of the column title semantic features and the row title semantic features is calculated to obtain a first title feature.

[0013] In an implementation of the first aspect, taking the first title feature as a K vector, taking the first visual feature as a V vector, and taking the first text feature as a Q vector, obtaining a first matching feature based on a cross-attention mechanism includes the following steps: taking the first title feature as a K vector, taking the first visual feature as a V vector, and taking the first text feature as a Q vector, and obtaining a first fused feature based on a cross-attention mechanism; inputting the first fused feature into a multi-layer perceptron to obtain a first fused feature vector; and inputting the first fused feature vector into a sigmoid function to obtain the first matching feature.

[0014] In an implementation of the first aspect, performing feature extraction based on the second table image to obtain a second matching feature includes the following steps: performing feature extraction on the second table image to obtain a second title feature, a second visual feature, and a second text feature, respectively; using the second title feature as a K vector, the second visual feature as a V vector, and the second text feature as a Q vector, and obtaining the second matching feature based on a cross-attention mechanism.

[0015] In a second aspect, the present application provides a table image matching system, comprising: an acquisition module, a first feature extraction module, a second feature extraction module and an image matching module; the acquisition module is used to acquire a first table image and a second table image to be compared; the first feature extraction module is used to perform feature extraction based on the first table image to obtain a first matching feature; the second feature extraction module is used to perform feature extraction based on the second table image to obtain a second matching feature; the image matching module is used to calculate the pairwise similarity between the first matching feature and the second matching feature, and determine the degree of matching between the first table image and the second table image based on the pairwise similarity.

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by an electronic device, the table image matching method described in any one of the first aspects of the present application is implemented.

[0017] In the last aspect, the present application provides a table image matching device, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the table image matching device performs the table image matching method.

[0018] As described above, the table image matching method, system, medium and device of the present invention have the following beneficial effects: The present application provides a table image matching method, which extracts features from two different table images to be compared, obtains the text features, title features and visual features of each table image, and then inputs the text features, title features and visual features corresponding to the table images into the cross-attention mechanism to extract the matching features corresponding to different table images; then calculates the cosine similarity of the matching features of the two table images to be compared; then, determines the matching degree of the two table images based on the cosine similarity, which can effectively improve the precision and accuracy of the matching of different table images. At the same time, the matching method adopted in the present application can improve the robustness of different image matching technologies, and further improve the matching efficiency and security. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present application.

[0020] Figure 2 Shown is a schematic flow chart of a table image matching method in one embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of the matching process flow of different table images in an embodiment of the table image matching method described in this application.

[0022] Figure 4 Shown is a schematic diagram of the flow of S2 in the table image matching method described in this application.

[0023] Figure 5 Shown is a schematic diagram of the process of extracting the first text feature in an embodiment of the table image matching method described in this application.

[0024] Figure 6 Shown is a schematic diagram of the process of extracting the first title feature in an embodiment of the table image matching method described in this application.

[0025] Figure 7 Shown is a schematic diagram of the flow of S22 in the table image matching method described in this application.

[0026] Figure 8 Shown is a flow chart of S3 in the table image matching method described in this application.

[0027] Fig. 9 Shown is a schematic diagram of the principle structure of the table image matching system described in this application in one embodiment.

[0028] Fig.10 Shown is a schematic diagram of the principle structure of the table image matching device described in this application in one embodiment.

[0029] Component number description 11 cell phone 12 Tablet 13 Laptop 91 Get Module 92 The first feature extraction module 93 The second feature extraction module 94 Image matching module 101 processor 102 Memory S1~S4 step DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0032] The following embodiments of the present application provide a table image matching method, system, medium and device, which respectively perform feature extraction from two different table images to be compared, obtain text features, title features and visual features of each table image, and then input the text features, title features and visual features corresponding to the table images into a cross-attention mechanism to extract the matching features corresponding to different table images; then calculate the cosine similarity of the matching features of the two table images to be compared; then, determine the matching degree of the two table images based on the cosine similarity, which can effectively improve the matching accuracy of different table images.

[0033] The table image matching method of the present application can be applied to Figure 1 The electronic device described in this application may include a mobile phone 11 with wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of this application do not impose any restrictions on the specific types of electronic devices.

[0034] For example, the electronic device may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a personal digital assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved public land mobile network (PLMN), or a mobile terminal in a future evolved non-terrestrial network (NTN).

[0035] For example, the electronic device can communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Satellite Navigation System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS).

[0036] The table image matching method provided in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0037] See also Figure 2 and Figure 3 , respectively showing a flow chart of the table image matching method described in the present application in one embodiment and a flow chart of the matching process of different table images in one embodiment of the table image matching method described in the present application. Figure 2 and Figure 3 As shown, this embodiment provides a table image matching method.

[0038] The table image matching method specifically comprises the following steps: S1, obtaining a first table image and a second table image to be compared.

[0039] In this embodiment, the table image in this application specifically refers to an image that only contains a table (the table of course contains text). The table image can be obtained by reading from a local file, downloading from the Internet, or uploading by a user.

[0040] It should be noted that the first table image and the second table image can be read through image processing tools (such as OpenCV, PIL, etc.).

[0041] S2, extracting features based on the first table image to obtain a first matching feature. Figure 4 , which is a schematic diagram of the process of S2 in the table image matching method described in this application. Figure 4 As shown, S2 comprises the following steps: S21, performing feature extraction on the first table image to obtain a first title feature, a first visual feature and a first text feature respectively.

[0042] In this embodiment, the first visual feature of the first table image is extracted based on the VIT model.

[0043] Specifically, first, the first table image is preprocessed, such as adjusting the size of the table image, scaling, denoising, binarization, and normalization operations, to ensure that the quality and format of the input data meet the requirements of the VIT model, which helps to improve the performance and stability of the model. Then, the preprocessed table image is divided into several small blocks, that is, each small block is a patch. Then, for each image patch, it is converted into an embedding vector in a high-dimensional space through linear projection (such as a fully connected layer) to capture the visual features of the patch. The self-attention mechanism in the Transformer model is then used to process these embedding vectors. By calculating the correlation between each patch, the model can learn the dependency between different patches and extract visual features that contribute to the final task. Finally, the feature vectors of all patches are aggregated to form a feature representation of a complete table image, which is the first visual feature of the first table image.

[0044] Similarly, feature extraction is performed on the first table image to obtain the first text feature. Figure 5 , which is a schematic diagram of the process of extracting the first text feature of the table image matching method described in the present application in one embodiment.

[0045] In this embodiment, text recognition is performed on the first table image to obtain the text row position and the text row content; based on the text row position, all the text, vertical text, and horizontal text in the first table image are extracted; based on the BERT model, text semantic features of all the text, the vertical text, and the horizontal text are extracted to obtain the first semantic feature of all the text, the first semantic feature of the vertical text, and the first semantic feature of the horizontal text; the average value of the first semantic feature of all the text, the first semantic feature of the vertical text, and the first semantic feature of the horizontal text is calculated to obtain the first text feature.

[0046] Specifically, the OCR (Optical Character Recognition) method is used to recognize the first input table image, and the text row position and text row content can be obtained. Then, according to the text row position, the text row of each column is found and named as vertical text (there are multiple columns); the text row of each row is found and named as horizontal text (there are multiple rows). Then, for all texts (arranged from top to bottom and from left to right in the order of text rows, and each text row is spliced ​​with commas), vertical texts and horizontal texts, the BERT model is used to extract text semantic features, and three types of text semantic features can be obtained, namely: the first semantic feature of all texts, the first vertical text semantic feature and the first horizontal text semantic feature. Next, the average of the first semantic feature of all texts, the first vertical text semantic feature and the first horizontal text semantic feature is calculated, and the result is used as the first text feature.

[0047] The BERT (Bidirectional Encoder Representation from Transformers) model is a Transformer-based pre-trained language model that is pre-trained with a large amount of unsupervised corpus and then fine-tuned with a small amount of annotated corpus to complete specific NLP tasks. The core of the BERT model is based on the Transformer structure, especially its Encoder part, which is stacked with multiple layers of Transformer encoders (BertLayer), each of which contains a self-attention mechanism and a feedforward neural network. BERT's pre-training tasks include Masked Language Model (MLM) and Next Sentence Prediction (NSP). The former randomly masks some words in a sentence and predicts the masked content, while the latter predicts whether two sentences are from the same document. These pre-training tasks enable BERT to learn sentence-level semantic information, so that it performs well in a variety of NLP tasks.

[0048] Extract features from the first table image to obtain the first title feature. Figure 6 , which is a schematic diagram of the process of extracting the first title feature in one embodiment of the table image matching method described in the present application.

[0049] In this embodiment, in combination with text prompts, the vertical text and horizontal text extracted from the text row position are input into a large language model to obtain column titles and row titles; a CLIP-based text encoder extracts semantic features of the column titles and row titles to obtain column title semantic features and row title semantic features; the average value of the column title semantic features and the row title semantic features is calculated to obtain a first title feature.

[0050] Specifically, the horizontal text and vertical text obtained in the above steps are used as the input of the large language model LLM, and the input is set as: "Find the row title in the text. The text content is like horizontal text. If there is a title, directly answer the title name without explanation. If there is no title, answer no without explanation." After inputting the text into the LLM, the answer (row title) is obtained; similarly, the column title can be obtained. Then, the semantic features of the column title and row title are extracted respectively through the CLIP text encoder to obtain the column title semantic feature and the row title semantic feature, and then the column title semantic feature and the row title semantic feature are averaged to obtain the first title feature.

[0051] For example, after inputting horizontal text into the LLM model, the answer is obtained, namely, the row title; after inputting vertical text into the LLM model, the answer is obtained, namely, the column title. The extracted row title and column title content are preprocessed, such as removing noise, punctuation marks, etc., and performing necessary text cleaning and normalization. This helps to improve the text encoder's ability to understand and represent text content. Then, the preprocessed column title and row title are input into the CLIP text encoder to extract key features and convert them into vectors to obtain column title semantic features and row title semantic features. Next, the extracted column title semantic features and row title semantic features are averaged to obtain the first title feature.

[0052] It should be noted that the CLIP text encoder is trained based on a large-scale dataset. Therefore, when extracting text features, it can capture broader and deeper semantic information, which makes the text feature matrix extracted based on the CLIP text encoder have better performance in text classification, clustering, retrieval and other tasks.

[0053] S22, taking the first title feature as a K vector, taking the first visual feature as a V vector, taking the first text feature as a Q vector, and obtaining a first matching feature based on a cross attention mechanism. Figure 7 , showing a flow chart of S22 in the table image matching method described in the present application.

[0054] In this embodiment, the first title feature is used as the K vector, the first visual feature is used as the V vector, and the first text feature is used as the Q vector, and a first fused feature is obtained based on a cross-attention mechanism; the first fused feature is input into a multi-layer perceptron to obtain a first fused feature vector; the first fused feature vector is input into a sigmoid function to obtain the first matching feature.

[0055] Specifically, first, the first title feature is used as the K vector, the first visual feature is used as the V vector, and the first text feature is used as the Q vector. The similarity between the first title feature and the first text feature is calculated through the attention layer, that is, the similarity between the first title feature and the first text feature is calculated using the dot product, cosine similarity or other similarity measurement methods; then, these similarity values ​​are normalized into attention weights through the softmax function. Then, the first visual feature is weighted according to these similarities to obtain the weighted feature, that is, the first fusion feature.

[0056] Furthermore, the first fused feature is used as input and sent to the constructed MLP model. The MLP will perform forward propagation and transform and process the input features through the weights and activation functions between each layer. After calculation, the MLP model will output one or more feature vectors. These output vectors are the results of the original fused features after being mapped by the MLP, and usually represent the abstract representation of the input features at a higher level. The output of the MLP is used as the input of the sigmoid function. The sigmoid function maps the input value to a range between 0 and 1 to obtain a probability or activation representation. The output of the sigmoid function is the first matching feature. This feature combines the original fused features with the nonlinear transformation capabilities of the MLP and the mapping characteristics of the sigmoid function.

[0057] It should be noted that the specific implementation steps may be adjusted according to the specific task and dataset. For example, the parameters of the MLP may need to be tuned to achieve better performance; or other processing steps may need to be added before or after the sigmoid function, such as normalization, dimensionality reduction, etc.

[0058] S3, extracting features based on the second table image to obtain a second matching feature. Figure 8 , which is a flow chart of S3 in the table image matching method described in this application. Figure 8 As shown, S3 comprises the following steps: S31, performing feature extraction on the second table image to obtain a second title feature, a second visual feature and a second text feature respectively.

[0059] In this embodiment, the second visual feature of the second table image is extracted based on the VIT model.

[0060] Specifically, first, the second table image is preprocessed, such as adjusting the size of the table image, scaling, denoising, binarization, and normalization operations, to ensure that the quality and format of the input data meet the requirements of the VIT model, which helps to improve the performance and stability of the model. Then, the preprocessed table image is divided into several small blocks, that is, each small block is a patch. Then, for each image patch, it is converted into an embedding vector in a high-dimensional space through linear projection (such as a fully connected layer) to capture the visual features of the patch. These embedding vectors are then processed using the self-attention mechanism in the Transformer model. By calculating the correlation between each patch, the model can learn the dependency between different patches and extract visual features that contribute to the final task. Finally, the feature vectors of all patches are aggregated to form a feature representation of a complete table image, which is the second visual feature of the second table image.

[0061] Similarly, feature extraction is performed on the second table image to obtain a second text feature.

[0062] In this embodiment, text recognition is performed on the second table image to obtain the text row position and the text row content; based on the text row position, all the text, vertical text, and horizontal text in the second table image are extracted; based on the BERT model, text semantic features of all the text, the vertical text, and the horizontal text are extracted to obtain the second all-text semantic feature, the second vertical text semantic feature, and the second horizontal text semantic feature; the average value of the second all-text semantic feature, the second vertical text semantic feature, and the second horizontal text semantic feature is calculated to obtain the second text feature.

[0063] Specifically, the OCR method is used to identify the input second table image to obtain the text row position and text row content. According to the text row position, find the text row in each column and name it vertical text (there are multiple columns); find the text row in each row and name it horizontal text (there are multiple rows). Then, for all text (arranged in order from top to bottom and from left to right according to the text row, and each text row is spliced ​​with commas), vertical text and horizontal text, the BERT model is used to extract text semantic features, and three types of text semantic features can be obtained, namely: the second all text semantic features, the second vertical text semantic features, and the second horizontal text semantic features. Next, the average of the second all text semantic features, the second vertical text semantic features, and the second horizontal text semantic features is calculated, and the result is used as the second text feature.

[0064] Perform feature extraction on the second table image to obtain a second title feature.

[0065] In this embodiment, in combination with text prompts, the vertical text and horizontal text extracted from the text row position are input into the large language model to obtain column titles and row titles; the CLIP-based text encoder extracts the semantic features of the column titles and the row titles to obtain column title semantic features and row title semantic features; the average value of the column title semantic features and the row title semantic features is calculated to obtain the second title feature.

[0066] Specifically, the horizontal text and vertical text obtained in the above steps are used as the input of the large language model LLM, and the input is set as: "Find the row title in the text. The text content is like horizontal text. If there is a title, directly answer the title name without explanation. If there is no title, answer no without explanation." After inputting the text into the LLM, the answer (row title) is obtained; similarly, the column title can be obtained. Then, the semantic features of the column title and row title are extracted respectively through the CLIP text encoder to obtain the column title semantic feature and the row title semantic feature, and then the column title semantic feature and the row title semantic feature are averaged to obtain the second title feature.

[0067] S32, taking the second title feature as the K vector, taking the second visual feature as the V vector, taking the second text feature as the Q vector, and obtaining the second matching feature based on the cross-attention mechanism.

[0068] In this embodiment, the second title feature is used as the K vector, the second visual feature is used as the V vector, and the second text feature is used as the Q vector, and the second fused feature is obtained based on the cross-attention mechanism; the second fused feature is input into the multi-layer perceptron to obtain the second fused feature vector; the second fused feature vector is input into the sigmoid function to obtain the second matching feature.

[0069] Specifically, first, the second title feature is input as the K vector, the second visual feature is input as the V vector, and the second text feature is input as the Q vector into the MLP. The similarity between the second title feature and the second text feature is calculated through the attention layer, that is, the similarity between the second title feature and the second text feature is calculated using the dot product, cosine similarity or other similarity measurement methods; then, these similarity values ​​are normalized into attention weights through the softmax function. Then, the second visual feature is weighted according to these similarities to obtain the weighted feature, that is, the second fusion feature.

[0070] Furthermore, the second fused feature is used as input to the constructed MLP model. The MLP will perform forward propagation and transform and process the input features through the weights and activation functions between each layer. After calculation, the MLP model will output a feature vector. The output of the MLP is used as the input of the sigmoid function. The sigmoid function maps the input value to a range between 0 and 1 to obtain a probability or activation representation. The output of the sigmoid function is the second matching feature. This feature combines the original fused feature with the nonlinear transformation capabilities of the MLP and the mapping characteristics of the sigmoid function.

[0071] It should be noted that the entire process from the table image to extracting the matching features corresponding to the table image can be regarded as a feature extraction process, and the feature extraction method and the entire process used to obtain the matching features of the first table image and the second table image are completely consistent. Therefore, the feature extraction process of the first table image and the second table image is consistent.

[0072] S4, calculating the pairwise similarities between the first matching features and the second matching features, and determining the matching degree between the first table image and the second table image based on the pairwise similarities.

[0073] In this embodiment, it can be known from the above steps that the first matching feature is obtained by extracting features from the first table image; the second matching feature is obtained by extracting features from the second table image. Then, the cosine similarity between the first matching feature and the second matching feature is calculated; the matching degree between the first table image and the second table image is determined by the value of the cosine similarity; for example, the closer the cosine similarity value is to 1, the closer the match is; conversely, the greater the difference between the first table image and the second table image is.

[0074] Therefore, after completing all the above steps, the matching results of the two table images can be obtained by judging the matching degree of the two table images through the cosine similarity value finally calculated.

[0075] The table image matching method provided by the present application performs feature extraction from two different table images to be compared, obtains the text features, title features and visual features of each table image, and then inputs the text features, title features and visual features corresponding to the table images into the cross-attention mechanism to extract the matching features corresponding to different table images; then calculates the cosine similarity of the matching features of the two table images to be compared; then, determines the matching degree of the two table images based on the cosine similarity, which can effectively improve the matching precision and accuracy of different table images. At the same time, the matching method adopted by the present application can improve the robustness of different image matching technologies and further improve matching efficiency and security.

[0076] The protection scope of the table image matching method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0077] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 1 The table image matching method.

[0078] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0079] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or protruding structures in grooves on which instructions are stored, and any suitable combination of the above. The computer-readable storage media used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0080] The computer-readable program described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives a computer-readable program instruction from the network, and forwards the computer-readable program instruction for storage in the computer-readable storage medium in each computing / processing device. The computer program instruction for performing the operation of the present application can be an assembly instruction, an instruction set architecture (ISA) instruction, a machine instruction, a machine-related instruction, a microcode, a firmware instruction, a state setting data, an integrated circuit configuration data, or a source code or object code written in any combination of one or more programming languages, wherein the programming language includes an object-oriented programming language, such as Smalltalk, C++, etc., and a procedural programming language, such as "C" language or similar programming language. Computer readable program instructions can be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a computer, or completely on a computer or server. In the case of a computer, the computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of computer readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0081] The embodiment of the present application also provides a table image matching system, which can implement the table image matching method described in the present application. However, the implementation device of the table image matching method described in the present application includes but is not limited to the structure of the table image matching system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0082] The table image matching system provided in this embodiment will be described in detail below with reference to the diagrams.

[0083] This embodiment provides a table image matching system, including: See also Fig. 9 , which is a schematic diagram of the principle structure of the table image matching system described in this application in one embodiment. Fig. 9As shown, the table image matching system includes: an acquisition module 91, a first feature extraction module 92, a second feature extraction module 93, and an image matching module 94.

[0084] The acquisition module 91 is used to acquire a first table image and a second table image to be compared.

[0085] In this embodiment, the table image in this application specifically refers to an image that only contains a table (the table of course contains text). The table image can be obtained by reading from a local file, downloading from the Internet, or uploading by a user.

[0086] The first feature extraction module 92 is used to extract features based on the first table image to obtain a first matching feature.

[0087] Feature extraction is performed on the first table image to obtain a first title feature, a first visual feature and a first text feature.

[0088] In this embodiment, the first visual feature of the first table image is extracted based on the VIT model.

[0089] Specifically, first, the first table image is preprocessed, such as adjusting the size of the table image, scaling, denoising, binarization, and normalization operations, to ensure that the quality and format of the input data meet the requirements of the VIT model, which helps to improve the performance and stability of the model. Then, the preprocessed table image is divided into several small blocks, that is, each small block is a patch. Then, for each image patch, it is converted into an embedding vector in a high-dimensional space through linear projection (such as a fully connected layer) to capture the visual features of the patch. The self-attention mechanism in the Transformer model is then used to process these embedding vectors. By calculating the correlation between each patch, the model can learn the dependency between different patches and extract visual features that contribute to the final task. Finally, the feature vectors of all patches are aggregated to form a feature representation of a complete table image, which is the first visual feature of the first table image.

[0090] Similarly, feature extraction is performed on the first table image to obtain a first text feature.

[0091] In this embodiment, text recognition is performed on the first table image to obtain the text row position and the text row content; based on the text row position, all the text, vertical text, and horizontal text in the first table image are extracted; based on the BERT model, text semantic features of all the text, the vertical text, and the horizontal text are extracted to obtain the first semantic feature of all the text, the first semantic feature of the vertical text, and the first semantic feature of the horizontal text; the average value of the first semantic feature of all the text, the first semantic feature of the vertical text, and the first semantic feature of the horizontal text is calculated to obtain the first text feature.

[0092] Feature extraction is performed on the first table image to obtain a first title feature.

[0093] In this embodiment, in combination with text prompts, vertical text and horizontal text are input into a large language model to obtain column titles and row titles; a CLIP-based text encoder extracts semantic features of the column titles and row titles to obtain column title semantic features and row title semantic features; the average value of the column title semantic features and the row title semantic features is calculated to obtain a first title feature.

[0094] The first title feature is used as a K vector, the first visual feature is used as a V vector, and the first text feature is used as a Q vector, and a first matching feature is obtained based on a cross-attention mechanism.

[0095] In this embodiment, the first title feature is used as the K vector, the first visual feature is used as the V vector, and the first text feature is used as the Q vector, and a first fused feature is obtained based on a cross-attention mechanism; the first fused feature is input into a multi-layer perceptron to obtain a first fused feature vector; the first fused feature vector is input into a sigmoid function to obtain the first matching feature.

[0096] It should be noted that the above-mentioned entire implementation steps can be regarded as a whole, that is, the feature extraction process of the first table image, and this process can also be regarded as a large feature extraction module; thus, the same large feature extraction module is used to perform feature extraction on the second table image to obtain the second matching feature. Details are shown below.

[0097] The second feature extraction module 93 is used to extract features based on the second table image to obtain a second matching feature.

[0098] Feature extraction is performed on the second table image to obtain a second title feature, a second visual feature and a second text feature respectively.

[0099] In this embodiment, the second visual feature of the second table image is extracted based on the VIT model.

[0100] Similarly, feature extraction is performed on the second table image to obtain a second text feature.

[0101] In this embodiment, text recognition is performed on the second table image to obtain the text row position and the text row content; based on the text row position, all the text, vertical text, and horizontal text in the second table image are extracted; based on the BERT model, text semantic features of all the text, the vertical text, and the horizontal text are extracted to obtain the second all-text semantic feature, the second vertical text semantic feature, and the second horizontal text semantic feature; the average value of the second all-text semantic feature, the second vertical text semantic feature, and the second horizontal text semantic feature is calculated to obtain the second text feature.

[0102] Perform feature extraction on the second table image to obtain a second title feature.

[0103] In this embodiment, in combination with text prompts, vertical text and horizontal text are input into a large language model to obtain column titles and row titles; a CLIP-based text encoder extracts semantic features of the column titles and row titles to obtain column title semantic features and row title semantic features; the average value of the column title semantic features and the row title semantic features is calculated to obtain a second title feature.

[0104] Furthermore, the second title feature is used as a K vector, the second visual feature is used as a V vector, and the second text feature is used as a Q vector, and a second matching feature is obtained based on a cross-attention mechanism.

[0105] In this embodiment, the second title feature is used as the K vector, the second visual feature is used as the V vector, and the second text feature is used as the Q vector, and the second fused feature is obtained based on the cross-attention mechanism; the second fused feature is input into the multi-layer perceptron to obtain the second fused feature vector; the second fused feature vector is input into the sigmoid function to obtain the second matching feature.

[0106] The image matching module 94 is used to calculate the pairwise similarity between the first matching features and the second matching features, and determine the matching degree between the first table image and the second table image based on the pairwise similarity.

[0107] In this embodiment, it can be known from the above steps that the first matching feature is obtained by extracting features from the first table image; the second matching feature is obtained by extracting features from the second table image. Then, the cosine similarity between the first matching feature and the second matching feature is calculated; the matching degree between the first table image and the second table image is determined by the value of the cosine similarity; for example, the closer the cosine similarity value is to 1, the closer the match is; conversely, the greater the difference between the first table image and the second table image is.

[0108] Therefore, after completing all the above steps, the matching results of the two table images can be obtained by judging the matching degree of the two table images through the cosine similarity value finally calculated.

[0109] The table image matching model builds a table image matching system, which can effectively improve the matching accuracy of different table images.

[0110] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a certain chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a certain processing element of the above system. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0111] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0112] See also Fig.10 , which is a schematic diagram of the principle structure of the table image matching device described in this application in one embodiment. Fig.10As shown, this embodiment provides a table image matching device, which includes: a processor 101 and a memory 102; the memory 102 is used to store computer programs; the processor 101 is connected to the memory 102, and is used to execute the computer program stored in the memory 102, so that the table image matching device performs the various steps of the table image matching method as described above.

[0113] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0114] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0115] In summary, the table image matching method, system, medium and device provided by the present application have the following beneficial effects: The table image matching method provided by the present application performs feature extraction from two different table images to be compared, obtains the text features, title features and visual features of each table image, and then inputs the text features, title features and visual features corresponding to the table images into the cross-attention mechanism to extract the matching features corresponding to different table images; then calculates the cosine similarity of the matching features of the two table images to be compared; then, determines the matching degree of the two table images based on the cosine similarity, which can effectively improve the matching precision and accuracy of different table images. At the same time, the matching method adopted by the present application can improve the robustness of different image matching technologies and further improve matching efficiency and security.

[0116] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A table image matching method, characterized in that: The following steps are involved: Obtain a first table image and a second table image to be compared; Extracting features based on the first table image to obtain a first matching feature; including: extracting features from the first table image to obtain a first title feature, a first visual feature, and a first text feature; using the first title feature as a K vector, the first visual feature as a V vector, and the first text feature as a Q vector, and obtaining the first matching feature based on a cross-attention mechanism; Extracting features based on the second table image to obtain a second matching feature; including: extracting features from the second table image to obtain a second title feature, a second visual feature, and a second text feature respectively; using the second title feature as a K vector, the second visual feature as a V vector, and the second text feature as a Q vector, and obtaining a second matching feature based on a cross-attention mechanism; The pairwise similarities between the first matching features and the second matching features are calculated, and the matching degree between the first table image and the second table image is determined based on the pairwise similarities.

2. The table image matching method according to claim 1, characterized in that: The first visual feature of the first table image is extracted based on the VIT model.

3. The table image matching method according to claim 1, characterized in that: Performing feature extraction on the first table image to obtain a first text feature includes the following steps: Perform text recognition on the first table image to obtain the text row position and text row content; Extract all the text, vertical text, and horizontal text in the first table image according to the text line position; Extracting text semantic features of all the characters, the vertical characters, and the horizontal characters based on the BERT model to obtain a first semantic feature of all the characters, a first semantic feature of the vertical characters, and a first semantic feature of the horizontal characters; An average value of the first all-character semantic feature, the first vertical character semantic feature, and the first horizontal character semantic feature is calculated to obtain a first text feature.

4. The table image matching method according to claim 3, characterized in that: Extracting features from the first table image to obtain a first title feature includes the following steps: Combined with the text prompt, the vertical text and the horizontal text in the text line position are input into the large language model to obtain the column title and the row title; A text encoder based on CLIP extracts semantic features of the column header and the row header to obtain column header semantic features and row header semantic features; An average value of the column title semantic feature and the row title semantic feature is calculated to obtain a first title feature.

5. The table image matching method according to claim 1, characterized in that: Taking the first title feature as a K vector, taking the first visual feature as a V vector, taking the first text feature as a Q vector, and obtaining a first matching feature based on a cross attention mechanism includes the following steps: Taking the first title feature as a K vector, taking the first visual feature as a V vector, taking the first text feature as a Q vector, and obtaining a first fusion feature based on a cross attention mechanism; Inputting the first fused feature into a multi-layer perceptron to obtain a first fused feature vector; The first fused feature vector is input into a sigmoid function to obtain the first matching feature.

6. A table image matching system, characterized in that: include: An acquisition module, a first feature extraction module, a second feature extraction module and a picture matching module; The acquisition module is used to acquire the first table image and the second table image to be compared; The first feature extraction module is used to extract features based on the first table image to obtain a first matching feature; including: extracting features from the first table image to obtain a first title feature, a first visual feature, and a first text feature; using the first title feature as a K vector, the first visual feature as a V vector, and the first text feature as a Q vector, and obtaining the first matching feature based on a cross attention mechanism; The second feature extraction module is used to extract features based on the second table image to obtain a second matching feature; including: extracting features from the second table image to obtain a second title feature, a second visual feature, and a second text feature respectively; using the second title feature as a K vector, the second visual feature as a V vector, and the second text feature as a Q vector, and obtaining the second matching feature based on a cross attention mechanism; The image matching module is used to calculate the pairwise similarity between the first matching features and the second matching features, and determine the matching degree between the first table image and the second table image based on the pairwise similarity.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the table image matching method described in any one of claims 1 to 5 is implemented.

8. A table image matching device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory so that the table image matching device executes the table image matching method according to any one of claims 1 to 5.

Images