Multimodal table encoding for information retrieval systems

Abstract

Multi-modal table encoding, including receiving an electronic document containing a table. The table includes a plurality of rows, a plurality of columns, and a schema including column labels or row labels. The electronic document includes a description of the table located outside of the table. Next, separate machine learning encoders are operated to separately encode the description of the table, the schema, each of the rows, and each of the columns individually. The schema, rows, and columns are encoded with column end tokens and row end tokens marking the end of each column and each row, respectively. Then, a machine learning gating mechanism is applied to the encoded description, the encoded schema, the encoded rows, and the encoded columns to produce a fused encoding of the table, where the fused encoding represents both the structure of the table and the content of the table.

Description

Background Technology

[0001] This invention relates to the field of computerized information retrieval.

[0002] Information retrieval systems typically operate by receiving queries and searching for relevant information within a corpus of electronic data—which can be text, images, audio, video, or any other type of data. For example, web search engines receive user queries and search for relevant information on the World Wide Web. Technically, the search is conducted within a centralized index of the World Wide Web maintained by each search engine to facilitate near real-time delivery of search results. Similar indexing and searching exist in many organizational information retrieval systems used to search within an organization's internal collection of electronic documents and other data (corpora).

[0003] Electronic documents (which typically contain unstructured data in the form of free text) sometimes also include tables (which are in the form of structured data). Because tables are organized according to a specific pattern that provides column and / or row labels, they can be considered small structured databases. The data contained in tables is usually of high quality, and it is of great benefit to information retrieval systems that can index and search such tabular data.

[0004] Due to the large number of tables used in HTML (Hypertext Markup Language) on the World Wide Web, a field known as "Web form retrieval" has emerged. Many automated Web form retrieval techniques have been developed and proposed, which typically operate by receiving a search query in natural language and returning a sorted list of the most relevant Web forms found.

[0005] The examples and limitations of the prior art described above are intended to be illustrative rather than exclusive. Other limitations in the relevant art will become apparent to those skilled in the art upon reading the specification and studying the accompanying drawings. Summary of the Invention

[0006] The following embodiments and aspects thereof are described and illustrated in conjunction with systems, tools and methods, which are intended to be exemplary and illustrative, and not to limit the scope.

[0007] The embodiment relates to a method comprising automatically: (a) receiving an electronic document containing a table, wherein: the table includes a plurality of rows, a plurality of columns, and a pattern including column labels or row labels; and the electronic document includes a description of the table located outside the table; (b) operating a separate machine learning encoder to encode, separately and individually, the table description, the table pattern, each row of the table, and each column of the table, wherein: (i) the table pattern is encoded together with column end tokens or row end tokens that respectively mark the end of each of the column labels or row labels, and (ii) each row of the table is encoded together with each number marking the corresponding row. (iii) Each column of the table is encoded together with the row-end token marking the end of each data cell in the corresponding column, and together with the column-end token marking the end of the corresponding column; (c) a machine learning gating mechanism is applied to the encoded description, the encoded pattern, the encoded rows, and the encoded columns to produce a fusion code of the table, wherein the fusion code represents both the structure of the table and the content of the table; and (d) the fusion code of the table is stored in an index of a computerized information retrieval system.

[0008] Another embodiment relates to a system comprising: (i) at least one hardware processor; and (ii) a non-transitory computer-readable storage medium having program code embodied therein, executable by the at least one hardware processor to automatically: (a) receive an electronic document containing a table, wherein: the table includes a plurality of rows, a plurality of columns, and a pattern including column labels or row labels; and the electronic document includes a description of the table located outside the table; and (b) operate separate machine learning encoders to encode, separately and individually, the table description, the table pattern, each row of the table, and each column of the table, wherein: (i) the table pattern and the columns respectively label the ends of each of the column labels or row labels. (ii) Each row in the table is encoded together with a column end token marking the end of each data unit of the corresponding row, and is encoded together with a row end token marking the end of the corresponding row; and (iii) Each column in the table is encoded together with a row end token marking the end of each data unit of the corresponding column, and is encoded together with a column end token marking the end of the corresponding column; (c) Applying a machine learning gating mechanism to the encoded description, encoded pattern, encoded rows, and encoded columns to produce a fusion code of the table, wherein the fusion code represents both the structure of the table and the content of the table; and (d) storing the fusion code of the table in an index of a computerized information retrieval system.

[0009] Another embodiment relates to a computer program product comprising a non-transitory computer-readable storage medium having program code embodied therein, the program code being executable by at least one hardware processor to: (a) receive an electronic document containing a table, wherein: the table includes a plurality of rows, a plurality of columns, and a pattern including column labels or row labels; and the electronic document includes a description of the table located outside the table; (b) operate separate machine learning encoders to encode, separately and individually, the table description, the table pattern, each row of the table, and each column of the table, wherein: (i) the table pattern is associated with column-end tokens or row labels that respectively mark the ends of each of the column labels or row labels. (ii) Each row in the table is encoded together with the column end token marking the end of each data unit of the corresponding row, and with the row end token marking the end of the corresponding row; and (iii) Each column in the table is encoded together with the row end token marking the end of each data unit of the corresponding column, and with the column end token marking the end of the corresponding column; (c) applying a machine learning gating mechanism to the encoded description, the encoded pattern, the encoded rows, and the encoded columns to produce a fusion code of the table, wherein the fusion code represents both the structure of the table and the content of the table; and (d) storing the fusion code of the table in an index of a computerized information retrieval system.

[0010] In some embodiments, the standalone machine learning encoder includes a recurrent convolutional neural network (RCNN) that encodes the description of the table.

[0011] In some embodiments, a standalone machine learning encoder includes a multilayer perceptron (MLP) that encodes patterns in a table, or a transformer that encodes patterns in a table.

[0012] In some embodiments, the individual machine learning encoder includes: a first three-dimensional convolutional neural network (3D-CNN) that encodes rows of a table; and a second 3D-CNN that encodes columns of a table.

[0013] In some embodiments, the machine learning gating mechanism includes a gated multimodal unit (GMU).

[0014] In some embodiments, the method further includes, or program code is further executable, for automatically transforming the description, pattern, rows, and columns into token embeddings before encoding them.

[0015] In some embodiments, the method further includes, or program code is further executable to automatically: receive a search query; operate a machine learning encoder that encodes a description to encode the search query; and operate a computerized information retrieval system to search an index based on the encoded search query and return at least some of the tables as search results.

[0016] In some embodiments, the method further includes, or program code is further executable, for automatically transforming the search query into one or more token embeddings before encoding the search query.

[0017] In addition to the exemplary aspects and embodiments described above, other aspects and embodiments will become clear from reference to the accompanying drawings and from studying the following detailed description. Attached Figure Description

[0018] Exemplary embodiments are illustrated in the accompanying drawings. The dimensions of components and features shown in the figures are typically chosen for ease of presentation and clarity, and are not necessarily shown to scale. The drawings are listed below.

[0019] Figure 1 A block diagram of an exemplary system for multimodal table encoding is shown.

[0020] Figure 2 A process diagram for a method of multimodal table encoding is shown.

[0021] Figure 3 An example electronic document including a table is shown.

[0022] Figure 4 A block diagram of an exemplary information retrieval system is shown. Detailed Implementation

[0023] This paper discloses a technique for multimodal encoding of tables embodied in systems, methods, and computer program products, enabling the joint (or “fusion”) encoding of all modalities to be stored in the index of the information retrieval system and later used to respond to search queries. Advantageously, joint encoding allows the information retrieval (IR) system to selectively return only relevant portions of a table in response to a query, rather than directly returning the entire table. This is facilitated in particular by joint encoding of different information levels (“modalities”) of each table, such as its column and / or row headers, the data it contains, and even the descriptions surrounding it in the electronic document. Each such modality is advantageously encoded using a machine learning encoder specifically tailored for that modality.

[0024] In an alternative embodiment, the IR system returns the entire table or a sorted list of tables as the search results, but with an improved level of relevance.

[0025] In this technique, a separate machine learning encoder is first operated to perform separate unimodal encoding of different table modalities—its description (natural language text surrounding the table), patterns (column and / or row headers), rows, and columns. To enable the encoder to also understand the structure of the table, special markers can be added to the table boundaries—such as the ends of each row and column—before the unimodal encoding.

[0026] Recurrent Convolutional Neural Networks (RCNNs) can be used to encode the description of a table. Multilayer Perceptrons (MLPs) can be used to encode the pattern of a table. Three-dimensional Convolutional Neural Networks (3D-CNNs) can be used to encode each row and column of a table.

[0027] Next, machine learning gating mechanisms are applied to the encoded descriptions, patterns, rows, and columns to produce a fused encoding of the table. Advantageously, the fused encoding represents the structure of the table and its content (where “content” is not strictly contained within the table, but rather surrounds the table’s description—which characterizes its content).

[0028] Gated multimodal units (GMUs) can be used as a gating mechanism for machine learning.

[0029] The fused encoding of a table (also known as a fused "representation" of the table) can be stored in the index of a computerized information retrieval system. Then, in response to a search query, the information retrieval system can search the index and, if the fused encoding satisfies the query, selectively return at least a portion of the table in its search results. Advantageously, because the fused encoding represents both the structure and content of the table, the IR system can return results highly specific to fact queries. For example, Figure 3 The table lists the seven continents of the world, along with various data for each continent—its population, most populous city, and area. When a user performs a fact-oriented search query such as "the most populous city in continent D," "what is the population of continent C," or "what is the second largest continent," the IR system can leverage the table's fused encoding and return highly specific results (such as "fourth largest city," "4.5 billion," or "continent A") in a manner similar to extracting structured data centrally from a traditional database.

[0030] See now Figure 1 The diagram illustrates a block diagram of an exemplary system 100 for multimodal table encoding according to an embodiment. System 100 may also be referred to as an electronic document indexing system, a spreadsheet indexing system, etc. System 100 may include one or more hardware processors 102, random access memory (RAM) 104, and one or more non-transitory computer-readable storage devices 106.

[0031] Multiple storage devices 106 may have program instructions and / or components stored thereon, configured to operate multiple hardware processors 102. The program instructions may include one or more software modules, such as a multimodal table encoding module 108. The software components may include an operating system having various software components and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0032] System 100 can operate by loading instructions of the multimodal table encoding module 108 into RAM 104 when executed by processor(s) 102. The instructions of the multimodal table encoding module 108 can enable system 100 to receive an electronic document including table 110, process the electronic document, and output the fused encoding of table 112.

[0033] As described herein, System 100 is merely an exemplary embodiment of the invention and may actually be implemented in hardware only, software only, or a combination of both. System 100 may have more or fewer components and modules than shown, may combine two or more components, or may have different configurations or arrangements of components. System 100 may include any additional components that enable it to function as an operable computer system, such as a motherboard, data bus, power supply, network interface card, display, input devices (e.g., keyboard, pointing device, touch-sensitive display), etc. The components of System 100 may be co-located or distributed (e.g., in a distributed computing architecture). Furthermore, the components of System 100 may be co-located or distributed, or the system may operate as one or more cloud computing “instances,” “containers,” and / or “virtual machines,” as known in the art.

[0034] See now Figure 2 The process (method) diagram is used to discuss the instructions of the multimodal table encoding module 108. Figure 2 The process (method) diagram illustrates a method 200 for multimodal table encoding according to an embodiment.

[0035] The steps of method 200 can be performed in the order they are presented or in a different order (or even in parallel), as long as the order allows the necessary input for a certain step to be obtained from the output of an earlier step. Additionally, the steps of method 200 can be performed automatically (e.g., by...). Figure 1 (System 100), unless specifically stated that a step (or part thereof) involves manual action by the user.

[0036] In step 202, an electronic document (hereinafter referred to as the "document") containing at least one table may be received. This may be a Hypertext Markup Language (HTML) document, a Portable Document Format (PDF) document, a Rich Text Format (RTF) document, a word processing document (such as a Microsoft Word document, an open document), a LaTeX document, or any other digital file containing at least text and tables. Tables can be detected and parsed using conventional techniques. As an example only, tables can be detected based on the presence of "tables" in the document. ”、" ”、" "and" The `` tag is used to detect and parse tables in an HTML document.

[0037] A table can be viewed as comprising four modalities—description (d), schema (s), rows (r), and columns (c). See also Figure 3 This allows for a better understanding of these modalities. Figure 3 An exemplary document 300 including Table 302 is shown.

[0038] The description (d) modality includes the natural language text accompanying the table externally, excluding the text within the table itself. This modality typically provides general descriptive information about the table's content, which improves the reader's understanding of it and is therefore used for inclusion in fusion encoding. Such information may include one or more of the following, for example: first, the title of the document containing the table (e.g., such as...). Figure 3 The word "continent" in the document. Secondly, subheadings in the document (e.g., Figure 3 The "area and population" in the table, such as the chapter title of the chapter containing the table. Third, the explanatory text of the table, which can be placed immediately before or after the table (e.g., in the table). Figure 3 (See "Table 1: Area and Population" located below the table).

[0039] The modality of schema(s) imposes structure on the content of a table and includes M tags (i.e., (These are sometimes referred to as column names, headings, or titles.) Figure 3 In the text, the markers are "continent", "population", "most populous city", and "region (km)". 2 Each modal tag corresponds to a column in the table. Alternatively, the table schema can be hierarchical (not shown in the figures), where some modal tags contain several sub-tags, and subsequently several associated columns. In such a case, the schema can be assumed to be flat, and therefore, each tag corresponds to a single column. That is, each tag spanning multiple sub-tags is treated as if it were actually copied for use with those sub-tags. Tables 1 and 2 illustrate this:

[0040]

[0041] Table 1

[0042]

[0043] Table 2

[0044] Suppose a given table is arranged like Table 1, where “Label 1” spans two sublabels “Sublabel 1.1” and “Sublabel 1.2”, it can be thought of as if it were arranged like Table 2, where “Label 1” is copied to span “Sublabel 1.1” and “Sublabel 1.2” separately, such that each instance of “Label 1” corresponds to a single column.

[0045] To some extent, each modal label has a semantic function, indicating what type of data is being populated within the cells of the corresponding column. Labels are often very short in the text, can be abbreviated, and can sometimes be difficult for many readers to interpret. Furthermore, some table schemas can capture metadata-level information with complex semantic structures, such as hierarchy or functional dependencies, data constraints, etc. This requires encoding the table schema in a special way, different from how the text describing the table is encoded.

[0046] The row (r) modality of a table includes table rows (i.e., (This is sometimes referred to as a "record" or "line"). Furthermore, each line... A table typically consists of multiple data units, each corresponding to a column in the table. However, in some tables, a data unit may span multiple adjacent columns. Such tables can be handled similarly to how tags with sub-tags are handled above—viewing the table as flat, where each such data unit is copied for the columns it spans. From an IR perspective, table rows can represent entities or facts of interest that can be used to answer queries that require selecting specific information from the table.

[0047] The column (c) modality of the table includes M table columns (i.e., (Sometimes called "faces"), which vertically divide the data in a table. Each column Defined by a set of data units, each typically corresponding to a different row. Columns allow queries that express information needs about one or more (optionally aggregated) attributes of a set of rows, such as "population and area of ​​each continent", "average area of ​​a continent", etc.

[0048] Now return to the reference. Figure 2 In preprocessing step 204, each modality in the modalities can be tokenized and embedded. Tokenization can include a process, as routinely done in the field of Natural Language Processing (NLP), of separating the content of each modality into discrete words, which also involves removing any punctuation marks. For example, the text " The tokenization of "" can generate three tokens: "、" "and" For digital content, such as digital... Tokenization can involve simply removing thousands of delimiters, or any other symbols required for improved readability rather than for a true numerical representation. Tokenization can produce per-modality... of Token The sequence. Optionally, by including less than Any sequence of tokens applies padding (where) (represents the longest sequence of any modality), ensuring that the lengths of token sequences across different modalities remain consistent.

[0049] Next, token embedding can be performed using conventional NLP techniques. For example, the GloVe algorithm can be utilized (see J. Pennington et al., "GloVe: Global Vectors for Word Representation," Online at https: / / nlp.stanford.edu / pubs / glove.pdf, last viewed May 24, 2020). Embedding involves representing each token as a vector of numbers in a way that ensures tokens with similar contexts produce similar embeddings. For modalities... The embedded token is , where represents the mode The t-th token. For simplicity, whenever a modality is clear from its context, its embedding token is simply represented as .

[0050] Additionally, for non-descriptive (structured) table modalities (i.e., Two types of special tokens can be advantageously added: column end ( ) tokens and line ends ( Tokens, as follows:

[0051] For this pattern, you can add a label to the end of each column. Tokens, i.e., after (multiple) embedded tokens in each tag.

[0052] For rows, you can add after each data unit of the row (i.e., after the embedding token(s) of each such data unit). Tokens. Additionally, you can add a token at the end of each line. Token (from the last data cell of the row) (After the token).

[0053] For columns, you can add after each data cell of the column (i.e., after the embedding token(s) of each such data cell) Tokens. Additionally, you can add a token at the end of each column. Token (the last data element in the column) (After the token).

[0054] These token types, which mark the boundaries of the table (the ends of each row and column), enable the encoding of patterns, row and column modalities along with the table's structural definition. They signal this structure to the machine learning encoder, which then encodes these modalities.

[0055] and This is, of course, just an example of tokens that can be used to mark the boundaries of a table. Any other types of tokens for the boundaries of a table, which become embedded with the relevant modality, are explicitly described in this article.

[0056] To simplify the presentation, Figure 2 The illustration only shows the tokenization and embedding of a single row and a single column of the table, but step 204 of course generates a sequence of tokens for each of these modalities that exist in all rows / columns of the table.

[0057] When step 204 ends, it provides a token sequence for each of these modalities, including token embeddings and... and Token (if added).

[0058] In step 206, single-mode encoding of each of these modes can be performed individually, as follows:

[0059] Descriptive Modality: The first type of machine learning encoder can be operated to encode a sequence of tokens describing a modality. This type of machine learning encoder can be an encoder particularly well-suited for processing natural language input. For example, this could be R-CNN, which has shown high efficiency for text classification and similarity NLP tasks. As is known in the art, R-CNN is a bidirectional long short-term memory (bi-LSTM) followed by a max-pooling layer. Applying bi-LSTM, from left to right... and from right to left Process the input (token sequence mode d), for each token: and Two LSTM hidden states are obtained respectively. Then, the representation of each token is formed from its left-to-right context. Its initial embedding and its right-to-left context Composition. Then, using Activation is used to obtain the representation of the convolutional part that is fed back to RCNN:

[0060]

[0061] Have for each token The recursive representation is then used, and a max-pooling layer is applied to extract the main recursive features over the entire descriptive text, namely:

[0062]

[0063] Where max represents the element-wise maximum function. The pooling layer allows for a uniform vector representation across all tokens, capturing the most important underlying factors in the text modality.

[0064] Applying first-type machine learning encoders, such as RCNN, to generate latent representations for describing the modality. (That is, encoding).

[0065] Modal Patterns: A second type of machine learning encoder can be operated to encode token sequences of modal patterns. Such a machine learning encoder can be one particularly well-suited for processing and representing unordered sequences of tokens—each modality label can be considered an unordered sequence of tokens. For example, this could be a multilayer perceptron (MLP), for which a pattern can be modeled as an unordered set of token sequences (a sequence of tokens per modality label), where a particular... Tokens separate consecutive sequences of tags. To capture potential intra-tag and inter-tag relationships, two fully connected layers can be applied to each token embedding first, as follows:

[0066]

[0067] Then, all hidden states can be connected. The sequence. Finally, the latent representation (i.e., encoding) of the pattern. This can be achieved by first applying the third fully connected layer and then applying... To obtain it, i.e.: .

[0068] Another example of a suitable machine learning encoder is the transformer model. Transformers are designed to process ordered sequences of data, such as natural language. However, unlike recurrent networks, transformers do not need to process sequences sequentially. So, if the data in question (each label of the pattern) is natural language, the transformer does not need to process the beginning of the label before processing its end.

[0069] Row and Column Modalities: A third type of machine learning encoder can be operated independently to encode token sequences for both the row and column modalities. This type of machine learning encoder can be particularly well-suited for multidimensional inputs. For example, this could be a 3D-CNN. Each token in these modalities, along with its specific... and Token Dimensional embedding, where the token sequence for each modality is essentially a three-dimensional matrix: one dimension corresponds to the latent dimension projected by the embedding, the second dimension corresponds to the position of the token within the sequence, and the third dimension corresponds to the position of the token within the sequence. and The structure for token-based signaling.

[0070] Given a token sequence for each of these two modalities, the embedding of the token associated with each data cell in that row or column can be extracted by averaging the embeddings within that cell; that is, in the case of rows, for each adjacent cell... Tokens are arranged horizontally between each other, and in the case of columns, between each adjacent token. Vertically between them. Then, each potential feature can be viewed as having... Embedded features, column sum Entries of a line To extract the 3D matrix. By flattening the 3D representation of the input and multiplying it by a flattened 3D filter, the application of the 3D convolutional layer can be viewed as a regular 2D matrix multiplication. The position on the f-th feature map in the l-th layer. The value at is:

[0071]

[0072] in It is along the three-dimensional kernel size, It is the first The f-th feature in the layer Value, and It is the bias of the l-th layer on the f-th feature map.

[0073] In this exemplary 3D-CNN architecture, the application has Three convolutional layers, activated and max-pooled (as defined above), are used to obtain the row... and column The latent representation (encoding) of a modality.

[0074] Next, in step 208, the latent representation of all modalities is performed, for example by applying machine learning gating mechanisms to these individually encoded modalities. Multimodal coding to generate tables The fusion encoding. "d"

[0075] For example, gating mechanisms may include GMUs (see J. Arevalo et al., "Gated Multimodal Units for Information Fusion," in the 5th International Conference on Learning Representations (ICLR), Toulon, France, April 2017). The GMU receives potential representations. As input (represented in this paper for simplicity), , , and Then, use its original representation. Independent encoding, calculating the intermediate representation for each modality. Then, control pairs are used for joint representation (encoding). Each contributing gate neuron To combine these intermediate representations. The formal definition of GMU is as follows:

[0076]

[0077] in It is the i-th mode and This indicates element-wise multiplication.

[0078] After step 208, the tables (in the specific example given here) can be stored in the index of the IR system. or ) fusion encoding.

[0079] It is worth noting that the steps of method 200 can be repeated for each document containing a table, which is expected to be included in the index on the IR system. For example, if the IR system is an internet search engine that indexes documents on the World Wide Web, the search engine can employ method 200 each time it detects a table in a document it is attempting to index; the search engine can use method 200 to index the table and then use the search engine's regular indexing techniques (which are not table-oriented) to index the rest of the document. Any type of organizational IR system that indexes and searches a specific corpus can operate based on the same principle.

[0080] See now Figure 4 It shows a block diagram of an exemplary IR system 400, which utilizes method 200 ( Figure 2 The fusion table encoding generated is used to enhance its query response. According to some embodiments, the operation of the IR system 400 discussed will be considered as a method for information retrieval and / or table retrieval. IR system 400 may include the above-mentioned information regarding system 100 (…). Figure 1 Some or all of the hardware and / or software components discussed. In an embodiment, system 100 ( Figure 1 ) and IR system 400 ( Figure 4 This can be achieved by a single system, which is called an IR system, indexing and search system, etc.

[0081] The IR system 400 includes an index 402 that stores fused table encodings of different tables included in a document, which is part of a corpus to which the system is assigned the tasks of indexing and searching.

[0082] In response to receiving a search query 404 at IR system 400, the system can communicate with Figure 2 The query is processed in a similar manner to the modality descriptions in steps 204 and 206. Because the query is similarly assumed to be written in natural language: the query can first be tokenized and embedded, and then unimodal encoded using a first-type machine learning encoder (the same machine learning encoder used to encode descriptive modalities). For example, this could be with Figure 2 Step 206 is the same as the RCNN encoder. The single-modal encoding of the query produces its latent representation. .

[0083] Next, the IR system 400 can base its query on the encoded query. The search is performed using the fused table encoding stored in its index, and search results 406 are returned. Due to the multimodal encoding of the table, and the special tokens used to mark the table's structure, the IR system 400 can selectively return only one or more relevant portions of the table (e.g., data cells), rather than the entire table (unless the query specifically requests all content). See, for example, [link to relevant documentation]. Figure 3 The table shown indicates that if the query is "What is the population of Australia?", IR system 400 can return the following: This is the data cell at the intersection of the "Australia" row and the "Population" label. Similarly, if the query is "What is the combined area of ​​all continents?", the IR system 400 can calculate the sum of the data cells in the rightmost column of the table and return "". In other words, information contained in a single or several related data cells in a related table can be returned as a search result.

[0084] Alternatively, the entire table or even a sorted list of the entire table can be returned in response to a search query. While the entire table may not provide a specific answer to the user's intent, but rather require him or her to read the table to find relevant information, this technique can still be beneficial because it retrieves the correct table or correctly sorts the list of tables in a way that is highly relevant to the query.

[0085] Experimental results

[0086] This multimodal table encoding technique ( Figure 2 ) and its associated IR system operation ( Figure 4 We tested various existing table retrieval techniques to evaluate the contribution of multimodal table encoding to table retrieval performance.

[0087] The first experiment was conducted on the Wikitables corpus (CS Bhagavatula et al., "Methods for Exploring and Mining Tables on Wikipedia," in Proceedings of IDEA'13, Chicago, IL, USA, 2013; and CS Bhagavatula et al., "TabEL: Entity Linking in Web Tables," in Proceedings of The 14th International Semantic Web Conference, 2015), which includes approximately 1.6 million tables extracted from Wikipedia and The Free Encyclopedia. For each table, the corpus also includes its Wikipedia article title, section title, and descriptive text.

[0088] The first experiment evaluated the performance of this technique in retrieving relevant table information based on a pool of 60 specialized queries proposed by S. Zhang et al., "Ad Hoc Table Retrieval using Semantic Similarity," in Proceedings of the Web Conference 2018, April 2018. This technique was used to index the Wikitables corpus and then perform 60 specialized queries on the index (referencing IR system 400, processing each query as described above).

[0089] Performance evaluation involved comparing the table information retrieved using this technique with underlying facts defined by Zhang (2018), who manually labeled each table in the corpus as irrelevant (level 0), relevant (level 1), or highly relevant (level 2) to each of the 60 queries. It is noteworthy that Zhang's (2018) work is limited to retrieving the entire table in response to a query and does not address the task of retrieving only a portion of the table data that directly answers the query (e.g., a single data unit). Therefore, in the first experiment, only the ability of this technique to retrieve relevant tables was evaluated.

[0090] The performance of this technology compared to seven existing table retrieval technologies is presented in Table 3. (Usage deadline) Precision (P), cutoff The normalized discontinuous gain (NDCG) and average accuracy (MAP) were used to evaluate the table retrieval quality. The "trec_eval" tool, available from the National Institute of Standards and Technology at https: / / trec.nist.gov / trec_eval (last viewed May 25, 2020), was used to calculate various evaluation measures for this technique, and publicly available TREC files for seven other techniques (via the "trec_eval" tool) were downloaded and used to populate the corresponding rows in the table below.

[0091]

[0092] Table 3

[0093] "Single-field" is a technique proposed by M. Cafarella et al., "WebTables: Exploring the Power of Tables on the Web," VLDB'08, Auckland, New Zealand, 2008, which represents all table attributes within a single-field text document. Documents are scored using BM25 similarity.

[0094] "Multi-field" is a technique proposed by R. Pimpikar et al., "Answering Table Queries on the Web using Column Keywords," in Proceedings of the VLDB Endowment (PVLDB), Vol. 5, No. 10, pp. 908-919 (2012). It represents tables as multi-field documents, keeping each table attribute within a different field and using BM25 similarity scoring.

[0095] "NRM-F" is a technique proposed by H. Zamani et al., "Neural Ranking Models with MultipleDocument Fields," in Proceedings of WSDM'18, Los Angeles, CA, USA (2018), which serves as an important multi-field document retrieval alternative for this task. For this purpose, each table attribute is again treated as a text field. Since the code for such a method is not publicly available, it is implemented by closely following its published paper.

[0096] "LTR" is a technique proposed by Zhang (2018) that implements a point-to-rank learning (LTR) method trained using different queries, tables, and query table features (a total of 18).

[0097] "STR" is another technique proposed by Zhang (2018), which extends LTR with 16 additional semantic table similarity features: concept packs, entity packs, word and graph embeddings (computed using various early and late fusion strategies).

[0098] "T2VW" and "T2VE" are techniques proposed by L. Deng et al., "Table2Vec: Neural Word and EntityEmbeddings for Table Population and Retrieval," in Proceedings of SIGIR 2019, July 2019. They expand the LTR feature set with Table2Vec (T2V) embeddings learned from words (in T2VW) or entities (in T2VE) that appear in the table.

[0099] As shown in Table 3, this technique performs significantly better than the table retrieval techniques of these seven prior art techniques.

[0100] A second experiment was conducted on a corpus specifically generated for this experiment, which included approximately [number missing] corpora extracted from Wikipedia and The Free Encyclopedia. The corpus includes several tables. For each table, the corpus also includes its Wikipedia article title, section title, and descriptive text—as in the Wikitables tables. These tables are accompanied by a collection of 610 natural language queries, each with a single table as the correct answer. For each query, these tables are labeled as either irrelevant (level 0) or relevant (level 1).

[0101] The performance of this technique relative to three of the seven existing table retrieval techniques is presented in Table 4. The remaining four retrieval techniques were not evaluated because their full feature sets are not publicly available, but it is estimated that they would also outperform this technique when applied to the specific corpus of the second experiment.

[0102]

[0103] Table 4

[0104] As shown in Table 4, this technique also outperforms the three existing table retrieval techniques when tested on the special corpus of the second experiment.

[0105] The present invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0106] A computer-readable storage medium can be a tangible device capable of residing and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires. Rather, a computer-readable storage medium is a non-transient (i.e., non-volatile) medium.

[0107] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0108] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to perform aspects of the invention.

[0109] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function(s). It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0113] The description of a numerical range should be considered to have specifically disclosed all possible subranges and individual numerical values ​​within that range. For example, the description of a range from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6, etc., and individual numbers within that range, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the width of the range.

[0114] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for table encoding, comprising, Automatically: Receive an electronic document containing a table, wherein: The table includes: multiple rows, multiple columns, and a pattern including column labels or row labels, and The electronic document includes a description of the table located outside the table itself; A separate machine learning encoder is operated to encode the description of the table, the pattern of the table, each row of the table, and each column of the table, respectively and individually. The machine learning encoder includes a neural network for encoding the description of the table, wherein: The pattern of the table is encoded together with column end tokens or row end tokens that respectively mark the end of each of the column labels or row labels. Each row in the table is encoded together with a column-end token marking the end of each data unit in the corresponding row and a row-end token marking the end of the corresponding row. Each column of the table is encoded together with a row-end token marking the end of each data cell in the corresponding column and a column-end token marking the end of the corresponding column; A machine learning gating mechanism is applied to the encoded description, the encoded pattern, the encoded rows, and the encoded columns to generate a fused encoding of the table, wherein the fused encoding represents both the structure and the content of the table; and The fusion encoding of the table is stored in the index of the computerized information retrieval system.

2. The method of claim 1, wherein the separate machine learning encoder comprises a multilayer perceptron (MLP) encoding the pattern of the table, or a transformer encoding the pattern of the table.

3. The method of claim 1, wherein the separate machine learning encoder comprises: A first three-dimensional convolutional neural network (3D-CNN) encodes the rows of the table; as well as A second 3D-CNN encodes the columns of the table.

4. The method according to claim 1, wherein the machine learning gating mechanism includes a gated multimodal unit (GMU).

5. The method of claim 1, further comprising, prior to the encoding of the description, the pattern, the rows, and the columns: The description, the pattern, the rows, and the columns are automatically transformed into token embeddings.

6. The method of claim 1, further comprising, automatically: Receive search queries; The machine learning encoder, which encodes the description, is operated to encode the search query; as well as Operate the computerized information retrieval system to search the index based on the encoded search query and return at least some portions of the table as search results.

7. The method of claim 6, further comprising, before encoding the search query: The search query is automatically transformed into one or more token embeddings.

8. A system for table encoding, comprising: At least one hardware processor; as well as A non-transitory computer-readable storage medium having program code embodied therein, the program code being executable by the at least one hardware processor to automatically: Receive an electronic document containing a table, wherein: The table includes: multiple rows, multiple columns, and a pattern including column labels or row labels, and The electronic document includes a description of the table located outside the table itself; A separate machine learning encoder is operated to encode the description of the table, the pattern of the table, each row of the table, and each column of the table, respectively and individually. The machine learning encoder includes a neural network for encoding the description of the table, wherein: The pattern of the table is encoded together with column end tokens or row end tokens that respectively mark the end of each of the column labels or row labels. Each row in the table is encoded together with a column-end token marking the end of each data unit in the corresponding row and a row-end token marking the end of the corresponding row. Each column of the table is encoded together with a row-end token marking the end of each data cell in the corresponding column and a column-end token marking the end of the corresponding column; A machine learning gating mechanism is applied to the encoded description, the encoded pattern, the encoded rows, and the encoded columns to generate a fused encoding of the table, wherein the fused encoding represents both the structure and the content of the table; and The fusion encoding of the table is stored in the index of the computerized information retrieval system.

9. The system of claim 8, wherein the separate machine learning encoder comprises a multilayer perceptron (MLP) encoding the pattern of the table, or a transformer encoding the pattern of the table.

10. The system of claim 8, wherein the separate machine learning encoder comprises: A first three-dimensional convolutional neural network (3D-CNN) encodes the rows of the table; as well as A second 3D-CNN encodes the columns of the table.

11. The system according to claim 8, wherein the machine learning gating mechanism includes a gated multimodal unit (GMU).

12. The system of claim 8, wherein the program code is further executable by the at least one hardware processor to, prior to the encoding of the description, the pattern, the rows, and the columns: The description, the pattern, the rows, and the columns are automatically transformed into token embeddings.

13. The system of claim 8, wherein the program code is further executable by the at least one hardware processor to automatically: Receive search queries; The machine learning encoder that encodes the description is used to encode the search query; and Operate the computerized information retrieval system to search the index based on the encoded search query and return at least some portions of the table as search results.

14. The system of claim 13, wherein the program code is further executable by the at least one hardware processor to, prior to encoding the search query: The search query is automatically transformed into one or more token embeddings.

15. A computer program product comprising a non-transitory computer-readable storage medium having program code embodied therein, the program code being executable by at least one hardware processor to: Receive an electronic document containing a table, wherein: The table includes: multiple rows, multiple columns, and a pattern including column labels or row labels, and The electronic document includes a description of the table located outside the table itself; A separate machine learning encoder is operated to encode the description of the table, the pattern of the table, each row of the table, and each column of the table, respectively and individually. The machine learning encoder includes a neural network for encoding the description of the table, wherein: The pattern of the table is encoded together with column end tokens or row end tokens that respectively mark the end of each of the column labels or row labels. Each row in the table is encoded together with a column-end token marking the end of each data unit in the corresponding row and a row-end token marking the end of the corresponding row. Each column of the table is encoded together with a row-end token marking the end of each data cell in the corresponding column and a column-end token marking the end of the corresponding column; A machine learning gating mechanism is applied to the encoded description, the encoded pattern, the encoded rows, and the encoded columns to generate a fused encoding of the table, wherein the fused encoding represents both the structure and the content of the table; and The fusion encoding of the table is stored in the index of the computerized information retrieval system.

16. The computer program product of claim 15, wherein the separate machine learning encoder comprises: A recurrent convolutional neural network (RCNN) that encodes the description of the table; A multilayer perceptron (MLP) that encodes the pattern of the table, or a converter that encodes the pattern of the table; A first three-dimensional convolutional neural network (3D-CNN) encodes the rows of the table; and A second 3D-CNN encodes the columns of the table.

17. The computer program product of claim 15, wherein the machine learning gating mechanism includes a gated multimodal unit (GMU).

18. The computer program product of claim 15, wherein the program code is further executable by the at least one hardware processor to automatically: Receive search queries; The machine learning encoder that encodes the description is used to encode the search query; and Operate the computerized information retrieval system to search the index based on the encoded search query and return at least some portions of the table as search results.

Images