Systems and methods for automatic data type detection

Abstract

The system and method utilize masked language models for data type detection, such as (but not limited to) predicting column headings. Two masked language models are pre-trained on example columnar text. One model predicts missing data at the entity level (e.g., masked entity names that may consist of whole words), and the other model predicts missing data at the character level (e.g., masked individual characters). A table with missing column headings is fed into both models, and the output is contextual word embeddings and contextual character embeddings. These results are merged and then fed into a neural network classifier to predict column names.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 413,077, filed October 4, 2023, which is incorporated herein by reference in its entirety. [Background technology]

[0002] In many real-world applications, data may be received without additional information that provides the necessary context or meaning for the data to be processed. For example, data may be received in a columnar table format, but without column headings. To make the data useful in this particular example, the column heading information must be obtained in some way. Typically, column headings are added by humans, who examine the data and derive context by looking at the content and format of the information in each column. However, the need for human intervention slows the process of utilizing the data and increases the costs associated with using the data.

[0003] Prior art includes attempts to automate the assignment of column headings based solely on the column's content. The 2019 Sherlock project solves this problem using neural network techniques. The system is trained on a large columnar data repository, and each matched column is characterized by a large set of features describing statistical properties, character distribution, word embeddings, and paragraph vectors for column values. An extension of the Sherlock project is the 2020 Sato model, a machine learning model also trained on a large data table corpus. Sato uses a top-aware single-column prediction module and a structured output prediction module. Sato extends the Sherlock project by incorporating the concept of table intent into the single-column prediction, then combines topic-aware results for all columns with the structured output prediction module.

[0004] While methods such as Sherlock and Sato have shown promising results in machine learning approaches for data type detection, improvements to the accuracy of these methods are desirable. Such improvements would be useful for processing input data that arrives without context, such as column type headers. In particular, Sato improves on Sherlock by using table context between columns, but this only works for known column header types. The inventors recognized that a process that can manage previously unknown column header types, for example, would be highly desirable and could significantly expand the applicability of such techniques.

[0005] The references mentioned in this background section are not admitted to be prior art with respect to the present invention. Summary of the Invention [Means for solving the problem]

[0006] The present invention is directed to systems and methods that utilize masked language models to provide data type detection, such as (but not limited to) column header prediction. A language model is a natural language processing (NLP) technique that is trained to understand input words / sentences and predict output words or sentences from large data sets. A masked language model is a type of language model that is trained on a data corpus by masking certain words in the input data and predicting the same sentences again. This helps to better learn the context of the words. In various embodiments, the present invention uses this technique in systems and methods for imputing missing column headers. In one embodiment, two masked language models are pre-trained on example column-type text. One model predicts missing data at the entity level (e.g., masked entity names that may consist of whole words), and the other model predicts missing data at the character level (e.g., individual characters that are masked). A table with missing column headers is fed into both models, and the output is contextual word embeddings and contextual character embeddings. These results are merged and then fed into a neural network classifier to predict the column names.

[0007] These and other features, objects, and advantages of the present invention will be better understood from the following detailed description of the preferred embodiments and the appended claims, taken in conjunction with the drawings described below. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a flow chart illustrating a method according to one embodiment of the present invention. [Figure 2] 1 is a table of exemplary input values ​​according to one embodiment of the present invention. [Figure 3] 1 is a pre-trained masked language model for an entity according to one embodiment of the present invention. [Figure 4]1 is a pre-trained masked language model for individual characters according to one embodiment of the present invention. [Figure 5] 10 is a table with column headings as an exemplary output according to one embodiment of the present invention. [Figure 6] 1 is a hardware schematic diagram of a computer system for implementing a method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Before describing the present invention in further detail, it is to be understood that the invention is not limited to the particular embodiments described, since the scope of the present invention is limited only by the claims, and that the terminology used in describing particular embodiments is for the purpose of describing those particular embodiments only, and is not intended to be limiting.

[0010] A method according to one embodiment of the present invention will now be described with reference to Figures 1-5 using the non-limiting example of columnar data, however, it will be appreciated that the present invention can also be used with other types of data where data type information is missing.

[0011] Table 20 represents a set of input data in a columnar format. Each row represents a particular entity (such as a person) with various fields that describe that person. Each column contains the same fields across multiple entities. For example, a column may contain name, address, city, state, zip code, etc. However, in the example of Table 20, the actual column headings are unknown. In this example, no actual column headings are supplied with the data. Only the data itself is provided, so the system's task is to predict this data type information based on the data contained in the columns.

[0012] A masked language model is a type of language model that is trained on a data corpus by masking about 10-15 percent of the words in the input data and predicting them again. In the process of re-predicting the masked words, the model learns to recognize all the surrounding context that is important for predicting this word and encodes that context into its word embedding.

[0013] In one embodiment, the present invention utilizes an entity-level masked language model 12 and a character-level masked language model 14, as shown in FIG. 1. Model 12 is trained to generate embeddings by masking some entities in the data (which may consist of multiple words, as in the case of names) and re-predicting them similarly to a language model. Model 14 uses the same technique but is trained at the character level. Here, entire entities are masked, and then the entities are re-predicted character-by-character rather than as complete entities. Embeddings generated through this model have character-level context. The result of entity-level model 12 is contextual entity embedding 16, and the result of character-level model 14 is contextual character embedding 18.

[0014] Entity masking is an improvement over traditional masked language models. Entity-level masking is well-suited for this purpose because the entities that are intended to be predicted are the ones being considered. The rationale behind character-level models14 is that tabular data often contains proper nouns that do not exist in the vocabulary, such as people's names or place names. Character context, in addition to word context, can also help determine, for example, whether a field value is a place name such as a city, a person's name, or another type of proper noun.

[0015] Figures 3 and 4 provide details about the operation of the entity-level model 12 and the character-level model 14. Training data is provided from open-source sources. The data is guaranteed to be diverse in terms of type, number of fields, and random field ordering, to name a few requirements. A certain amount of noise is introduced to ensure that the output is immune to various types of noise, such as misspellings or misplaced values, such as a person's name in an address field. The process begins with tokenization of the input, which is performed depending on the model type. Each tokenized input (e.g., a field in a table) begins with a [CLS] classification token and ends with a [SEP] delimiter token. The [MASK] token is used to represent the word (in the case of the entity-level model 12) or character (in the case of the character-level model 14) that is masked in a particular instance. In the example of Figure 3, the input sentence for the entity-level model 12 is "[MASK]seen in [MASK] city" and the input sentence for the character-level model 12 is "[MASK]<space>s e...ty", and the words or characters to be masked may be selected randomly, for example based on the total percentage of words / characters used in the model.

[0016] Once the tokens are prepared, they are presented to the transformer portion of the entity-level model 12 and character-level model 14. The transformer is a neural network architecture that improves upon recurrent neural networks (RNNs) in terms of long-term memory, computation time, and better context; therefore, the transformer facilitates a good language model. The transformer excels at self-attention, which allows the transformer to highlight regions that are important for providing context and ignore others. The transformer retains only what is important for prediction. Self-attention allows the model to draw from the state of any previous point in the sequence, not just the last point considered by the model, and the resulting attention weighting can determine how much attention to devote to each point.

[0017] The transformers may undergo unsupervised pre-training followed by supervised fine-tuning for a specific task. The output of the transformer portion of the entity-level model 12 is a predicted entity in word-level context embeddings 16, and the output of the character-level model 14 is a predicted entity formed character-by-character in character-level context embeddings 18. Context embeddings are vectors that represent the semantics of a word or character, preserving context rather than just definitional meaning. A masked language model leverages these transformers to predict masked words / tokens. The entity-level model 12 is such a masked language model. However, the architecture of the entity-level model 12 and the character-level model 14 is such that the model outputs multiple words or a complete entity, even though the masked token is a single token representing multiple words. Thus, it is similar to a sentence prediction model.

[0018] In the next step, the outputs of the word-level context embeddings 16 and the character-level context embeddings 18 are combined in a merge step 22. In this step, the word embeddings and character embeddings for each entity are concatenated. An average embedding set 24 is the output of this process. The average embedding set 24 is then sent to a neural network classifier 26. The average embedding set 24 is calculated as the average of the character-level embeddings and the word-level embeddings, and this average represents the string sent to the neural network classifier 26. The neural network classifier 26 matches this data to perform entity classification. The neural network must be trained, and therefore, before use in generation, the neural network classifier 26 is trained using different types of table data. The entity-level model 12 and the character-level model 14 are trained using a single data corpus, after which they output embeddings. To generate embeddings using two language models, different data sets are obtained and used to train the classifier 26. The neural network classifier 26 is trained independently of the language models. In that case, the output of the neural network classifier 26 is a predicted column name 28, which may be added to table 10 to facilitate future generation of the columnar data contained in table 20.

[0019] Training of the components described herein may be performed using, for example, PyTorch or TensorFlow deep learning frameworks. PyTorch is an open source machine learning framework by Meta AI based on the Torch library, and TensorFlow is an earlier open source package developed by Google Brain.

[0020] The methods described herein may be implemented by any combination of hardware and software in various embodiments. For example, in one embodiment, a method may be implemented by a computer system (e.g., a computer system such as that of FIG. 6) or a collection of computer systems, each of which includes one or more hardware processors that execute program instructions stored on computer-readable physical storage media coupled to the hardware processors. The program instructions may implement the functions described herein (e.g., the functions of various hardware servers and other components that implement the network-based cloud and non-cloud computing resources described herein). The various methods described herein that are illustrated in the figures represent example implementations. The order of any method may be changed, and various elements may be added, modified, or omitted.

[0021] 6 is a block diagram illustrating an exemplary computer hardware system according to various embodiments. Computer system 500 may implement the hardware portion of a cloud computing system or a non-cloud computing system, forming part of various implementations of the present invention. Computer system 500 may be any of various types of hardware devices, including, but not limited to, a general-purpose server, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a consumer device, an application server, a physical storage device, a telephone, a mobile phone, or generally any type of computing node, computing node, computing device, and / or hardware computing device.

[0022] Computer system 500 includes one or more hardware processors 601a, 601b...601n (each processor may include multiple processing cores, which may be single-threaded or multi-threaded), coupled to physical system memory 602 via an input / output (I / O) interface 604. Computer system 500 may further include a network interface 606 coupled to I / O interface 604. In various embodiments, computer system 500 may be a single-processor system including one hardware processor 601a, or a multiprocessor system including multiple hardware processors 601a, 601b...601n, as shown in FIG. 6. Processor 601a, etc. may be any suitable processor capable of executing computational instructions. For example, in various embodiments, processor 601a, etc. may be a general-purpose processor or an embedded processor implementing any of a variety of instruction set architectures. In a multiprocessor system, each of processors 601a, etc. may generally, but need not necessarily, implement the same instruction set. Computer system 500 also includes one or more hardware network communication devices (e.g., network interface 606) for communicating with other systems and / or components over a communications network, such as a local area network, a wide area network, or the Internet. For example, a client application executing on system 500 may use network interface 606 to communicate with a server application executing on a single hardware server or a cluster of hardware servers that implements one or more of the components of the systems described herein in a cloud computing environment or in a non-cloud computing environment, such as those implemented in various subsystems.In another example, an instance of a server application running on computer system 500 may use network interface 606 to communicate with other instances of the application that may be running on other computer systems.

[0023] In the illustrated embodiment, computer system 500 also includes one or more physical persistent storage devices 608 and / or one or more I / O devices 610. In various embodiments, persistent storage device 608 may correspond to a disk drive, a tape drive, a solid-state memory or drive, other mass storage device, or any other persistent storage device. Computer system 500 (or a distributed application or operating system running on computer system 500) may store instructions and / or data in persistent storage device 608 as desired and retrieve the stored instructions and / or data as needed. For example, in some embodiments, computer system 500 may implement one or more nodes of a control plane or control system, and persistent storage 608 may include solid-state drives (SSDs) attached to the server nodes. Multiple computer systems 500 may share the same persistent storage device 608 or may share a pool of persistent storage devices, where the devices in the pool represent the same or different storage technologies, including those described above.

[0024] The computer system 500 includes one or more physical system memories 602 capable of storing code / instructions 603 and data 605 accessible by the processor 601a or other processors. The system memory 602 may include multiple levels of memory and memory caches within the system, designed to swap information within the memory based on access speed, for example. Interleaving and swapping can extend to persistent storage devices 608 in virtual memory implementations where memory space is mapped to persistent storage devices 608. Technologies used to implement the system memory 602 may include, by way of example, static random-access memory (RAM), dynamic RAM, read-only memory (ROM), non-volatile memory, solid-state memory, or flash memory. Similar to persistent storage devices 608, multiple computer systems 500 may share the same system memory 602 or may share a pool of system memories 602. One or more system memories 602 may contain program instructions 603 executable by a processor 601a or the like to implement the routines described herein.

[0025] In various embodiments, program instructions 603 may be coded in binary, assembly language, any interpreted language such as Java, a compiled language such as C / C++, or any combination thereof, although the particular languages ​​shown herein are merely examples. In some embodiments, program instructions 603 may implement multiple separate client, server nodes, and / or other components.

[0026] In some implementations, program instructions 603 may include executable instructions to implement an operating system (not shown), which may be any of a variety of operating systems, such as UNIX, LINUX, Solaris, MacOS, or Microsoft Windows. Any or all of program instructions 603 may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having stored thereon instructions that can be used to program a computer system (or other electronic device) to perform a process according to various implementations. A non-transitory computer-readable storage medium may include any mechanism for storing information in a form (e.g., software or processing application) readable by a machine (e.g., a physical computer). Generally speaking, a non-transitory computer-accessible medium may include a computer-readable storage medium or memory medium, such as a magnetic or optical medium, e.g., a disk or DVD / CD-ROM, coupled to or in communication with computer system 500 via I / O interface 604. Non-transitory computer-readable storage media may also include any volatile or non-volatile media, such as RAM or ROM, which may be included as system memory 602 or another type of memory in some embodiments of computer system 500. In other implementations, program instructions may be communicated using optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) conveyed over a communications medium, such as a network and / or a wired or wireless link, as may be implemented via network interface 606. Network interface 606 may be used to interface with other devices 612, which may include other computer systems or any type of external electronic device.

[0027] In some embodiments, system memory 602 may include a data store 605 as described herein. In general, system memory 602 and persistent storage 608 may be accessible on other devices 602 over a network and may store data blocks, replicas of data blocks, metadata related to the data blocks and / or the state of the data blocks, database configuration information, and / or any other information usable in performing the routines described herein.

[0028] In one embodiment, I / O interface 604 may coordinate I / O traffic between, for example, processor 601a, system memory 602, and any peripheral devices in the system, including those via network interface 606 or other peripheral interfaces. In some embodiments, I / O interface 604 may perform any necessary protocol conversion, timing conversion, or other data conversion to convert data signals from one component (e.g., system memory 602) into a format suitable for use by another component (e.g., processor 601a). In some embodiments, I / O interface 604 may include support for devices attached via various types of peripheral buses, such as, for example, variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. Also, in some embodiments, some or all of the functionality of I / O interface 604, such as the interface to system memory 602, may be incorporated directly into processor 601a.

[0029] The network interface 606 may enable data exchange between the computer system 500 and other devices attached to the network, such as, for example, other computer systems (which may implement server nodes, primary nodes, read-only nodes of one or more storage systems, and / or clients of the database systems described herein). Additionally, the I / O interface 604 may enable communication between the computer system 500 and various I / O devices 610 and / or remote storage devices 608. In some embodiments, the input / output devices 610 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or retrieving data by one or more computer systems 500. These input / output devices 610 may connect directly to a particular computer system 500 or may generally connect to multiple computer systems 500 in a cloud computing environment, a grid computing environment, or other system, including multiple computer systems 500. Multiple input / output devices 610 may be present in communication with computer system 500 or may be distributed among various nodes of a distributed system that includes computer system 500. In some embodiments, similar input / output devices may be separate from computer system 500 and may interact with one or more nodes of a distributed system that includes computer system 500 via a wired or wireless connection, such as network interface 106. Network interface 106 may generally support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11, or another wireless networking standard). Network interface 106 may support communication over any suitable wired or wireless general-purpose data network, such as, for example, other types of Ethernet networks.Additionally, network interface 106 may support communication over a telecommunications / telecommunications network, such as an analog voice network or a digital fiber communications network, a storage area network, such as a Fibre Channel SAN, or any other suitable type of network and / or protocol. In various embodiments, computer system 500 may include more, fewer, or different components (e.g., a display, a video card, an audio card, peripheral devices, or an Ethernet interface) than those shown in FIG.

[0030] Any of the embodiments of the distributed system described herein, or any of the components of the embodiments, may be implemented as one or more network-based services in a cloud computing environment. For example, read-write and / or read-only nodes in a database hierarchy of a hardware database system may present database services and / or other types of physical data storage services using the distributed storage system described herein to clients as network-based services. In some embodiments, a network-based service may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interaction over a network. A web service may have an interface described in a machine-processable format. Other systems may interact with the network-based service in a manner specified by the network-based service's interface description. For example, a network-based service may define various operations that other systems can invoke and may define specific application programming interfaces (APIs) to which other systems may be expected to conform when requesting the various operations.

[0031] In various embodiments, a network-based service may be requested or invoked by using a message that includes parameters and / or data associated with the network-based service request. Such messages may be formatted according to a particular markup language, such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol. To perform a network-based service request, a client of the network-based service may assemble a message that includes the request and communicate the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the web service using an Internet-based application-layer transport protocol, such as Hypertext Transfer Protocol (HTTP).

[0032] In some embodiments, a network-based service may be implemented using Representational State Transfer (REST) ​​techniques rather than message-based techniques. For example, a network-based service implemented according to REST techniques may be invoked by parameters contained within an HTTP method such as PUT, GET, or DELETE.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. It will be apparent to those skilled in the art that many more modifications are possible without departing from the concept of the present invention.

[0034] All terms used herein should be interpreted as broadly as possible within the context. In particular, the terms "comprises" and "comprising" refer to elements, components, or steps in a non-exclusive manner and should be interpreted to indicate that the referenced element, component, or step may be present, utilized, or combined with other elements, components, or steps not expressly mentioned. When groupings are used herein, all individual elements of the group and all possible combinations and subcombinations of the group are intended to be included individually. When ranges are given herein, the ranges are intended to include all subranges and all individual points within the ranges. When terms such as "about," "approximately," and the like are used herein, they are intended to include amounts, measurements, etc. that do not deviate significantly from the explicitly stated amounts, measurements, etc., so as not to lose sight of the purpose of the device or process being described. All references cited herein are incorporated by reference to the extent they do not contradict the disclosure herein.

[0035] The present invention has been described with reference to certain preferred and alternative embodiments that are intended to be illustrative only and not limiting of the full scope of the invention as set forth in the appended claims.

Claims

1. 1. A method for automatic semantic data type detection, comprising: receiving a data set lacking data type information in a known data set format; applying an entity-masked language model to the data set to create a set of context embeddings for whole words; applying a character-wise masked language model to the data set to create a character-wise context embedding set; merging the whole-word context embedding set and the character-by-character context embedding data set to create an average embedding set; applying the mean embedding set to a neural network classifier to produce a set of predicted data type information; A method comprising:

2. 10. The method of claim 1, further comprising: training the entity-masked language model on a text dataset before applying the entity-masked language model to the dataset to create a whole-word context embedding set; and training the character-by-character masked language model on the text dataset before applying the character-by-character masked language model to the dataset to create a character-by-character context embedding set.

3. 3. The method of claim 2, further comprising training the neural network classifier on a second text data set prior to applying the average embedding set to the neural network classifier to produce the set of predicted data type information.

4. 4. The method of claim 3, wherein the entity-masked language model and the character-by-character masked language model each comprise a transformer configured to perform deep learning natural language processing on the data set.

5. 5. The method of claim 4, further comprising tokenizing the data set by whole words before processing the entity-masked language model, and tokenizing the data set by characters before processing the character-based masked language model.

6. 6. The method of claim 5, wherein the data set is columnar data and the data type information is a column name of the columnar data set.

7. a language model with entity masks; a character-by-character masked language model; a neural network classifier; one or more computer processors; The memory space where instructions are stored wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to: receiving an input data set in a data set format that lacks data type information; applying the entity-masked language model to the dataset to create a set of context embeddings of whole words; applying the character-wise masked language model to the data set to create a character-wise context embedding set; merging the whole-word context embedding set and the character-by-character context embedding data set to create an average embedding set; applying the mean embedding set to the neural network classifier to generate a set of predicted data type information; An automatic semantic data type detection system that performs

8. 8. The automatic semantic data type detection system of claim 7, wherein the instructions, when executed by the one or more computer processors, further cause the one or more computer processors to: train the entity-masked language model on a text dataset before applying the entity-masked language model to the dataset to create a whole-word context embedding set; and train the character-by-character masked language model on the text dataset before applying the character-by-character masked language model to the dataset to create a character-by-character context embedding set.

9. 10. The automatic semantic data type detection system of claim 8, wherein the instructions, when executed by the one or more computer processors, further cause the one or more computer processors to train the neural network classifier on a second text data set prior to applying the average embedding set to the neural network classifier to produce the set of predicted data type information.

10. 10. The automatic semantic data type detection system of claim 9, wherein the entity-masked language model and the character-by-character masked language model each comprise a transformer configured to perform deep learning natural language processing on the data set.

11. 11. The automatic semantic data type detection system of claim 10, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to tokenize the data set by whole words before processing the entity-masked language model, and tokenize the data set by characters before processing the character-by-character masked language model.

12. 12. The automatic semantic data type detection system of claim 11, wherein the data set is columnar data and the data type information is a column name of the columnar data set.

13. When executed, the computer receiving a data set lacking data type information in a known data set format; applying an entity-masked language model to the data set to create a set of context embeddings of whole words; applying a character-wise masked language model to the data set to create a character-wise context embedding set; merging the whole-word context embedding set and the character-by-character context embedding data set to create an average embedding set; applying the mean embedding set to a neural network classifier to generate a set of predicted data type information; A machine-readable non-transitory physical medium storing machine-readable instructions for causing a program to be executed.

14. 14. The machine-readable non-transitory physical medium of claim 13, further storing machine-readable instructions that, when executed, cause the computer to: train the entity-masked language model on a text dataset before applying the entity-masked language model to the dataset to create whole-word context embedding sets; and train the character-by-character masked language model on the text dataset before applying the character-by-character masked language model to the dataset to create character-by-character context embedding sets.

15. 15. The machine-readable non-transitory physical medium of claim 14, further storing machine-readable instructions that, when executed, cause the computer to train the neural network classifier on a second text data set before applying the average embedding set to the neural network classifier to produce the set of predicted data type information.

16. 16. The machine-readable non-transitory physical medium of claim 15, wherein the entity-masked language model and the character-by-character masked language model each comprise a transformer configured to perform deep learning natural language processing on the data set.

17. 17. The machine-readable non-transitory physical medium of claim 16, further storing machine-readable instructions that, when executed, cause the computer to tokenize the data set by whole words before processing the entity-masked language model, and tokenize the data set by characters before processing the character-based masked language model.

18. 20. The machine-readable non-transitory physical medium of claim 17, wherein the data set is columnar data and the data type information is a column name of the columnar data set.

Images