Database pattern matching driven by artificial intelligence

Abstract

A computer-implemented method for improved pattern matching of two databases is disclosed. The method may retrieve a schema of a source table from a first database and receive schema of a plurality of target tables from a second database, identify one or more matching tables among the plurality of target tables based on a comparison of the schema of the source table and the schema of the plurality of target tables using a large language model, first sample attribute data is obtained from the source table and second sample attribute data is obtained from the selected matching table, and one or more pairs of matching attributes between the source table and the selected matching table are identified based on a comparison of the first sample attribute data and the second sample attribute data using a large language model. Related systems and software for implementing the method are also disclosed.

Description

Technical Field

[0001] This document generally deals with computing systems. More specifically, this document deals with database schema matching. Background Technology

[0002] Pattern matching is essential for data management, especially in the era of big data, where vast amounts of information are stored in various databases. Pattern matching acts as a bridge, connecting different data sources and enabling seamless data integration. This process involves comparing the schemas of different databases to find correspondences or matches between their elements. While pattern matching is important, it presents several technical challenges. These include handling heterogeneous data types, addressing inconsistencies in data representation, and the complexity of managing large-scale databases. Furthermore, the dynamic nature of data, which can change over time, adds another layer of complexity to the pattern matching process. Therefore, there is room for improvement in developing more robust and adaptive pattern matching techniques that can effectively handle these challenges. Summary of the Invention

[0003] Embodiments of this disclosure relate to a computational system for improved pattern matching of two databases. The computational system includes: a memory; one or more hardware processors coupled to the memory; and one or more computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations including: retrieving patterns from a source table in a first database and retrieving patterns from a plurality of target tables in a second database at runtime; identifying one or more matching tables among the plurality of target tables at runtime based on comparisons of the patterns in the source tables and the patterns in the plurality of target tables using a large language model; obtaining first sample attribute data from the source tables and second sample attribute data from selected matching tables at runtime; and identifying one or more pairs of matching attributes between the source tables and the selected matching tables at runtime based on comparisons of the first sample attribute data and the second sample attribute data using a large language model. Attached Figure Description

[0004] Figure 1 This is an overall block diagram of an example computational system that uses artificial intelligence to improve pattern matching between two databases.

[0005] Figure 2 This is a flowchart illustrating a two-stage method for pattern matching.

[0006] Figure 3 This is an example architecture diagram of a large language model.

[0007] Figure 4 This is a flowchart illustrating an example overall approach for improved pattern matching between two databases in two phases.

[0008] Figure 5 It shows that it can be shown Figure 4 A flowchart illustrating an example process used in the two phases of the pattern matching method.

[0009] Figure 6 This is a flowchart illustrating an example method for implementing the first stage of pattern matching.

[0010] Figure 7 This is a flowchart illustrating an example method for implementing the second stage of pattern matching.

[0011] Figure 8 An example structure for the prompt template is described.

[0012] Figure 9 The schema information for the example source table is described.

[0013] Figure 10 The schema information of the example target table is described.

[0014] Figure 11 The schema information for another example target table is described.

[0015] Figure 12 The schema information for another example target table is described.

[0016] Figure 13 The schema information for another example target table is described.

[0017] Figure 14 The schema information for another example target table is described.

[0018] Figure 15 Depicting from Figure 9 source table and Figures 10-11 The example sample data extracted from the target table.

[0019] Figure 16 This is a block diagram of an example computing system in which the described embodiments can be implemented.

[0020] Figure 17 This is a block diagram of an example cloud computing environment that can be used in conjunction with the technologies described in this article. Detailed Implementation

[0021] Architecture Matching Overview

[0022] As the volume and complexity of data continue to grow, pattern matching is becoming increasingly important in database (DB) management, playing a key role in establishing correspondences between schemas of different databases.

[0023] Schema matching ensures effective alignment of schemas for efficient data integration between different databases, thereby facilitating data integration (e.g., enabling the combination of data from various sources), migration (e.g., helping data transition smoothly from one database to another), interoperability (e.g., ensuring effective communication between systems with different database structures), and master data management (e.g., providing a unified view of key data across multiple databases).

[0024] Despite its importance, pattern matching presents several technical challenges. Traditional pattern matching methods often rely solely on syntactic features, such as table and column names, which may overlook subtle semantic differences. Furthermore, exhaustive matching of all possible pattern elements can lead to significant computational costs, thus limiting scalability. In addition, the dynamic nature of data, which can change over time, adds another layer of complexity to the pattern matching process.

[0025] The technique described in this paper addresses the challenges of pattern matching by leveraging the capabilities of generative artificial intelligence (AI) through a two-stage pattern matching approach. As described more fully below, this method provides a comprehensive and adaptive solution that combines the advantages of preliminary filtering and advanced data modeling, thereby achieving accurate, efficient, and scalable pattern matching for diverse database applications.

[0026] Example computational system for improved pattern matching

[0027] Figure 1 An overall block diagram of an example computational system 100 for improved pattern matching for different databases is shown.

[0028] As an example, Figure 1 Two databases are illustrated that require pattern matching (e.g., for data integration, migration, or other purposes): source database 112 in source database system 110 and target database 122 in target database system 120. Source database 112 includes multiple database tables, also referred to as source tables 114. Similarly, target database 122 includes multiple database tables, also referred to as target tables 124. Each table may include multiple attributes or data fields (e.g., arranged in different columns of the table). Pattern matching in this context involves evaluating the similarity or alignment between each source table 114 and each target table 124, and further evaluating the similarity or alignment between source table attributes and target table attributes.

[0029] For example, considering the pattern matching between the source table EmployeeInfo (a table representing basic information about company employees) and the target table WorkerDetails (a table recording detailed information about workers), the following pattern information is available:

[0030]

[0031] Pattern matching can indicate that the source table EmployeeInfo and the target table WorkerDetails are similar or match, even though they have different table names and different sets of attributes. Pattern matching can also indicate that the two tables have the following matching attribute pairs, although they may have different attribute names and / or data types: EmpID and WorkerID, First_Name and Given_Name, Last_Name and Family_Name, and Department_ID and Dept_ID.

[0032] like Figure 1 As shown, source database system 110 may include metadata 115 associated with source database 112. Metadata 115 may include schema information 116 of source table 114, one or more dictionaries 117 of source table 114, and some statistics 118 of source table 114. Similarly, target database system 120 may include metadata 125 associated with target database 122. Metadata 125 may include schema information 126 of target table 124, one or more dictionaries 127 of target table 124, and some statistics 128 of target table 124. For simplicity, metadata 115 or 125 may also be referred to as the "schema" of the source or target table.

[0033] As described in this article, schema information (e.g., 116, 126) defines how data is organized within the corresponding database, such as table names, attribute or column names, attribute data types, relationships between related tables (e.g., primary keys, foreign keys, etc.), and attribute uniqueness status. Dictionaries (e.g., 117, 127) include plain text descriptions of tables, attributes, or other textual information, serving as a guide for understanding the data. Statistical information (e.g., 118, 128) provides quantitative descriptions of the data, such as counts, averages, medians, maximums, minimums, and distributions, which can be used for data analysis and / or query optimization.

[0034] As described herein, pattern matching can be performed by pattern matching engine 130 using generative AI model 150 (or "GenAI"). Pattern matching engine 130 may include generative AI access layer 148, which may be configured (e.g., via different application programming interfaces) to interface with various generative AI models, such as large language models (LLMs) described more fully below. In some examples, generative AI model 150 may be externally hosted (e.g., on a third-party platform) or deployed locally on computing system 100.

[0035] User 102 can interact with pattern matching engine 130 via user interface 146 (or "UI"). In some examples, via user interface 146, user 102 can input task context, which can be incorporated into prompts sent to generative AI model 150 for pattern matching, as further described below. In some examples, via user interface 146, user 102 can manually select which tables to continue pattern matching, for example, manually removing some target tables 124 from attribute-level matching, as further described below.

[0036] Through user interface 146, user 102 can configure 144 of pattern matching engine 130. In some examples, configuration 144 may include connection information for both source database system 110 and target database system 120, allowing pattern matching engine 130 to automatically retrieve relevant information (e.g., metadata 115, 125, etc.) about source table 114 and target table 124. In some examples, configuration 144 may include specific settings for generative AI access layer 148, such as connection information for generative AI model 150, and operational parameters of generative AI model 150 (e.g., maximum number of tokens, model temperature, frequency penalty, presence penalty, etc.). In some examples, user 102 may specify in configuration 144 one or more criteria and / or parameters for filtering metadata and / or sample data before sending metadata and / or sample data to generative AI model 150. In some examples, user 102 may specify a similarity metric in configuration 144, which measures the similarity between tables and / or attributes. In some examples, user 102 can set a threshold in configuration 144, which can be used to determine whether two tables or two attributes are similar based on a similarity measure. In some examples, user 102 can specify a sampling method in configuration 144, which can be used to determine how attribute data is selected for attribute-level matching, as described more fully below. Other types of configuration (e.g., output format, etc.) can also be set by user 102.

[0037] As described more fully below, pattern matching engine 130 can perform pattern matching between source database 112 and target database 122 in two phases. In the first phase (also known as table-level matching), for each source table 114 in source database 112, pattern matching engine 130 can be configured to identify one or more matching tables among a plurality of target tables 124 in target database 122. In the second phase (also known as attribute-level matching), for selected matching tables, pattern matching engine 130 can be configured to find matching attributes between the source table and the selected matching table.

[0038] like Figure 1As shown, the pattern matching engine 130 may include a pattern extractor 132, one or more filters 134, a prompt assembly agent 136, and a sampler 138.

[0039] Pattern extractor 132 can be used in the first stage (table level) of pattern matching, which involves the automatic and runtime retrieval of metadata (e.g., 115, 125) of the source and target tables, such as pattern information (e.g., 116, 126), textual descriptions from dictionaries (e.g., 117, 127), and statistical information (e.g., 118, 128).

[0040] Sampler 138 can be used in the second stage (attribute level) of pattern matching, where it can automatically sample attribute data from the source table and the selected target table (e.g., the matching table).

[0041] Filter 134 can be used to filter out sensitive or irrelevant data (e.g., in the first and second phases) to ensure the quality and security of the pattern matching process.

[0042] In the first and second phases, the prompt assembly agent 136 can be configured to dynamically generate corresponding prompts 142 at runtime by inserting extracted data (e.g., metadata or sampled attribute data) into the corresponding prompt template 140. As described herein, the prompt template is a predefined text structure with parameters or placeholders that can be replaced with different values, guiding the generative AI model to generate specific types of responses or content. The generated prompts 142 include detailed instructions (and may also include examples) for the generative AI model 150 to measure similarity between tables (for table-level matching) or between attributes (for attribute-level matching). The generated prompts 142 can then be sent to the generative AI model 150 (e.g., via the generative AI access layer 148) for pattern matching.

[0043] The response generated by the generative AI model 150 includes the results of pattern matching (in two stages), which can be passed to the pattern matching engine 130 via the generative AI access layer 148 and presented to the user 102 on the user interface 146. The results of pattern matching can then be used for various applications such as data integration, migration, interoperability, master data management, etc.

[0044] In practice, the systems illustrated herein (such as computing system 100) can vary in complexity, incorporating additional functionalities, more complex components, etc. For example, additional functionalities may exist within the pattern matching engine 130. Additional components may be included to implement security, redundancy, load balancing, reporting design, data logging, etc.

[0045] The described computing system can be networked via a wired or wireless network connection (including the Internet). Alternatively, the system can be connected via an intranet connection (e.g., in a corporate environment, a government environment, etc.).

[0046] Computing system 100 and any other system described herein can be implemented in conjunction with any of the hardware components described herein, such as the computing systems described below (e.g., processing units, memory, etc.). In any example herein, metadata, prompt templates, prompts, sample attribute data, etc., can be stored in one or more computer-readable storage media or computer-readable storage devices. The techniques described herein may be generic to the details of the operating system or hardware and can be applied to any kind of environment to take advantage of the described features.

[0047] Example of a two-stage pattern matching framework

[0048] Figure 2 This is a flowchart illustrating the two-stage pattern matching framework 200 disclosed herein.

[0049] Framework 200 begins in the first matching phase, or table-level matching phase 210. In this phase, for each source table in the source database, one or more matching tables 218 among multiple target tables in the target database can be identified based on a comparison of metadata 212 (e.g., schema information) of the source and target tables. This phase can significantly reduce the number of potential matches, ensuring that only the most promising candidate or target tables advance to the next phase. In the second matching phase, or attribute-level matching phase 220, matching attributes 228 between the source tables and the selected matching tables can be identified based on a comparison of sample data 222 (also referred to as “sample attribute data”) of the source and target tables. This second phase is dedicated to detailed analysis of attribute-level or column-level alignment. Potential relationships and patterns within the sample data are analyzed in this phase.

[0050] Note that in attribute-level matching phase 220, only matching tables 218 identified in table-level matching phase 210 are analyzed. In other words, target tables that fail to match the source table in the first phase are excluded from further attribute matching in the second phase. Typically, attribute-level matching involves content comparison (comparison of data within tables) and usually requires more resources for data collection and embedding generation. Therefore, by prioritizing the most relevant matches (e.g., matching tables) first, framework 200 reduces the computational overhead associated with exhaustively matching attributes across all tables. As a result, framework 200 can improve pattern matching by balancing accuracy and efficiency.

[0051] In the first matching phase or table-level matching phase 210, metadata 212 (e.g., metadata 115 and 125) of both the source and target database systems can be dynamically retrieved at runtime. Metadata 212 may include schema information of the source and target tables. In some examples, metadata 212 may also include textual descriptions of the source and target tables and their attributes. In some examples, metadata 212 may also include statistical information of the source and target tables.

[0052] In some examples, filtering process 214 can be applied to retrieved metadata 212 to remove any metadata deemed sensitive, invalid, and / or non-semantic. For example, sensitive information, such as personally identifiable information that may potentially violate regulations like the European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), can be detected and filtered out. This can be achieved using various techniques such as natural language processing, machine learning methods, or rule-based approaches. Furthermore, table or column names lacking meaningful or semantic context can be identified and removed (e.g., a column named "xyz" without any corresponding textual description, or attributes containing binary data). Additionally, invalid or irrelevant data can be detected and filtered out. For example, an attribute with an integer data type but containing alphabetic strings would be considered invalid and therefore filtered out.

[0053] Filtered metadata from the source and target tables can be used for similarity measurement 216 to identify matching tables 218. In some examples, this comparison can be performed by first generating vector embeddings from the filtered metadata and then calculating a similarity metric (e.g., cosine similarity) between the vector embeddings. Note that the filtering process, particularly the removal of non-semantic data, can enhance the quality of the vector embeddings and the subsequent similarity measurement. In some examples, matching tables can be selected from target tables whose corresponding similarity measurements are above a predefined threshold. In some examples, matching tables can be identified by sorting the target tables based on their similarity values ​​and selecting the top N target tables, where N is a predefined number.

[0054] In some examples, a generative AI model (e.g., generative AI model 150) can be used to perform similarity measurements 216. For example, filtered metadata of a source table and a target table can be forwarded to the generative AI model. The generative AI model can be prompted (using specific instructions from a prompt template) to measure the similarity between the source and target tables to identify matching tables based on a comparison of the filtered metadata of the source and target tables 218. The generative AI model can have an embedding layer or stack configured to perform vector embeddings of the input data, such as the filtered metadata, and prompts can instruct the generative AI model to compute similarity based on such vector embeddings.

[0055] In other examples, similarity measurement 216 can be performed without using a generative AI model. For instance, a vector embedding engine can be employed to generate vector embeddings of filtered metadata using various embedding algorithms (e.g., Word2Vec, GloVe, BERT, etc.). Once the vector embeddings are generated, a similarity calculator (such as a similarity calculator using cosine similarity) can be used to calculate the similarity between the vector embeddings of the source and target tables.

[0056] The second attribute-level matching phase 220 begins by obtaining sample data 222 from both the source table and the selected matching table. Sample data 222 can be obtained by applying a specific sampling method to select rows from the selected columns of both the source table and the selected matching table. In some examples, random sampling can be used, where each row has an equal probability of being selected. In some examples, unique value sampling can be applied, where rows with unique values ​​for a specific column are selected. In some examples, global sampling can be used, where a fixed percentage of rows are selected from the entire table. Other sampling methods may also be used.

[0057] Filtering process 224 can be applied to sample data 222 to remove any data deemed sensitive, invalid, or non-semantic, similar to filtering process 214 applied in the first stage. Another similarity measurement 226 can then be performed to identify pairs of matching attributes 228 by comparing filtered sample data from the source table with filtered sample data from the selected matching table. These matching attributes 228 (each consisting of an attribute from the source table and an attribute from the selected matching table) can be identified based on a similarity threshold or by sorting similarity scores and then selecting the top N pairs of attributes, similar to the process in the first stage.

[0058] In some examples, similarity measurement 226 can be performed using a generative AI model (which may be the same as or different from the generative AI model used in the first stage). Alternatively, similarity measurement 226 can be performed using an embedding engine and a subsequent similarity calculator, as described above.

[0059] Example Overview of LLM and Tips

[0060] Generative AI models, base models, and LLMs are interconnected concepts in the field of AI. Generative AI (a broad term) encompasses AI systems that generate content such as text, images, music, or code. Unlike discriminative AI models, which aim to make decisions or predictions based on features of input data, generative AI models focus on creating new data points. Base models are a subset of these generative AI models, serving as a starting point for developing more specialized models. LLMs, a specific type of generative AI, work with language and can understand and generate human-like text. In the context of generative AI, including LLMs, cues are used as inputs or instructions to inform the AI ​​of desired content, context, or tasks. This allows users to guide the AI ​​to generate customized responses, interpretations, or creative content based on the provided cues.

[0061] In any of the examples in this article, LLMs can take the form of AI models designed to understand and generate human language. Such models typically leverage deep learning techniques, such as transformer-based architectures, to handle languages ​​with a very large number (e.g., billions) of parameters. Examples include the Generative Pre-trained Transformer (GPT) developed by OpenAI, Bidirectional Encoder Representations from Transforms (BERT) developed by Google, Robustly Optimized BERT Pretraining Approach developed by Facebook AI, NVIDIA's Megatron-LM, and others. Pre-trained models are available from a variety of sources.

[0062] In any of the examples in this paper, cues can be provided to the LLM at runtime to generate responses. Cues in the LLM can be input instructions that guide the model's behavior. Cues can be textual prompts, questions, or statements provided by the user to elicit a desired response from the LLM. Cues can act as primers for the model's generation process. Cue sources can include user-generated queries, predefined templates, or system-generated suggestions. Technically, cues are tokenized and embedded into the model's input sequence as moderating signals for subsequent text generation. Techniques such as prefixes, temperature control, top-K sampling, and thought chains can be used to perform experiments with cue variations to manipulate the output. These cues, derived from different inputs and customized strategies, allow users to influence the content generated by the LLM by shaping the underlying context and guiding the neural network's language generation. For example, cues can include instructions and / or examples to encourage the LLM to provide results in a desired style and / or format.

[0063] Example architecture of LLM

[0064] Figure 3 An example architecture of the LLM 300 is shown. The LLM 300 can be... Figure 1 An example of generative AI model 150.

[0065] In the depicted example, LLM 300 uses an autoregressive model (such as implemented in OpenAI's GPT) to generate text content by predicting the next word in the sequence given previous words. LLM 300 can be trained to maximize the probability of each word in the training dataset given its context.

[0066] like Figure 3 As shown, the LLM 300 can have an encoder 320 and a decoder 340, and their combination can be referred to as a "transformer". The encoder 320 processes the input text, transforming it into a context-rich representation. The decoder 340 takes this representation and generates the text output.

[0067] For autoregressive text generation, the LLM 300 generates text sequentially, and for each word it generates, it relies on the previous words used for context. During training, the model is learning the generated target or output sequence presented to the decoder 340. However, the output is shifted one position to the right compared to the outputs generated by the decoder 340 so far. In other words, the model sees the context of the previous words and is responsible for predicting the next word. As a result, the LLM 300 can learn to generate text from left to right, which is generally how language is constructed.

[0068] The text input to encoder 320 can be preprocessed by input embedding unit 302. Specifically, input embedding unit 302 can lexicalize the text input into a sequence of lexical units, each representing a word or part of a word. Each lexical unit can then be mapped to a fixed-length vector called the input embedding, which provides a continuous representation capturing the meaning and context of the text input. Similarly, for training LLM 300, the target or output sequence presented to decoder 340 can be preprocessed by output embedding unit 322. Similar to input embedding unit 302, output embedding unit 322 can provide a continuous representation or output embedding for each lexical unit in the output sequence.

[0069] Typically, the vocabulary in an LLM 300 is fixed and derived from the training data. The vocabulary in an LLM 300 consists of the lexical units generated during the training process. Words not in the vocabulary are not output. These lexical units are concatenated to form sentences in the text output.

[0070] In some examples, positional encoding (e.g., 304 and 324) can be performed to provide the order information of the words generated by the input embedding unit 302 and the output embedding unit 322, respectively. Positional encoding is necessary because, unlike recurrent neural networks, the transformer processes all words in parallel and does not inherently capture the order of words. Without positional encoding, the model treats the sentence as a collection of words, thus losing the context provided by the order of words. Positional encoding can be performed by mapping each position / index in the sequence to a unique vector and then adding the unique vector to the corresponding vector in the input or output embedding. By adding positional encoding to the input embedding, the model can understand the relative positions of words in the sentence. Similarly, by adding positional encoding to the output encoding, the model can preserve the order of words when generating text output.

[0071] Each of the encoder 320 and decoder 340 may include multiple stacked or repeated layers (in Figure 3 (Nx represents the number of stacked layers in the encoder 320 and / or decoder 340). The number of stacked layers in the encoder 320 and / or decoder 340 can vary depending on the specific LLM architecture. Generally, a higher "N" usually means a deeper model, which can capture more complex patterns and dependencies in the data, but may require more computational resources for training and inference. In some examples, the number of stacked layers in the encoder 320 can be the same as the number of stacked layers in the decoder 340. In other examples, the LLM 300 can be configured so that the encoder 320 and decoder 340 can have different numbers of layers. For example, a deeper encoder (more layers) can be used to better capture the complexity of the input text, while a shallower decoder (fewer layers) can be used if the output generation task is less complex.

[0072] The encoder 320 and decoder 340 are related through shared embedding and attention mechanisms, which allows the decoder 340 to access the contextual information generated by the encoder 320, enabling the LLM 300 to generate coherent and context-accurate responses. In other words, the output of the encoder 320 can be used as the basis for the decoder network to construct the generated text.

[0073] Both encoder 320 and decoder 340 include multi-layered attention and feedforward neural networks. The attention neural network implements an "attention" mechanism by calculating the relevance or importance of different words or lemmas within the input sequence to a given word or lemma in the output sequence, allowing the model to focus on context-relevant information when generating text. In other words, the attention neural network exerts "attention" on certain parts of the sentence most relevant to the task of generating the text output. The feedforward neural network processes and transforms the information captured by the attention mechanism, applying non-linear transformations to the contextual embeddings of lemmas, enabling the model to learn complex relationships in the data and generate more context-accurate and expressive text.

[0074] exist Figure 3 In the depicted example, encoder 320 includes an in-attention or self-attention neural network 306 and a feedforward neural network 310, and decoder 340 includes a self-attention neural network 326 and a feedforward neural network 334. Self-attention neural networks 306 and 326 allow LLM 300 to weight the importance of different words or lexical units within the same input sequence (self-attention in encoder 320) and between the input and output sequences (self-attention in decoder 340), respectively.

[0075] Furthermore, decoder 340 includes an inter-attention or encoder-decoder attention neural network 330, which receives input from the output of encoder 320. The encoder-decoder attention neural network 330 allows decoder 340 to focus on relevant portions of the input sequence (output of encoder 320) when generating the output sequence. As described below, the output of encoder 320 is a continuous representation or embedding of the input sequence. By feeding the output of encoder 320 to encoder-decoder attention neural network 330, contextual information and relationships captured in the input sequence (by encoder 320) can be passed to decoder 340. This connection allows decoder 340 to access the entire input sequence, not just the final hidden state. Because decoder 340 can focus on all words in the input sequence, the input information can be aligned with the generation of the output to improve the contextual accuracy of the generated text output.

[0076] In some examples, one or more of the attention neural networks (e.g., 306, 326, 330) can be configured to implement a single-head attention mechanism, in which the model captures relationships between words in an input sequence by assigning attention weights to each word based on the relevance of each word to the target word. The term "single-head" indicates that there is only one set of attention weights or one mechanism for capturing relationships between words in an input sequence. In some examples, one or more of the attention neural networks (e.g., 306, 326, 330) can be configured to implement a multi-head attention mechanism, in which multiple sets of attention weights or "heads" capture different aspects of the input sequence in parallel. Each head learns different relationships and dependencies within the input sequence. These multiple attention heads can enhance the model's ability to focus on various features and patterns, enabling it to understand complex, multifaceted contexts, resulting in more accurate and context-sensitive text generation. The outputs from the multiple heads can be cascaded or linearly combined to produce the final attention output.

[0077] like Figure 3 As shown, both encoder 320 and decoder 340 can include one or more addition and normalization layers (e.g., layers 308 and 312 in encoder 320, and layers 328, 332, and 336 in decoder 340). Addition layers, also known as residual connections, add the output of another layer (e.g., an attention neural network or a feedforward network) to its input. After the addition operation, a normalization operation can be performed by a corresponding normalization layer, which normalizes the features (e.g., gives the features zero mean and unit variance), which can help stabilize the learning process and reduce training time.

[0078] The linear layer 342 at the output of the decoder 340 can transform the output embedding back into the original input space. Specifically, the output embedding generated by the decoder 340 is forwarded to the linear layer 342, which can transform the high-dimensional output embedding back into a space where each dimension corresponds to a word in the vocabulary of the LLM 300.

[0079] The output of the linear layer 342 can be fed into the softmax layer 344, which is configured to implement the softmax function, also known as softargmax or the normalized exponential function, a generalization of a logistic function that compresses values ​​to a given range. Specifically, the softmax layer 344 takes the output (also known as the logit) from the linear layer 342 and transforms it into probabilities. These probabilities sum to 1, and each probability corresponds to the likelihood that a particular word is the next word in the sequence. Typically, the word with the highest probability is selected as the next word in the generated text output.

[0080] Still referencing Figure 3 The following describes the general operation process by which the LLM 300 generates a response or text output in response to received prompt input.

[0081] First, for example, the input text is lexicalized into a sequence of lexical units by input embedding unit 302, where each lexical unit represents a word or a portion of a word. Each lexical unit is then mapped to a fixed-length vector or input embedding. Position encoding 304 is then added to the input embedding to preserve information about the order of words in the input text.

[0082] Next, the input embedding is processed by the self-attention neural network 306 of the encoder 320 to generate a set of hidden states. As described above, the multi-head attention mechanism can be used to focus on different parts of the input sequence. The output from the self-attention neural network 306 is added to its input (residual connection) and then normalized at the addition and normalization layer 308.

[0083] Then, the feedforward neural network 310 is applied independently to each word. The feedforward neural network 310 includes fully connected layers with non-linear activation functions, allowing the model to capture complex interactions between words. The output from the feedforward neural network 310 is added to its input (residual connection) and then normalized at the addition and normalization layer 312.

[0084] Decoder 340 uses the hidden state from encoder 320 and its own previous output sequence to generate the next word in an autoregressive manner, enabling sequential output generation by focusing on previously generated words. Specifically, the output of encoder 320 (the input embedding processed by encoder 320) is fed into encoder-decoder attention neural network 330 of decoder 340, which allows decoder 340 to focus on all words in the input sequence. As described above, encoder-decoder attention neural network 330 can implement a multi-head attention mechanism, for example, by computing a weighted sum of all encoded input vectors, where the most relevant vector is assigned the highest weight.

[0085] The previous output sequence of decoder 340 is first lexicalized by output embedding unit 322 to generate an output embedding for each lexical in the output sequence. Similarly, positional embedding 324 is added to the output embedding to preserve information about the order of words in the output sequence.

[0086] The output embedding is processed by the self-attention neural network 326 of the decoder 340 to generate a set of hidden states. The self-attention mechanism allows each word in the text output to pay attention to all words in the input sequence as well as all previous words in the output sequence. The output from the self-attention neural network 326 is added to its input (residual connection) and then normalized at the addition and normalization layer 328.

[0087] The encoder-decoder attention neural network 330 receives the output embedding processed by the self-attention neural network 326 and the addition and normalization layers 328. Additionally, the encoder-decoder attention neural network 330 also receives the output from the addition and normalization layers 312, which represents the input embedding processed by the encoder 320. By considering both the processed input embedding and the output embedding, the output of the encoder-decoder attention neural network 330 represents the output embedding considering both the input sequence and the previously generated output. As a result, the decoder 340 can generate an output sequence that is context-aligned with the input sequence.

[0088] The output from the encoder-decoder attention neural network 330 is added to a portion of its input (residual connection), namely the output from the addition and normalization layer 328, and then normalized at the addition and normalization layer 332. The normalized output from the addition and normalization layer 332 then passes through the feedforward neural network 334. The output of the feedforward neural network 334 is then added to its input (residual connection) and normalized at the addition and normalization layer 336.

[0089] The processed output embedding from decoder 340 passes through linear layer 342, which maps the high-dimensional output embedding back to the size of the vocabulary; that is, it transforms the output embedding into a space where each dimension corresponds to a word in the vocabulary. Softmax layer 344 then converts the output of linear layer 342 into probabilities, each probability corresponding to the likelihood that a particular word is the next word in the sequence. Finally, LLM 300 samples output lexical units from the probability distribution generated by softmax layer 344 (e.g., selecting the lexical unit with the highest probability) and adds that lexical unit to the sequence of lexical units generated for the text output.

[0090] The above steps are repeated for each new lexical unit until the generated sequence ends or the maximum length is reached. Additionally, if the encoder 320 and / or decoder 340 have multiple stacked layers, the steps performed by the encoder 320 and decoder 340 are repeated on each layer of the encoder 320 and decoder 340 to generate each new lexical unit.

[0091] Example general method for two-phase pattern matching

[0092] Figure 4 This is a flowchart illustrating an example general method 400 for two-stage pattern matching. Method 400 can be, for example, derived from... Figure 1 The computing system 100 executes.

[0093] At step 410, the method may retrieve the schema of a source table (e.g., schema information 116) from a first database (e.g., source database 112) and the schema of multiple target tables (e.g., schema information 126) from a second database (e.g., target database 122). Schema retrieval may be automatic and performed at runtime (e.g., by schema extractor 132).

[0094] At step 420, the method can automatically identify one or more matching tables among multiple target tables at runtime based on a comparison of the schema of the source table and the schema of multiple target tables.

[0095] In some examples, the identification of matching tables can be performed using a generative AI model (such as the generative AI model 150 or LLM 300 mentioned above). For example, the method can construct a first hint at runtime by inserting the schema of the source table and the schemas of multiple target tables into a first hint template (e.g., using hint assembly agent 136), and then use the first hint at runtime to prompt the generative AI model or LLM.

[0096] In some examples, identifying one or more matching tables may include generating a first vector embedding based on patterns from a source table, and generating a second vector embedding based on patterns from multiple target tables. In some examples, identifying one or more matching tables may also include measuring table similarity between the first and second vector embeddings, and identifying table similarities greater than a predefined threshold. In some examples, the method may rank one or more matching tables based on table similarity.

[0097] At step 430, the method can obtain first sample attribute data from the source table and second sample attribute data from the selected matching table at runtime. Data sampling can be performed, for example, by sample 138. Various sampling methods (e.g., specified by the user in configuration 144) can be used to obtain such sample attribute data.

[0098] Then, at step 440, the method can, at runtime, identify one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data.

[0099] In some examples, the identification of matching attributes can be performed using a generative AI model (such as the generative AI model 150 or LLM 300 described above). For example, the method can construct a second cue at runtime by inserting the first sample attribute data and the second sample attribute data into a second cue template (e.g., using cue assembly agent 136), and then use the second cue at runtime to cue the generative AI model or LLM.

[0100] In some examples, identifying one or more pairs of matching attributes may include generating a third vector embedding based on first sample attribute data and a fourth vector embedding based on second sample attribute data. The method may also measure the attribute similarity between the third and fourth vector embeddings. In some examples, identifying one or more pairs of matching attributes may also include identifying attribute similarities greater than a predefined threshold. In some examples, the method may rank one or more pairs of matching attributes based on attribute similarity.

[0101] Method 400 and any other methods described herein can be performed by computer-executable instructions (e.g., instructions that cause a computing system to perform the method) stored in one or more computer-readable media (e.g., storage devices or other tangible media) or stored in one or more computer-readable storage devices. Such methods can be performed in software, firmware, hardware, or a combination thereof. Such methods can be performed at least in part by a computing system (e.g., one or more computing devices).

[0102] The actions shown can be described from alternative perspectives while still implementing the techniques. For example, "sending" can also be described as "receiving" from a different perspective.

[0103] Example processing flow

[0104] Figure 5 It shows that it can be shown Figure 4 The example process 500 used in the two phases of the pattern matching method is shown in the flowchart.

[0105] Figure 5 The pattern matching engine 530 is described, which can be Figure 1An example embodiment of the pattern matching engine 130 is shown. As illustrated, the pattern matching engine 530 includes a data collector 532 (similar to pattern extractor 132 or sampler 138), one or more filters 534 (similar to filter 134), a prompt assembly agent 536 (similar to prompt assembly agent 136), and a configuration 544 (similar to configuration 144).

[0106] In the first phase (table-based matching), data collector 532 can automatically retrieve the schema of the source table and the schemas of multiple target tables at runtime. As mentioned above, the schema may include table name, attribute name, data type, primary-foreign key relationship, attribute uniqueness status, etc. The schema may also include textual descriptions of the tables and attributes and / or table statistics. In the second phase (attribute-based matching), data collector 532 can be configured to obtain sample attribute data from the source table and sample attribute data from the selected matching tables. The sampling method can be specified by user 502 in configuration 544.

[0107] Filter 534 can be applied to retrieved patterns (in the first stage) or sample attribute data (in the second stage) to remove any detected sensitive, invalid, or non-semantic data. In some examples, user 502 can specify the settings and / or criteria used by filter 534 in configuration 544.

[0108] In the first phase, the prompt assembly agent 536 can construct a first prompt at runtime by inserting the schema of the source table and the schemas of multiple target tables (both schemas can be filtered) into a first prompt template, and then send the first prompt to the generative AI model 550. In response, the generative AI model 550 can produce a matching result 560 that includes the identified matching tables (which can be sorted). In the second phase, the prompt assembly agent 536 can construct a second prompt at runtime by inserting sample attribute data obtained from the source table and the selected matching tables into a second prompt template. In some examples, the second prompt may also include the schemas of the source table and the selected matching tables. Alternatively, the generative AI model 550 can remember such information from the first prompt. The second prompt is then sent to the generative AI model 550. In response, the generative AI model 550 can produce a matching result 560 that includes matching attributes (which can be sorted).

[0109] In some examples, user 502 may provide additional task context 504, which can be inserted into the first and / or second prompt template. Such task context 504 may include domain-specific information that helps the generative AI model 550 better understand the specific requirements of the pattern matching task. This may include, for example, information about the specific industry or business sector to which the database belongs, the data types typically stored in such databases, or any known relationships or patterns between tables or attributes. This additional task context 504 can help the generative AI model 550 generate more accurate and relevant pattern matches, thereby enhancing the overall effectiveness of the pattern matching process.

[0110] In some cases, the two-stage approach described above can be simplified to a single stage. For example, in scenarios where both the source and target databases are small, and the source and target tables are also small, the first stage can be skipped. In such scenarios, it can be assumed that all target tables are candidate matching tables. The data collector 532 can insert both pattern data and sample attribute data (which can be filtered using filter 534) into the prompt template to construct a prompt. This prompt is then sent to the generative AI model 550 to directly generate a matching result 560, which may include the top-ranked matching tables and matching attributes.

[0111] Example of first-stage data processing

[0112] Figure 6 This is a flowchart depicting an example method 600 that implements the first stage of pattern matching.

[0113] At step 610, it can be done at runtime (e.g., via...) Figure 5 Data collector 532 or Figure 1 The schema extractor 132) automatically retrieves the schema of the source table from the source database and the schema of multiple target tables from the target database.

[0114] In some cases, retrieving the schema of a source table involves obtaining a textual description and attributes of the source table from a first dictionary associated with the source database at runtime. Retrieving the schema of multiple target tables involves obtaining textual descriptions and attributes of multiple target tables from a second dictionary associated with the target database at runtime.

[0115] In some examples, retrieving a source table schema includes obtaining statistics for the source table at runtime. Retrieving multiple target tables schemas includes obtaining statistics for all target tables at runtime.

[0116] At step 620, the retrieved pattern data can be filtered (e.g., by filter 534 or filter 134) to remove any detected sensitive, invalid, or non-semantic data from the pattern.

[0117] The method then proceeds to step 630, where a prompt can be constructed, for example, by inserting filtered schema data of the source and target tables into the first prompt template (e.g., by prompt assembly agent 536 or 136). Optionally, additional task context 602, including domain-specific information about the source and target databases, can be provided by the user and inserted into the first prompt template.

[0118] At step 640, the constructed hints can be sent to the generative AI model (e.g., generative AI model 550, 150, or LLM 300). Upon receiving the hints, the generative AI model can measure the similarity between the source table and each target table. For example, a first vector embedding can be created based on the pattern of the source table. Furthermore, for each of the multiple target tables, a corresponding second vector embedding can be generated. These vector embeddings are based on the pattern of their respective target tables; that is, each target table will have its own unique second vector embedding. In some examples, table similarity can be calculated as the cosine similarity between the first vector embedding and each second vector embedding.

[0119] Then, at step 650, a condition check can be performed to determine whether the calculated table similarity is higher than a predefined threshold.

[0120] Each target table with a correspondence similarity higher than a predefined threshold can be considered a matching target table (or simply a "matching table"), meaning that the target table and the source table are considered to have similar patterns. If one or more matching tables are identified, the method proceeds to step 670 or the second phase (see below). Figure 7 The method described in step 700 (further described) performs attribute-level matching. During this phase, a more detailed comparison of the attributes of the source table and the target table is performed to identify more precise matches. If no matching table is found, i.e., if no target table has a table similarity higher than a predefined threshold, the method terminates and returns at step 660.

[0121] Example of second-stage data processing

[0122] Figure 7 This is a flowchart depicting an example method 700 that implements the second stage of pattern matching.

[0123] At step 710, it can be done at runtime (e.g., via...) Figure 5 Data collector 532 or Figure 1 The sampler 138 automatically retrieves sample attribute data (or simply "sample data") from the source table and each matching target table. As mentioned above, various sampling methods (e.g., random sampling, unique value sampling, etc.) can be used to obtain sample data.

[0124] At step 720, the retrieved sample data can be filtered (e.g., by filter 534 or filter 134) to remove any detected sensitive, invalid, or non-semantic data.

[0125] The method then proceeds to step 730, where a second prompt is constructed, for example, by inserting filtered sample data from the source table and the matching target table into the second prompt template (e.g., by prompt assembly agent 536 or 136). Optionally, additional task context 702, including domain-specific information about the source and target databases, can be provided by the user and inserted into the second prompt template.

[0126] At step 740, the constructed second hint can be sent to the generative AI model. Upon receiving the second hint, the generative AI model can measure the similarity between attributes of the source table and each matching target table. For example, a third vector embedding can be created based on sample attribute data from the source table. Furthermore, for each matching target table, a corresponding fourth vector embedding can be generated based on its sample attribute data. Attribute similarity can be calculated as the cosine similarity between the third vector embedding and each fourth vector embedding.

[0127] Then, at step 750, matching attribute pairs between the source and target tables can be identified and presented. In some examples, a pair of matching attributes can be identified if the attribute similarity between the source and target tables exceeds a predefined threshold. In some examples, after identifying all matching attribute pairs, they can be sorted based on their attribute similarity. This sorting provides a prioritized list of matching attributes, allowing the user to focus first on the most relevant matches.

[0128] Prompt Template Example

[0129] As mentioned above, prompt templates can be used to construct prompts in both stages of pattern matching. Figure 8 An example structure for a prompt template 800 is depicted, which combines information or instructions from multiple categories for generative AI models. Placeholders in the prompt template 800 are indicated by pairs of curly braces. Figure 8 The template structure described can be used in the prompt templates for both the first and second stages of pattern matching.

[0130] For example, prompt template 800 has a task segment 810 including a description section 812 and a context section 814. The description section 812 may include details about the task, such as an introduction to the task, a definition of pattern matching, and the process for obtaining a matching pattern. The context section 814 may be optional; it is a placeholder and can be replaced with user-provided domain-specific or background information that ensures the relevance of the generative AI model's results.

[0131] The prompt template 800 also has a role segmentation 820, which can specify the role or function of the generative AI model in the pattern matching process.

[0132] Example segments 830 of the prompt template 800 include input example 832 and output example 834. Input example 832 provides a description of the expected format and structure of the input data, helping the generative model understand the nature and context of the data it will use. This can include examples of pattern information in the first stage of pattern matching or sample data in the second stage. Output example 834 illustrates the expected format and structure of the output data. They provide a blueprint for the generative AI model to follow when generating its own output, ensuring that the results are presented in an intuitive and useful way for the user. This can include presenting a matching table in descending order of similarity (in the first stage) or presenting sorted pairs of matching attributes (in the second stage).

[0133] The prompt template 800 also includes input segments 840, which are placeholders that can be replaced by patterns of dynamically retrieved source and target tables (in the first stage) or sample data (in the second stage).

[0134] The following is an example first hint template that can be used in the first stage of table-level matching:

[0135]

[0136]

[0137] The following is an example of a second prompt template that can be used in the second-stage attribute-level matching:

[0138]

[0139]

[0140]

[0141] Although specific prompt templates have been described above, it should be understood that these prompt templates are merely examples, and different prompt templates can be used based on the principles described in this article.

[0142] Example use cases

[0143] refer to Figures 9-15 Example use cases are described to further illustrate the two-stage pattern matching approach.

[0144] Figure 9 The example source table named "source_employee" is shown with dynamically extracted schema information 900. Figures 10-14The dynamically extracted pattern information (1000, 1100, 1200, 1300, and 1400) for five different target tables (i.e., “target_employee”, “target_employees”, “target_personnel”, “target_staff”, and “target_worker”) is shown respectively. Prompts can be generated by inserting the above pattern information into the input segment of the first prompt template.

[0145] Upon receiving this prompt, the generative AI model can generate output that identifies the following two matching tables (e.g., based on a predefined similarity threshold, etc.):

[0146]

[0147] Figure 15 The example data 1500, 1510, and 1520 are shown, obtained from the source table "source_employee" and the two matching tables "target_employee" and "target_employees," respectively. Another prompt can be generated by inserting such sample data into the input segment of the second prompt template described above.

[0148] Upon receiving this prompt, the generative AI model can generate output that identifies the following matching attributes:

[0149]

[0150] Example Advantages

[0151] The technology described in this article offers several technical advantages.

[0152] First, the two-stage pattern matching method described in this paper provides a comprehensive solution to the challenges posed by traditional pattern matching methods. By leveraging the capabilities of generative AI, it not only addresses the syntactic features of the schema but also captures subtle semantic differences that traditional methods often overlook, resulting in more accurate schema alignment.

[0153] Another technical advantage of the disclosed pattern matching method is its efficiency and scalability. The first stage of table-level matching filters out irrelevant or low-similarity matches, thereby reducing computational costs and resource consumption. This initial filtering allows for more focused and efficient attribute-level matching in the second stage, making the method scalable even for large databases with complex patterns.

[0154] The second stage of attribute-level matching further enhances the accuracy of the pattern matching process. By comparing data within the table and generating embeddings for the sample attribute data, this method can identify matching data columns with high precision.

[0155] Furthermore, the dynamic nature of the two-stage pattern matching method allows it to adapt to changes in data over time. This adaptability, combined with user-configurable options for data collection and comparison, makes the method universal and effective for a wide range of database applications, such as data integration, migration, interoperability, and master data management across different databases.

[0156] Example computing system

[0157] Figure 16 An example of a suitable computing system 1600 in which the described innovation can be implemented is depicted. The computing system 1600 is not intended to impose any limitation on the scope or functionality of this disclosure, as the innovation can be implemented in different computing systems.

[0158] refer to Figure 16 The computing system 1600 includes one or more processing units 1610, 1615 and memories 1620, 1625. Figure 16 In this document, the basic configuration 1630 is included within the dashed lines. Processing units 1610 and 1615 can execute computer-executable instructions, such as those for implementing the features described in the examples herein (e.g., methods 400, 600, and 700). Processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multiprocessor system, multiple processing units can execute computer-executable instructions to increase processing power. For example, Figure 16 A central processing unit 1610 and a graphics processing unit or coprocessor 1615 are shown. Physical memories 1620 and 1625 may be volatile memories (e.g., registers, caches, RAM), non-volatile memories (e.g., ROM, EEPROM, flash memory, etc.), or some combination thereof, accessible by the processing units 1610 and 1615. Memories 1620 and 1625 may be adapted to store, in the form of computer-executable instructions executed by the processing units 1610 and 1615, implementing one or more innovative software 1680 described herein.

[0159] The computing system 1600 may have additional features. For example, the computing system 1600 may include a storage device 1640, one or more input devices 1650, one or more output devices 1660, and one or more communication connections 1670, including input devices, output devices, and communication connections for user interaction. Interconnection mechanisms (not shown), such as buses, controllers, or networks, may interconnect the components of the computing system 1600. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system 1600 and coordinates the activities of the components of the computing system 1600.

[0160] The physical storage device 1640 may be removable or non-removable and includes a magnetic disk, magnetic tape or cassette tape, CD-ROM, DVD, or any other medium that can be used to store information in a non-transitory manner and can be accessed within the computing system 1600. The storage device 1640 may store instructions for implementing one or more innovative software described herein.

[0161] Input device 1650 may be an input device such as a keyboard, mouse, pen or trackball, a voice input device, a scanning device, a touch device (e.g., a touchpad, a display, etc.), or another device that provides input to computing system 1600. Output device 1660 may be a display, printer, speaker, CD burner, or another device that provides output from computing system 1600.

[0162] Communication connection 1670 enables communication with another computing entity via a communication medium. The communication medium can transmit information, such as computer-executable instructions, audio or video input or output, or other data in modulated data signals. A modulated data signal is a signal whose one or more characteristics are set or altered to encode information in the signal. By way of example and not limitation, the communication medium can be electrical, optical, RF, or other carriers.

[0163] Innovations can be described in the context of computer-executable instructions, such as those included in a program module that execute on a computing system on a target real or virtual processor (e.g., ultimately executing on one or more hardware processors). Typically, a program module or component may include routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. In various embodiments, the functionality of a program module can be combined or split among program modules as needed. The computer-executable instructions for a program module can execute within a local or distributed computing system.

[0164] For the sake of presentation, the computer operations in a computing system are described in detail using terms such as "determine" and "use". These terms are high-level descriptions of operations performed by a computer and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms vary depending on the implementation method.

[0165] Computer-readable media

[0166] Any computer-readable medium described herein may be non-transitory (e.g., volatile memory such as DRAM or SRAM, non-volatile memory such as magnetic memory, optical memory, etc.) and / or tangible. Any storage operation described herein may be implemented by storage in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Anything described as being stored (e.g., data created and used during implementation) may be stored in one or more computer-readable media (e.g., computer-readable storage media or other tangible media). Computer-readable media may be limited to implementations that do not consist of signals.

[0167] Any method described herein may be implemented by computer-executable instructions (e.g., stored thereon, encoded thereon, etc.) in one or more computer-readable media (e.g., computer-readable storage media or other tangible media) or one or more computer-readable storage devices (e.g., memory, magnetic storage, optical storage, etc.). Such instructions may cause a computing device to perform the method. The techniques described herein may be implemented in various programming languages.

[0168] Example cloud computing environment

[0169] Figure 17 An example cloud computing environment 1700, in which the described technologies can be implemented, is depicted, including, for example, system 100 and other systems described herein. Cloud computing environment 1700 may include cloud computing service 1710. Cloud computing service 1710 may include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. Cloud computing service 1710 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations (such as different data centers) and / or in different cities or countries).

[0170] Cloud computing service 1710 can be used by various types of computing devices (e.g., client computing devices), such as computing devices 1720, 1722, and 1724. For example, computing devices (e.g., 1720, 1722, and 1724) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablets or smartphones), or other types of computing devices. For example, computing devices (e.g., 1720, 1722, and 1724) can utilize cloud computing service 1710 to perform computing operations (e.g., data processing, data storage, etc.).

[0171] In practice, it can support cloud-based, on-premises, or hybrid deployment scenarios.

[0172] Example Implementation

[0173] In any of the examples in this article, a software application (or “application”) may take the form of a single application or a suite of applications, whether it is offered as a service (SaaS), in the cloud, on-premises, on desktop, on mobile devices, wearables, etc.

[0174] Although some of the operations of the disclosed methods are described in a specific order for ease of presentation, this description includes rearrangement unless the specific language used in this paper requires a particular order. For example, operations described sequentially may be rearranged or performed concurrently in some cases.

[0175] As stated in this application and the claims, the singular forms “a,” “an,” and “the” include the plural forms unless the context clearly indicates otherwise. Additionally, the term “include” means “comprise.” Furthermore, “and / or” means “and” or “or” as well as “and” and “or.”

[0176] Although specific prompt templates have been described above, it should be understood that these prompt templates are merely examples for illustrative purposes, and different prompt templates may be used based on the principles described herein.

[0177] In any of the examples described herein, an operation performed at runtime means that the operation can be completed in real time or with negligible processing delay (e.g., the operation can be completed in one second, etc.).

[0178] Example Terms

[0179] Any of the following example terms can be implemented.

[0180] Clause 1. A computational system for improved pattern matching of two databases, the computational system comprising: a memory; one or more hardware processors coupled to the memory; and one or more computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations including: retrieving, at runtime, patterns of a source table from a first database and patterns of a plurality of target tables from a second database; identifying, at runtime, one or more matching tables among the plurality of target tables based on a comparison of the patterns of the source tables and the patterns of the plurality of target tables using a large language model; obtaining, at runtime, first sample attribute data from the source table and second sample attribute data from the selected matching table; and identifying, at runtime, one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data using a large language model.

[0181] Clause 2. The computing system according to Clause 1, wherein identifying one or more matching tables comprises: at runtime, constructing a first prompt, wherein constructing the first prompt comprises inserting the schema of the source table and the schema of a plurality of target tables into a first prompt template; and at runtime, using the first prompt to prompt a large language model.

[0182] Clause 3. The computing system according to Clause 2, wherein constructing the first prompt further includes removing some attributes from the schema of the source table or the schema of multiple target tables from the first prompt based on one or more predefined filtering criteria.

[0183] Clause 4. A computing system according to any one of Clauses 1-3, wherein retrieving a source table pattern includes obtaining a textual description of the source table and attributes of the source table from a first dictionary associated with a first database at runtime, wherein retrieving multiple target tables pattern includes obtaining textual descriptions of multiple target tables and attributes of multiple target tables from a second dictionary associated with a second database at runtime.

[0184] Clause 5. A computing system according to any one of Clauses 1-4, wherein the mode of retrieving a source table includes obtaining statistics of the source table at runtime, and wherein the mode of retrieving multiple target tables includes obtaining statistics of multiple target tables at runtime.

[0185] Clause 6. A computing system according to any one of Clauses 1-5, wherein identifying one or more matching tables comprises: generating a first vector embedding based on patterns from a source table; and generating a second vector embedding based on patterns from multiple target tables.

[0186] Clause 7. The computing system according to Clause 6, wherein identifying one or more matching tables further includes: measuring table similarity between a first vector embedding and a second vector embedding; and identifying table similarity greater than a predefined threshold.

[0187] Clause 8. A computing system according to any one of Clauses 1-7, wherein identifying one or more pairs of matching attributes comprises: constructing a second cue at runtime; and using the second cue to cue a large language model at runtime, wherein constructing the second cue comprises inserting first sample attribute data and second sample attribute data into a second cue template.

[0188] Clause 9. The computing system according to Clause 8, wherein constructing the second prompt further includes removing some of the first sample attribute data or the second sample attribute data from the second prompt based on one or more predefined filtering criteria.

[0189] Clause 10. A computing system according to any one of Clauses 1-9, wherein identifying one or more pairs of matching attributes comprises: generating a third vector embedding based on first sample attribute data; generating a fourth vector embedding based on second sample attribute data; and measuring the attribute similarity between the third vector embedding and the fourth vector embedding.

[0190] Clause 11. A computer-implemented method for improved pattern matching of two databases, the method comprising: retrieving, at runtime, patterns of a source table from a first database and patterns of a plurality of target tables from a second database; identifying, at runtime, one or more matching tables among the plurality of target tables based on a comparison of the patterns of the source tables and the patterns of the plurality of target tables using a large language model; obtaining, at runtime, first sample attribute data from the source tables and second sample attribute data from the selected matching tables; and identifying, at runtime, one or more pairs of matching attributes between the source tables and the selected matching tables based on a comparison of the first sample attribute data and the second sample attribute data using a large language model.

[0191] Clause 12. The computer-implemented method according to Clause 11, wherein identifying one or more matching tables comprises: at runtime, constructing a first prompt, wherein constructing the first prompt comprises inserting the schema of a source table and the schemas of multiple target tables into a first prompt template; and at runtime, using the first prompt to prompt a large language model.

[0192] Clause 13. A computer-implemented method according to any one of Clauses 11-12, wherein retrieving the schema of a source table includes obtaining a textual description of the source table and attributes of the source table from a first dictionary associated with a first database at runtime, wherein retrieving the schema of a plurality of target tables includes obtaining a textual description of a plurality of target tables and attributes of a plurality of target tables from a second dictionary associated with a second database at runtime.

[0193] Clause 14. A computer-implemented method according to any one of Clauses 11-13, wherein retrieving a source table includes obtaining statistics of the source table at runtime, and wherein retrieving multiple target tables includes obtaining statistics of multiple target tables at runtime.

[0194] Clause 15. A computer-implemented method according to any one of Clauses 11-14, wherein identifying one or more matching tables comprises: generating a first vector embedding based on patterns from a source table; and generating a second vector embedding based on patterns from a plurality of target tables.

[0195] Clause 16. The computer-implemented method according to Clause 15, wherein identifying one or more matching tables further comprises: measuring table similarity between a first vector embedding and a second vector embedding; and ranking one or more matching tables based on the table similarity.

[0196] Clause 17. A computer-implemented method according to any one of Clauses 11-16, wherein identifying one or more pairs of matching attributes comprises: constructing a second prompt at runtime; and using the second prompt to prompt a large language model at runtime, wherein constructing the second prompt comprises inserting first sample attribute data and second sample attribute data into a second prompt template.

[0197] Clause 18. A computer-implemented method according to any one of Clauses 11-17, wherein identifying one or more pairs of matching attributes comprises: generating a third vector embedding based on first sample attribute data; and generating a fourth vector embedding based on second sample attribute data.

[0198] Clause 19. The computer-implemented method according to Clause 18, wherein identifying a pair or more pairs of matching attributes further comprises: measuring the attribute similarity between a third vector embedding and a fourth vector embedding; and ranking the pair or more pairs of matching attributes based on the attribute similarity.

[0199] Clause 20. One or more non-transitory computer-readable media having computer-executable instructions encoded thereon, the computer-executable instructions causing one or more processors to perform an improved pattern matching method for two databases, the method comprising: retrieving, at runtime, patterns of a source table from a first database and patterns of a plurality of target tables from a second database; identifying, at runtime, one or more matching tables among the plurality of target tables based on a comparison of the patterns of the source tables and the patterns of the plurality of target tables using a large language model; obtaining, at runtime, first sample attribute data from the source table and second sample attribute data from a selected matching table; and identifying, at runtime, one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data using a large language model.

[0200] Example Alternatives

[0201] The techniques from any example can be combined with the techniques described in any one or more other examples. Given the many possible embodiments to which the principles of the disclosed techniques can be applied, it should be understood that the illustrated embodiments are examples of the disclosed techniques and should not be considered as limiting the scope of the disclosed techniques. Rather, the scope of the disclosed techniques includes everything covered by the scope and spirit of the appended claims.

Claims

1. An improved pattern matching computational system for two databases, the computational system comprising: Memory; One or more hardware processors, coupled to memory; and One or more computer-readable storage media store instructions that, when loaded into memory, cause the one or more hardware processors to perform operations, the operations including: At runtime, the schema of the source table is retrieved from the first database, and the schema of multiple target tables is retrieved from the second database; At runtime, a large language model is used to identify one or more matching tables among the multiple target tables based on a comparison of the patterns of the source table and the patterns of the multiple target tables. During runtime, the system obtains first sample attribute data from the source table and second sample attribute data from the selected matching table; and At runtime, a large language model is used to identify one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data.

2. The computing system according to claim 1, wherein, Identifying the one or more matching tables includes: At runtime, a first prompt is constructed, wherein constructing the first prompt includes inserting the schema of the source table and the schemas of the plurality of target tables into a first prompt template; and At runtime, the first prompt is used to prompt the large language model.

3. The computing system according to claim 2, wherein, Constructing the first suggestion also includes removing some attributes from the schema of the source table or the schema of the multiple target tables from the first suggestion based on one or more predefined filtering criteria.

4. The computing system according to claim 1, in, Retrieving the schema of the source table involves obtaining the textual description and attributes of the source table from a first dictionary associated with the first database at runtime. The process of retrieving the patterns of the multiple target tables includes obtaining textual descriptions and attributes of the multiple target tables from a second dictionary associated with the second database at runtime.

5. The computing system according to claim 1, in, Retrieving the schema of the source table includes obtaining statistical information about the source table at runtime. The method for retrieving the multiple target tables includes obtaining statistical information about the multiple target tables at runtime.

6. The computing system according to claim 1, wherein, Identifying the one or more matching tables includes: Generate the first vector embedding based on the pattern from the source table; and A second vector embedding is generated based on the patterns of the multiple target tables.

7. The computing system according to claim 6, wherein, Identifying the one or more matching tables also includes: Measure the table similarity between the first and second vector embeddings; and Identify table similarities greater than a predefined threshold.

8. The computing system according to claim 1, wherein, Identifying one or more pairs of matching attributes includes: At runtime, construct the second prompt; and At runtime, a second prompt is used to suggest information to the large language model. The construction of the second prompt includes inserting the first sample attribute data and the second sample attribute data into the second prompt template.

9. The computing system according to claim 8, wherein, Constructing the second prompt also includes removing some of the first sample attribute data or the second sample attribute data from the second prompt based on one or more predefined filtering criteria.

10. The computing system according to claim 1, wherein, Identifying one or more pairs of matching attributes includes: A third vector embedding is generated based on the attribute data of the first sample. A fourth vector embedding is generated based on the attribute data of the second sample; and Measure the attribute similarity between the third and fourth vector embeddings.

11. A computer-implemented method for improved pattern matching of two databases, the method comprising: At runtime, the schema of the source table is retrieved from the first database, and the schema of multiple target tables is retrieved from the second database; At runtime, a large language model is used to identify one or more matching tables among the multiple target tables based on a comparison of the patterns of the source table and the patterns of the multiple target tables. During runtime, the system obtains first sample attribute data from the source table and second sample attribute data from the selected matching table; and At runtime, a large language model is used to identify one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data.

12. The computer-implemented method according to claim 11, wherein, Identifying the one or more matching tables includes: At runtime, a first prompt is constructed, wherein constructing the first prompt includes inserting the schema of the source table and the schemas of the plurality of target tables into a first prompt template; and At runtime, the first prompt is used to prompt the large language model.

13. The computer-implemented method according to claim 11, in, Retrieving the schema of the source table involves obtaining the textual description and attributes of the source table from a first dictionary associated with the first database at runtime. The process of retrieving the patterns of the multiple target tables includes obtaining textual descriptions and attributes of the multiple target tables from a second dictionary associated with the second database at runtime.

14. The computer-implemented method according to claim 11, in, Retrieving the schema of the source table includes obtaining statistical information about the source table at runtime. The method for retrieving the multiple target tables includes obtaining statistical information about the multiple target tables at runtime.

15. The computer-implemented method according to claim 11, wherein, Identifying the one or more matching tables includes: Generate the first vector embedding based on the pattern from the source table; and A second vector embedding is generated based on the patterns of the multiple target tables.

16. The computer-implemented method according to claim 15, wherein, Identifying the one or more matching tables also includes: Measure the table similarity between the first and second vector embeddings; and The one or more matching tables are sorted based on table similarity.

17. The computer-implemented method according to claim 11, wherein, Identifying one or more pairs of matching attributes includes: At runtime, construct the second prompt; and At runtime, a second prompt is used to suggest information to the large language model. The construction of the second prompt includes inserting the first sample attribute data and the second sample attribute data into the second prompt template.

18. The computer-implemented method according to claim 11, wherein, Identifying one or more pairs of matching attributes includes: Generate a third vector embedding based on the attribute data of the first sample; and A fourth vector embedding is generated based on the attribute data of the second sample.

19. The computer-implemented method according to claim 18, wherein, Identifying one or more pairs of matching attributes also includes: Measuring the attribute similarity between the third and fourth vector embeddings; and The one or more pairs of matching attributes are sorted based on attribute similarity.

20. One or more non-transitory computer-readable media having computer-executable instructions encoded thereon, the computer-executable instructions causing one or more processors to perform a method for improved pattern matching for two databases, the method comprising: At runtime, the schema of the source table is retrieved from the first database, and the schema of multiple target tables is retrieved from the second database; At runtime, a large language model is used to identify one or more matching tables among the multiple target tables based on a comparison of the patterns of the source table and the patterns of the multiple target tables. During runtime, the system obtains first sample attribute data from the source table and second sample attribute data from the selected matching table; and At runtime, a large language model is used to identify one or more pairs of matching attributes between the source table and the selected matching table based on a comparison of the first sample attribute data and the second sample attribute data.

Images