System and method for data standardization

Abstract

A system and a method for training a standardization algorithm is disclosed. The system may receive customer data. The customer data may be transformed into columnar data. Further, the columnar data may be analyzed to extract a set of features. The set of features may comprise a set of columnar features and a set of column type features. Further, context for the columnar data may be determined. Furthermore, standard column headers may be identified for columns in the columnar data. A standardization algorithm may be trained using the standard column headers and the customer data.

Description

SYSTEM AND METHOD FOR DATA STANDARDIZATIONPRIORITY INFORMATION

[0001] The present application claims priority from Indian Patent Application No. 202321040907 dated June 15, 2023.TECHNICAL FIELD

[0002] The present subject matter described herein, in general, relates to the analysis of customer data and transforming the customer data into a standardized format and, more specifically, to customer onboarding in the context of data standardization and mapping.BACKGROUND

[0003] In the era of big data, businesses and organizations often rely on Customer Relationship Management (CRM) systems to manage and analyze vast amounts of customer data. However, customer data collected from different sources frequently comes in various formats, with inconsistent column headers and structures, posing significant challenges for data integration, analysis, and utilization. The conventional approach involves product consultants spending significant time and effort understanding customer data and creating rules to map the customer data to a standardized template. The above process often includes back-and-forth communication with the customers to gain insights and ensure accurate mappings. However, several challenges limit the efficiency and effectiveness of this approach. Mapping numerous data fields associated with a customer's information can take considerable time, and this prolonged duration hampers the speed and agility of the customer onboarding process. To address these issues, there is a growing need for automated solutions that can standardize disparate datasets into a consistent format,facilitating seamless data processing and enhancing the quality of insights derived from the data.SUMMARY

[0004] Before the present system(s) and method(s), are described, it is to be understood that this application is not limited to the particular system(s), and methodologies described, as there can be multiple possible embodiments which are not expressly illustrated in the present disclosures. It is also to be understood that the terminology used in the description is for the purpose of describing the particular implementations or versions or embodiments only and is not intended to limit the scope of the present application. This summary is provided to introduce aspects related to a system and a method for training a standardization algorithm. This summary is not intended to identify essential features of the claimed subject matter nor is it intended for use in determining or limiting the scope of the claimed subject matter.

[0005] In one implementation, a method for training a standardization algorithm is disclosed. The method may comprise receiving customer data comprising one or more rows and one or more columns. A column of the one or more columns may comprise a column header and a set of column values. The method may further comprise splitting the customer data into columnar data. The columnar data may comprise a set of column headers and corresponding sets of column values. Further, a set of features for the columnar data may be determined based on the column header and the set of column values, wherein the set of features comprises a set of columnar features and a set of column type features. Subsequently, context of the columnar data may be determined based on the set of column headers, the set of column type features, and historical data. The historical data may comprise feedback related to context determined for the columnar data in the past. Furthermore, a set of standard column headers is identified based on the set of column values, the set of columnarfeatures. A standardization algorithm is then trained based on the customer data, the context, and the set of standard column headers. The standardization algorithm may be used to determine standard column headers for new customer data.

[0006] In an embodiment, a set of machine learning models may be trained based on the set of features. The set of machine learning models comprises a vector space model and a Multi-Input Deep Neural Network (MI-DNN) model. The vector space model may be trained based on column type features to identify context associated with the column-type features. Further, a Multi-Input Deep Neural Network (MIDNN) model may be trained to map the column values to standard column headers based on the set of features. Further, a standardization algorithm may be trained for mapping the new customer data to a standardized template by identifying the standard column headers for the columns in the new customer data based on the output of the set of machine learning models. It may be noted that the standardization algorithm may be continuously trained using the new customer data and feedback. The feedback may comprise a modification of the standard column headers determined by the standardization algorithm.

[0007] In one aspect, the aforementioned method for training a standardization algorithm may be performed by system comprising a processor and a memory using the processor to execute programmed instructions stored in the memory. The system may receive customer data comprising one or more rows and one or more columns. A column of the one or more columns may comprise a column header and a set of column values. The system may split the customer data into columnar data. The columnar data may comprise a set of column headers and corresponding sets of column values. Further, a set of features for the columnar data may be determined based on the column header and the set of column values, wherein the set of features comprises a set of columnar features and a set of column type features. Subsequently, context of the columnar data may be determined based on the set of column headers,the set of column type features, and historical data. The historical data may comprise feedback related to context determined for the columnar data in the past. Furthermore, a set of standard column headers is identified based on the set of column values, the set of columnar features. The system may then train a standardization algorithm based on the customer data, the context, and the set of standard column headers. The standardization algorithm may be used to determine standard column headers for new customer data.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing detailed description of embodiments is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present subject matter, an example of a construction of the present subject matter is provided as figures, however, the invention is not limited to the specific method and system for training a standardization algorithm disclosed in the document and the figures.

[0009] The present subject matter is described in detail with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the drawings to refer to various features of the present subject matter.

[0010] Figure 1 illustrates a network implementation for standardization, in accordance with an embodiment of the present subject matter.

[0011] Figure 2 illustrates a method for training a standardization algorithm, in accordance with an embodiment of the present subject matter.

[0012] Figure 3 illustrates an example of standardization, in accordance with an embodiment of the present subject matter.

[0013] Figure 4 illustrates a set of columnar features, in accordance with an embodiment of the present subject matter.

[0014] Figure 5 illustrates an example artificial neural network, in accordance with an embodiment of the present subject matter.

[0015] Figure 6 illustrates an example view of an embedding space, in accordance with an embodiment of the present subject matter.

[0016] The figure depicts an embodiment of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the disclosure described herein.DETAILED DESCRIPTION

[0017] Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The words "receiving," "determining," "analyzing," "training," "mapping," "extracting," "standardizing," and other forms thereof, are intended to be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any system and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the exemplary, system and methods are now described.

[0018] The disclosed embodiments are merely examples of the disclosure, which may be embodied in various forms. Various modifications to the embodiment will be readily apparent to those skilled in the art and the generic principles herein may be applied to other embodiments. However, one of ordinary skill in the art will readily recognize that the present disclosure is not intended to be limited to the embodiments described but is to be accorded the widest scope consistent with the principles and features described herein.

[0019] The present invention relates to training an algorithm trained to standardize CRM data, transforming diverse data structures into a uniform format based on a predefined template. This algorithm leverages advanced machine learning techniques to modify column headers and align data entries with standardized headers, enabling more efficient data integration and analysis. An algorithm to standardize the data is necessary to derive insights for different companies having their data stored in various formats. The standardized data is crucial for training artificial intelligence (Al) models that may be used for different purposes and improving their accuracy and performance by providing a consistent input format.

[0020] Referring now to Figure 1, a network implementation 100 of a system 102 for standardization is disclosed. Initially, the system 102 may receive data from a customer. In an example, the software may be installed on a user device 104-1. It may be noted that the one or more users may access the system 102 through one or more user devices 104-2, 104-3... 104-N, collectively referred to as user devices 104, hereinafter, or applications residing on the user devices 104. The system 102 may receive the customer data from one or more user devices 104. Further, the system 102 may receive feedback to improve the mapping of the customer data. The feedback may be provided by using the one or more user devices 104.

[0021] Although the present disclosure is explained considering that the system 102 is implemented on a server, it may be understood that the system 102 may be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, a virtual environment, a mainframe computer, a server, a network server, a cloud-based computing environment. It will be understood that the system 102 may be accessed by multiple users through one or more user devices 104-1, 104-3... 104-N. In one implementation, the system 102 may comprise the cloud-based computing environment in which the user may operate individual computing systems configured to execute remotely located applications. Examples of the user devices 104 may include, but are not limited to, a portable computer, apersonal digital assistant, a handheld device, and a workstation. The user devices 104 are communicatively coupled to the system 102 through a network 106.

[0022] In one implementation, the network 106 may be a wireless network, a wired network, or a combination thereof. The network 106 can be implemented as one of the different types of networks, such as intranet, local area network (LAN), wide area network (WAN), the internet, and the like. The network 106 may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further the network 106 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.

[0023] In one embodiment, the system 102 may include at least one processor 108, an input / output (I / O) interface 110, and a memory 112. The at least one processor 108 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, Central Processing Units (CPUs), state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the at least one processor 108 is configured to fetch and execute computer-readable instructions stored in the memory 112.

[0024] The I / O interface 110 may include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like. The I / O interface 110 may allow the system 102 to interact with the user directly or through the client devices 104. Further, the I / O interface 110 may enable the system 102 to communicate with other computing devices, such as web servers and external data servers (not shown). The I / O interface 110 can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such asWLAN, cellular, or satellite. The I / O interface 110 may include one or more ports for connecting a number of devices to one another or to another server.

[0025] The memory 112 may include any computer -readable medium or computer program product known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and / or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, Solid State Disks (SSD), optical disks, and magnetic tapes. The memory 112 may include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement particular abstract data types. The memory 112 may include programs or coded instructions that supplement applications and functions of the system 102. In one embodiment, the memory 112, amongst other things, serves as a repository for storing data processed, received, and generated by one or more of the programs or the coded instructions.

[0026] At first, a user may use the user device 104 to access the system 102 via the I / O interface 110. The user may register the user devices 104 using the I / O interface 110 in order to use the system 102. In one aspect, the user may access the I / O interface 110 of the system 102. The detailed functioning of the system 102 is described below with the help of more figures.

[0027] The present subject matter discloses a method and a system for training a standardization algorithm. The method may comprise receiving customer data. The customer data may be obtained from a Customer Relationship Management (CRM) platform. The customer data may be received using an automated system linked to a customer data server using Application Programming Interfaces (APIs). The automated system may be programmed to receive the customer data at regular intervals of time.

[0028] In an embodiment, the customer data may be in a tabular format and may comprise one or more rows and one or more columns. In an embodiment, a column of the one or more columns comprises a different type of data. The customer data maybe acquired from a Customer Relationship Management (CRM) tool, which typically utilizes its unique format and data types to provide and manage data. These CRM tools often employ proprietary choices of fields and data types and store the data in customized tables specifically defined by the CRM tool. In an embodiment, the customer data may be acquired from a marketing automation platform (MAP).

[0029] The customer data may comprise one or more rows and one or more columns. A column of the one or more columns may comprise a column header and a set of column values corresponding to the column header. The number of column values in the set of column values may be equal to the number of rows excluding the rows used to contain the column header.

[0030] The customer data may be then transformed into columnar data by splitting the data column wise. The columnar data may comprise a set of column headers and their corresponding sets of column values. For example, the columnar data may comprise 5 column headers and 5 sets of column values. One set of column values for each of the 5 column headers. Each set of column values may comprise zero or more column values. A column value may be a data entry related to the column header.

[0031] The system may then determine a set of features for the columnar data. The set of features may be determined for each column in the columnar data. The set of features may comprise a set of columnar features and a set of column type features. The set of columnar features may be derived based on the set of column values in a column and the set of column type features may be determined based on the column header of the column. The set of columnar features may include at least one of Column-wise aggregated features, Character Level Aggregated Features, Word embeddings, Contextual Paragraph Embeddings. A further breakdown of the columnar features is illustrated by Fig. 4. In an embodiment, the set of columnar features may be populated with a selection of features from the features illustrated in Fig. 4 for each column in the columnar data.

[0032] In an embodiment, the system may automatically select the columnar features based on the column values using a data processing algorithm. The data processingalgorithm may determine a data type of column values in each column to determine the suitable columnar features for each column.

[0033] The Column-wise Aggregated Features may include number of column values, column entropy, number of unique column values, number of column values of numeric data type, number of column values having alphabets, and statistical features including mean, median, mode, standard deviation, range, min, max, length distribution, and the like. Frequency based features such as value distribution and top frequent values may also be determined. Further, regex matches and date patterns may be identified in the column values.

[0034] The column-wise aggregated features such as mean, median, mode, and standard deviation are crucial for understanding and processing various types of data within each column. These features are not limited to character counts but extend to numerical values, dates, and other data types. For numerical values, the mean represents the average value, providing a measure of central tendency, while the median indicates the middle value in a sorted list, offering a robust measure of central tendency less affected by outliers. The mode identifies the most frequently occurring number, highlighting common values, and the standard deviation measures the dispersion or variability of the numbers, indicating how spread out the values are around the mean.

[0035] For dates and times, the mean can represent the average date or time, useful for identifying central tendencies in event timing. The median date or time provides a central point in a series of dates, and the mode indicates the most common event dates or times. The standard deviation in dates or times shows the variability in the timing of events, illustrating how spread out the events are over a period. In the case of categorical values, which might be numerically encoded or assessed based on counts, the mode identifies the most frequently occurring category, while entropy measures the unpredictability or disorder within the data, indicating the diversity of categories.

[0036] For text data, beyond character counts, aggregated features can include statistics on word counts per entry, the occurrence of special characters, and distributions from topic modeling. For example, word count statistics such as mean, median, mode, and standard deviation provide insights into the typical length and variability of text entries. Special character counts and their statistics reveal patterns in the usage of non-alphanumeric symbols, which can be critical for data normalization. Topic distribution statistics, derived from topic modeling, offer insights into the predominant themes within textual data.

[0037] These comprehensive aggregated features enable a detailed statistical profiling of each column, enhancing the ability to clean, normalize, and engineer features for data analysis and machine learning tasks. This detailed understanding of the data's statistical properties is essential for ensuring accurate and reliable processing, ultimately improving the effectiveness of data-driven decision-making and predictive modeling.

[0038] The Character Level Aggregated Features may be used to identify unique columns by calculating one or more parameters including at least one of count, mean, min, max, and skewness of column values having characters including a-z, A-Z, numerical characters, and special characters. The skewness of column values corresponds to distribution of the column values on a graph. The skewness may be positive or negative based on the column values. For example, in a dataset representing ages within a population, an exemplary method for determining skewness involves plotting the ages on a graph with the horizontal axis representing ages and the vertical axis indicating the frequency of each age occurrence. Upon analysis, if the majority of ages cluster towards younger individuals, with fewer occurrences of older ages, the data exhibits positive skewness, colloquially referred to as right-skewed, denoting a longer tail towards older ages. Conversely, if the distribution is characterized by a concentration of older ages and a scarcity of younger ages, the dataset demonstrates negative skewness, or left-skewness, with a lengthier tail towards younger ages. This skewness analysis aids in understanding theasymmetry of the age distribution, crucial for various statistical analyses and modeling endeavors.

[0039] The system may identify columns having a similar value of the one or more parameters as identical and not unique. The system may clean the columnar data based on the one or more parameters to erase duplicate columns.

[0040] Word embeddings are vector representations of words, where each word is mapped to a dense vector of real numbers in a high-dimensional vector space. These vectors capture semantic and syntactic relationships between words based on their contextual usage patterns in a large text corpus. Word embeddings may be generated using one or more pretrained models including at least one of Word to Vector (Word2Vec), Global Vectors (GloVe), or FastText. The pretrained models may be used to provide a mapping between column values and word vectors of the column values. The word vectors may be used for easier analysis of the column values of the column.

[0041] Contextual Paragraph Embeddings may be used to analyze all column values in the column collectively to determine a context of the column values and determine patterns in the column values. Contextual paragraph embedding may be determined using algorithms like Bidirectional Encoder Representations from Transformers (BERT) or other transformer models, Document to Vector (DOC2Vec), Paragraph Vectors with Distributed Bag of Words (PV-DBOW).

[0042] The set of column type features may comprise column type features for each column in the columnar data. The column type features are determined based on the column header. The column type features may include tokens, string similarity value, vocabulary matching value, past mappings, and column type. Tokens include meaningful words in a column header. String similarity value may be a comparison between a column header and a standard column header or another header from the columnar data. Vocabulary matching value may help in detecting null column headers and column headers that may not be required for a particular customer type.The vocabulary matching value may be determined by comparing the column header with a dataset comprising column headers relevant for a customer.

[0043] Past mappings may be extracted from historical data comprising standard column headers mapped to a column header in past iterations of the vector space algorithm. The column type may correspond to data type of column values in the column. The column type may be determined based on the column header and past mappings.

[0044] Further to determining the set of features, the system may determine context of the columnar data based on the set of column headers, the column type features, and historical data. The historical data may comprise feedback for context determined for a column header in the past. The feedback may be a change in the context or a rating for the determined context. In an embodiment, the system may determine the context using a vector space model. The vector space model may be trained using a training dataset comprising a plurality of customer databases comprising a plurality of columns, column headers, corresponding column type features for the columns, context labelled for each column of the plurality of columns, and the feedback. In an embodiment, the vector space model may be used to compare converted word vector of the column header with word vectors of column headers from the training data using an N-dimensional vector space. The similarity between the word vectors may be determined by calculating the Euclidean distance between the vectors plotted in the N-dimensional vector space. The context of a word vector of the column header in the training data that is most similar to the current column header is determined as the context of the column. Similarly, context of each column of the columnar data is determined.

[0045] Subsequently, the system may determine a set of standard column headers. The set of standard column headers may comprise a standard column header for each column in the columnar data. The standard column header for a column may be determined using a Multi-Input Deep Neural Network (MI-DNN) model. The MIDNN model may be trained using a training dataset comprising a plurality ofcustomer databases comprising a plurality of columns, corresponding sets of column values, corresponding set of features, and standard column headers labelled to the plurality of columns.

[0046] In an embodiment, the system may train the MI-DNN model based on a template. The template may comprise one or more standard column headers and one or more corresponding standard column values. The system may train the MI-DNN model to compare column values from the columnar data with the standard column values and determine the standard column header based on the comparison. The system may select the standard column header corresponding to the standard column values that are most similar to the column values.

[0047] In an embodiment, the MI-DNN may comprise of layers and nodes based on a custom architecture. The custom architecture may be determined by testing a plurality of architectures based on their accuracy. In an example, the architecture of the MI- DNN may be specifically designed for the standardization and processing of diverse input data sets through a structured series of layers. The architecture may initiate with four distinct input layers, each tailored to different sets of features with respective shapes 960, 201, 400, and 27. The term "shape" herein refers to the dimensions of the input data, indicating how many individual features are present in each set. Each of these input layers may be followed by a batch normalization layer, which normalizes the inputs by adjusting and scaling the activations. The term "activations" refers to the output values generated by neurons in a layer after the application of a specific activation function, which introduces non-linearity into the model. This normalization process ensures that each input feature contributes equally to the learning process, thereby enhancing the stability and performance of the neural network.

[0048] Subsequent to normalization, the outputs from the first three input layers may be processed through dense layers where each neuron receives input from all neurons in the preceding layer. These dense layers may comprise specified units of 300, 200, and 400 neurons, respectively. The term "units" refers to the number of neurons within a layer. Following these dense layers, dropout layers are employed to mitigateoverfitting by randomly setting a fraction of the neurons to zero during the training phase. The fourth input layer undergoes batch normalization directly, bypassing the initial dense layer, indicating a distinct role or varying importance of its features. The outputs from the dense and batch normalization layers are then concatenated into a single tensor, with the term "concatenation" referring to the process of combining multiple tensors along a specified dimension to create a unified representation. This concatenated tensor undergoes further standardization and is processed through additional dense layers with 500 neurons each, followed by a dropout layer for regularization. The final output layer consists of 12 neurons, producing the neural network's predictions or classifications. This detailed architecture, characterized by its systematic standardization and integration of features, effectively ensures that diverse input data is consistently and efficiently processed, resulting in accurate and reliable model outputs.

[0049] The MI-DNN model may analyze the set of features of a column with respect to the training data to identify a standard column header for the column. The model may select the standard column header of a column from the training data having the most similar features. The MI-DNN model may be used to determine the standard column header of each column in the columnar data thereby populating the set of standard column headers.

[0050] Further, the system may train a standardization algorithm based on the set of standard column headers, the customer data, and the context. The standardization algorithm may be trained to determine standard column headers for customer data of a new customer based on past customer data of the new customer.

[0051] In an embodiment, the standardization algorithm may be trained based on the vector space model and the MI-DNN model. A plurality of customer databases may be used to generate outputs from these models. The outputs of these models along with the plurality of customer databases may be used to train the standardization algorithm.

[0052] In an embodiment, the standardization algorithm may determine a list of suitable standard column headers for a column and a score for each of the suitable standard column headers. The score may be calculated based on the similarity between the set of features of the column and the set of features of a column in the training data. In an embodiment, standardization algorithm may be trained using at least one of supervised learning, unsupervised learning, reinforcement learning, and transfer learning. The standardization algorithm may determine weights for each feature of the set of features for determining the set of standard column headers.

[0053] In an embodiment, the standardization algorithm may modify weights assigned to the set of features based on at least one of accuracy, execution time, and feedback received on determined set of standard column headers.

[0054] In a multi-company scenario, different organizations often have diverse customer databases with varying structures, formats, and terminologies. Standardizing this data is crucial for creating personalized Al models tailored to each company's specific needs. The trained standardization algorithm ensures that disparate datasets are unified into a consistent format, enabling the development of accurate and effective Al models for various applications such as marketing, customer service, sales forecasting, and more.

[0055] In an embodiment, the system automatically calibrates rules for standardization of any customer data based on the training of the standardization algorithm.

[0056] In an embodiment, the customer data may include email data. The email data may be analyzed to determine a set of parameters including context of conversation and intent. The set of parameters may be determined based on a response rate of an email and text analysis using Natural Language Processing, Natural Language Understanding algorithms.

[0057] Referring now to figure 2, a method 200 for training a standardization algorithm is disclosed, in accordance with an embodiment of the present subject matter. The method 200 may be described in the general context of computerexecutable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types.

[0058] The order in which the method 200 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method 200 or alternate methods for standardization. Additionally, individual blocks may be deleted from the method 200 without departing from the scope of the subject matter described herein. Furthermore, the method 200 for standardization is disclosed can be implemented in any suitable hardware, software, firmware, or combination thereof. However, for ease of explanation, in the embodiments described below, the method 200 may be considered to be implemented in the above-described system 102.

[0059] At block 202, customer data may be received comprising one or more rows and one or more columns.

[0060] At block 204, the customer data may be transformed into columnar data by splitting the customer data column wise. Column headers and the corresponding sets of column values are determined to split the customer data.

[0061] At block 206, the customer data may be analyzed to determine a set of features comprising a set of columnar features and a set of column type features. The set of columnar features may be extracted based on column values and the set of column type features may be extracted based on column headers.

[0062] At block 208, a vector space model may generate a context for a column as an output based on the column-type features.

[0063] At block 210, a Multi-Input Deep Neural Network (MI-DNN) model may be trained to map the column values to standardized column headers based on the extracted set of features. The MI-DNN model may generate a standardized header for a column as an output based on the set of features.

[0064] At block 212, a standardization algorithm may be trained based on the set of machine learning models, a template, and a plurality of customer databases. The standardization algorithm may be trained using the context received from the vector space model and the standardized headers received from the MI-DNN model. The template may comprise a set of standard column headers mapped to corresponding sets of standard column values.

[0065] Referring to Figure 3, an example 300 of standardization is disclosed. The system may receive customer data comprising one or more rows and one or more columns. In the example, the one or more columns have column headers “Industry”, “Market Value”, and “Country”. The standardization algorithm may analyze the customer data and determine that the column header “Country” is incorrect. Further, the standardization algorithm may identify the standard column header for the column as City. The customer data is standardized by changing the column headers to the standard column headers - “Industry”, “Market Value”, and “City”. Consider that the customer data will be used to train an Al model to filter Top industries in a city based on the market value. Since the column header is “country” instead of “city,” the Al model may make mistakes in predicting top industries in a city because the Al model does not have information of cities from the customer data. Therefore, using the standardization algorithm can correct such mistakes or irregularities in the column headers of customer data files to train an Al model with better accuracy.

[0066] Referring to Figure 4, an example 400 of a set of columnar features is shown. The set of columnar features comprise column-wise aggregated features, character level aggregated features, word embeddings, and contextual paragraph embeddings. The column-wise aggregated features may comprise at least one of number of values, column entropy, fraction of values with unique content, fraction of values with numerical characters, fraction of values with alphabetical characters, mean and standard deviation (std.) of the number of numerical characters in values, mean and std. of the number of alphabetical characters in values, mean and std. of the number special characters in values, mean and std. of the number of words in values, stats,sum, min, max, median, mode, kurtosis, skewness, any / all-Boolean of the length of values. Further, the character level aggregated features are utilized to identify ID columns OR discrete columns.

[0067] Furthermore, the column may be analyzed to determine the relationship between the words or column values, or cell values by using techniques like Word2Vec or Glove or Fasttext Vectors. The system may calculate word vectors using the mentioned technologies. The word vector is a mathematical representation of a word in a vector space, typically in a high-dimensional space where each dimension corresponds to a certain aspect or feature of the word. It may be noted that the words that are not part of vocabulary are omitted from the analysis. Determining relationships between column values in a dataset utilizing word vectors involves analyzing semantic similarities encoded within the vectors. By assessing similarity metrics like cosine similarity and applying clustering techniques such as K-means or hierarchical clustering, columns with akin semantic meanings can be identified. Furthermore, algebraic manipulation of word vectors permits the inference of relationships between column values through vector arithmetic. It may be noted that the system does not utilize a context sensitive embedding to analyze the column.

[0068] Contextual Paragraph Embeddings may be used to capture the topic-word distribution across the column. It may be noted that the contextual embedding may be performed by using Bidirectional Encoder Representations from Transformers (BERT) or DOC2VEC / PV-DBOW (Distributed Bag of Words version of Paragraph Vector.

[0069] Figure 5 illustrates an example artificial neural network (“ANN”) 500 of the machine learning algorithms and models described above. In particular embodiments, an ANN may refer to a computational model comprising one or more nodes. Example ANN 500 may comprise an input layer 510, hidden layers 520, 530, 540, and an output layer 550. Each layer of the ANN 500 may comprise one or more nodes, such as a node 505 or a node 515. In particular embodiments, each node of an ANN may be connected to another node of the ANN. As an example, and not by way oflimitation, each node of the input layer 510 may be connected to one of more nodes of the hidden layer 520. In particular embodiments, one or more nodes may be a bias node (e.g., a node in a layer that is not connected to and does not receive input from any node in a previous layer). In particular embodiments, each node in each layer may be connected to one or more nodes of a previous or subsequent layer. Although Figure 5 depicts a particular ANN with a particular number of layers, a particular number of nodes, and particular connections between nodes, this disclosure contemplates any suitable ANN with any suitable number of layers, any suitable number of nodes, and any suitable connections between nodes. As an example, and not by way of limitation, although Figure 5 depicts a connection between each node of the input layer 510 and each node of the hidden layer 520, one or more nodes of the input layer 510 may not be connected to one or more nodes of the hidden layer 520.

[0070] In particular embodiments, an ANN may be a feedforward ANN (e.g., an ANN with no cycles or loops where communication between nodes flows in one direction beginning with the input layer and proceeding to successive layers). As an example, and not by way of limitation, the input to each node of the hidden layer 520 may comprise the output of one or more nodes of the input layer 510. As another example and not by way of limitation, the input to each node of the output layer 550 may comprise the output of one or more nodes of the hidden layer 540. In particular embodiments, an ANN may be a deep neural network (e.g., a neural network comprising at least two hidden layers). In particular embodiments, an ANN may be a deep residual network. A deep residual network may be a feedforward ANN comprising hidden layers organized into residual blocks. The input into each residual block after the first residual block may be a function of the output of the previous residual block and the input of the previous residual block. As an example, and not by way of limitation, the input into residual block N may be F(x)+x, where F(x) may be the output of residual block N-l, x may be the input into residual block N-l.Although this disclosure describes a particular ANN, this disclosure contemplates any suitable ANN.

[0071] In particular embodiments, an activation function may correspond to each node of an ANN. An activation function of a node may define the output of a node for a given input. In particular embodiments, an input to a node may comprise a set of inputs. As an example, and not by way of limitation, an activation function may be an identity function, a binary step function, a logistic function, or any other suitable function.

[0072] In particular embodiments, the input of an activation function corresponding to a node may be weighted. Each node may generate output using a corresponding activation function based on weighted inputs. In particular embodiments, each connection between nodes may be associated with a weight. As an example, and not by way of limitation, a connection 525 between the node 505 and the node 515 may have a weighting coefficient of 0.4, which may indicate that 0.4 multiplied by the output of the node 505 is used as an input to the node 515. In particular embodiments, the input to nodes of the input layer may be based on a vector representing an object. Although this disclosure describes particular inputs to and outputs of nodes, this disclosure contemplates any suitable inputs to and outputs of nodes. Moreover, although this disclosure may describe particular connections and weights between nodes, this disclosure contemplates any suitable connections and weights between nodes.

[0073] In particular embodiments, the ANN may be trained using training data. As an example, and not by way of limitation, training data may comprise inputs to the ANN 500 and an expected output. As another example and not by way of limitation, training data may comprise vectors each representing a training object and an expected label for each training object. In particular embodiments, training the ANN may comprise modifying the weights associated with the connections between nodes of the ANN by optimizing an objective function. As an example, and not by way of limitation, a training method may be used (e.g., the conjugate gradient method, thegradient descent method, the stochastic gradient descent) to backpropagate the sum- of-squares error measured as a distance between each vector representing a training object (e.g., using a cost function that minimizes the sum-of-squares error). In particular embodiments, the ANN may be trained using a dropout technique. As an example, and not by way of limitation, one or more nodes may be temporarily omitted (e.g., receive no input and generate no output) while training. For each training object, one or more nodes of the ANN may have some probability of being omitted. The nodes that are omitted for a particular training object may be different than the nodes omitted for other training objects (e.g., the nodes may be temporarily omitted on an object-by-object basis). Although this disclosure describes training the ANN in a particular manner, this disclosure contemplates training the ANN in any suitable manner.

[0074] Figure 6 illustrates an example view of a vector space 600. In particular embodiments, an object or an n-gram may be represented in a d-dimensional vector space, where d denotes any suitable number of dimensions. Although the vector space 600 is illustrated as a three-dimensional space, this is for illustrative purposes only, as the vector space 600 may be of any suitable dimension. In particular embodiments, an n-gram may be represented in the vector space 600 as a vector referred to as a term embedding. Each vector may comprise coordinates corresponding to a particular point in the vector space 600 (i.e., the terminal point of the vector). As an example, and not by way of limitation, vectors 610, 620, and 630 may be represented as points in the vector space 600, as illustrated in Figure 6.

[0075] As another example and not by way of limitation, a dictionary trained to map text to a vector representation may be utilized, or such a dictionary may be itself generated via training. As another example and not by way of limitation, a model, such as Word2vec, may be used to map an n-gram to a vector representation in the vector space 600. In particular embodiments, an n-gram may be mapped to a vector representation in the vector space 600 by using a machine leaning model (e.g., aneural network). The machine learning model may have been trained using a sequence of training data (e.g., a corpus of objects each comprising n-grams).

[0076] In particular embodiments, an object may be represented in the vector space 600 as a vector referred to as a feature vector or an object embedding. In particular embodiments, an object may be mapped to a vector based on one or more properties, attributes, or features of the object, relationships of the object with other objects, or any other suitable information associated with the object. As an example, and not by way of limitation, an object text may be mapped to a vector by using an algorithm to respond to the user query. Features used to calculate the vector may be based on information obtained from natural language processing, context analysis, and natural language understanding.

[0077] In particular embodiments, the system 102 may calculate a similarity metric of vectors in vector space 600. A similarity metric may be a cosine similarity, a Minkowski distance, a Mahalanobis distance, a Jaccard similarity coefficient, or any suitable similarity metric. The similarity metric of two vectors may represent how similar the two objects or n-grams corresponding to the two vectors, respectively, are to one another, as measured by the distance between the two vectors in the vector space 600. As an example, and not by way of limitation, vector 610 and vector 620 may correspond to objects that are more similar to one another than the objects corresponding to vector 610 and vector 630, based on the distance between the respective vectors. Although this disclosure describes calculating a similarity metric between vectors in a particular manner, this disclosure contemplates calculating a similarity metric between vectors in any suitable manner.

[0078] Exemplary embodiments discussed above may provide certain advantages. Though not required to practice aspects of the disclosure, these advantages may include those provided by the following features.

[0079] Some embodiments of the system and the method streamlines the customer data standardization and mapping process, significantly reducing the time and effort required. By leveraging machine learning models, the system can process and mapdata more quickly and accurately than manual methods, enhancing overall operational efficiency.

[0080] The use of machine learning models improves the accuracy and consistency of the mapping results. By analyzing data patterns, extracting relevant features, and learning from past mappings, the system reduces the potential for human errors and biases, ensuring more reliable and standardized mappings across different customers.

[0081] The system is designed for handling large volumes of customer data, including complex and diverse data fields. The system is scalable which allows the system to efficiently process and analyze extensive datasets, ensuring that every data element is examined for effective standardization. This enables the system to handle the demands of growing customer bases and evolving data requirements.

[0082] The set of machine learning models continuously learn and improve over time. By training the models periodically with new customer data, the system stays up-to-date with evolving data patterns and customer requirements. This adaptability ensures that the system can effectively map and standardize data for the customers, incorporating valuable learnings from previous mappings. The new customer data refers to customer data of the customer from a period of time after initial training of the standardization algorithm.

[0083] The system may comprise a User Interface (UI) that enables product consultants or users to interact with the system, view mapping results, and make manual adjustments if required. The UI enhances usability and allows for customization, providing a seamless experience for users involved in the customer onboarding process.

[0084] Although implementations for methods and systems for training a standardization algorithm have been described in language specific to structural features and / or methods, it is to be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as examples of implementations for standardization.

Claims

Claims;1. A method for training a standardization algorithm, the method comprising: receiving, by a processor, customer data comprising one or more rows and one or more columns, wherein a column of the one or more columns comprises a column header and a set of column values; splitting, by the processor, the customer data into columnar data, wherein the columnar data comprises a set of column headers and corresponding sets of column values; determining, by the processor, a set of features for the columnar data based on the column header and the set of column values, wherein the set of features comprises a set of columnar features and a set of column type features; determining, by the processor, context of the columnar data based on the set of column headers, the set of column type features, and historical data; identifying, by the processor, a set of standard column headers based on the set of column values, the set of columnar features, and a template; and training, by the processor, a standardization algorithm to standardize new customer data, wherein the standardization algorithm is trained based on the customer data, the context, and the set of standard column headers.

2. The method in claim 1, wherein the column type features for the column are determined based on the column header, and wherein the columnar features for the column are determined based on the corresponding set of column values.

3. The method in claim 1, wherein the context is determined for at least a column of the columnar data using a vector space model.

4. The method in claim 1, wherein the historical data comprises feedback related to the context determined using the vector space model.

5. The method in claim 3, wherein the vector space model is trained using a training dataset comprising a plurality of customer databases comprising a plurality of columns, column headers, corresponding column type features for the columns, context labelled for each column of the plurality of columns, and the feedback.

6. The method in claim 1, wherein the set of standard column headers comprises a standard column header for each column in the columnar data.

7. The method in claim 6, wherein the standard column header is determined using a Multi-Input Deep Neural Network (MI-DNN) model.

8. The method in claim 7, wherein the MI-DNN model is trained using a training dataset comprising a plurality of customer databases comprising a plurality of columns, corresponding sets of column values, corresponding columnar features, and standard column headers labelled to the plurality of columns.

9. The method in claim 1, wherein the template may comprise a set of standard column headers mapped to corresponding sets of standard column values.

10. The method in claim 1, wherein the standardization algorithm is trained to determine standard column headers for the new customer data.

11. A system for training a standardization algorithm, the system comprising: a memory; a processor coupled to the memory, wherein the processor is configured to execute a set of instructions stored in the memory for: receiving customer data comprising one or more rows and one or more columns, wherein a column of the one or more columns comprises a column header and a set of column values; splitting the customer data into columnar data, wherein the columnar data comprises a set of column headers and corresponding sets of column values;determining a set of features for the columnar data based on the column header and the set of column values, wherein the set of features comprises a set of columnar features and a set of column type features; determining context of the columnar data based on the set of column headers, the set of column type features, and historical data; identifying a set of standard column headers based on the set of column values, the set of columnar features, and a template; and training a standardization algorithm based on the customer data, the context, and the set of standard column headers.

12. A non-transitory computer program product having embodied thereon a computer program for standardization, the computer program product storing instructions for: receiving customer data comprising one or more rows and one or more columns, wherein a column of the one or more columns comprises a column header and a set of column values; splitting the customer data into columnar data, wherein the columnar data comprises a set of column headers and corresponding sets of column values; determining a set of features for the columnar data based on the column header and the set of column values, wherein the set of features comprises a set of columnar features and a set of column type features; determining context of the columnar data based on the set of column headers, the set of column type features, and historical data; identifying a set of standard column headers based on the set of column values, the set of columnar features, and a template; and training a standardization algorithm to standardize new customer data, wherein the standardization algorithm is trained based on the customer data, the context, and the set of standard column headers.

Images