Methods, apparatus and systems for data mapping

Abstract

The present invention is entitled "Method, Apparatus, and System for Data Mapping". The present invention provides methods, apparatus, and systems for improving data mapping. An exemplary method may include: retrieving from a database a first plurality of data objects associated with a first database schema; determining a first data classifier corresponding to the first database schema; generating a mapping specification based at least in part on the first data classifier and the first plurality of data objects; and generating a second plurality of data objects based at least in part on the first plurality of data objects and the mapping specification.

Description

Technical Field

[0001] This disclosure relates in general to methods, apparatuses, and systems for data mapping, and more specifically to methods, apparatuses, and systems for generating data classifiers for data mapping. Background Technology

[0002] A database is a collection of information or data (such as data objects) that can be stored, accessed, and / or managed electronically by a computing system. A database schema refers to the structure that defines how data is organized and / or related to each other within the database. Databases can be implemented in a variety of fields, including but not limited to healthcare, retail, and financial services.

[0003] Many systems and methods have failed to overcome the technical challenges and difficulties associated with databases. For example, data may be stored in traditional databases that do not conform to common data models, and many systems and methods do not provide the ability to transform this data so that it can be analyzed by computational systems. These challenges and difficulties can be further amplified when traditional databases store large amounts of data that need to be analyzed. Summary of the Invention

[0004] According to various examples, an apparatus may be provided. The apparatus may include at least one processor and at least one non-transitory memory including program code. The at least one non-transitory memory and the program code may be configured to utilize the at least one processor such that the apparatus at least: retrieves a first plurality of data objects associated with a first database schema; determines a first data classifier corresponding to the first database schema based at least on the first plurality of data objects; generates a mapping specification based at least partially on the first data classifier and the first plurality of data objects; and generates a second plurality of data objects based at least partially on the first plurality of data objects and the mapping specification. In some examples, the mapping specification may be configured to transform the first plurality of data objects associated with the first database schema into a second plurality of data objects associated with a second database schema.

[0005] In some examples, the first plurality of data objects may include a first data table. In some examples, the first data table may include at least one data field. In some examples, the first data table may include at least one of name metadata, column metadata, or row metadata.

[0006] In some examples, when determining a first data classifier corresponding to a first database schema, at least one non-transitory memory and program code can be configured to use the at least one processor to cause the device to: retrieve at least one of name metadata, column metadata, or row metadata associated with a first plurality of data objects; and further determine the first data classifier based on at least one of the name metadata, column metadata, or row metadata.

[0007] In some examples, the first plurality of data objects may include a second data table. In some examples, the at least one non-transitory memory and program code may be configured to utilize the at least one processor to cause the apparatus to: determine relevance metadata associated with the first and second data tables; and further determine a first data classifier based on the relevance metadata.

[0008] In some examples, when determining a first data classifier corresponding to a first database schema, at least one non-transitory memory and program code can be configured to use the at least one processor to cause the apparatus to: determine domain metadata associated with a first data table; and further determine the first data classifier based on the domain metadata.

[0009] In some examples, prior to generating the mapping specification, the at least one non-transitory memory and program code may be configured to utilize the at least one processor to enable the apparatus to further: calculate a confidence score associated with the first data classifier; and determine whether the confidence score meets a predetermined threshold.

[0010] In some examples, a mapping specification can be generated in response to determining that the confidence score meets a predetermined threshold.

[0011] In some examples, the at least one non-transitory memory and program code may be configured to utilize the at least one processor to cause the device to further: determine that the confidence score does not meet a predetermined threshold; generate a user input request associated with the first data classifier; and receive user input in response to the user input request. In some examples, the user input request may include an electronic request for confirmation of the first data classifier.

[0012] In some examples, user input may include confirmation of a first data classifier. In some examples, a mapping specification may be generated in response to confirmation of the first data classifier.

[0013] In some examples, user input may include modifications to the first data classifier, wherein the at least one non-transitory memory and program code are configured to utilize the at least one processor to enable the device to further modify the first data classifier based on the user input.

[0014] In some examples, the at least one non-transitory memory and program code may be configured to utilize the at least one processor to enable the device to further: generate feedback data based on the user input; retrieve a third plurality of data objects associated with the third database pattern; and determine a second data classifier corresponding to the third database pattern based at least on the third plurality of data objects and the feedback data.

[0015] According to various examples, a computer-implemented method may be provided. This computer-implemented method may include: retrieving from a database a first plurality of data objects associated with a first database schema; determining a first data classifier corresponding to the first database schema based at least on the first plurality of data objects; generating a mapping specification based at least in part on the first data classifier and the first plurality of data objects; and generating a second plurality of data objects based at least in part on the first plurality of data objects and the mapping specification. In some examples, the mapping specification may be configured to transform the first plurality of data objects associated with the first database schema into a second plurality of data objects associated with a second database schema.

[0016] According to various examples, a computer program product may be provided. The computer program product may include at least one non-transitory computer-readable storage medium in which a portion of computer-readable program code is stored. The computer-readable program code portion may include an executable portion configured to: retrieve a first plurality of data objects associated with a first database schema from a database; determine a first data classifier corresponding to the first database schema based at least on the first plurality of data objects; generate a mapping specification based at least in part on the first data classifier and the first plurality of data objects; and generate a second plurality of data objects based at least in part on the first plurality of data objects and the mapping specification. In some examples, the mapping specification may be configured to transform the first plurality of data objects associated with the first database schema into a second plurality of data objects associated with a second database schema.

[0017] The foregoing exemplary invention content, as well as other exemplary objects and / or advantages of this disclosure, and the ways of achieving these objects and / or advantages, are further explained in the following detailed description and accompanying drawings. Attached Figure Description

[0018] The description of the exemplary embodiments can be read in conjunction with the accompanying drawings. It should be understood that, for the sake of simplicity and clarity, unless otherwise stated, the elements shown in the drawings are not necessarily drawn to scale. For example, unless otherwise stated, the dimensions of some elements may be exaggerated relative to others. Embodiments incorporating the teachings of this disclosure are shown and described with reference to the accompanying drawings, in which:

[0019] Figure 1 Exemplary schematic diagrams of exemplary systems according to various embodiments of the present disclosure are shown;

[0020] Figure 2 An exemplary block diagram of an exemplary apparatus according to various embodiments of the present disclosure is shown;

[0021] Figure 3 Exemplary flowcharts of various embodiments according to this disclosure are shown;

[0022] Figure 4 Exemplary flowcharts of various embodiments according to this disclosure are shown;

[0023] Figure 5 Exemplary flowcharts of various embodiments according to this disclosure are shown;

[0024] Figure 6 Exemplary flowcharts of various embodiments according to this disclosure are shown; and

[0025] Figure 7 Exemplary flowcharts of various embodiments according to this disclosure are shown. Detailed Implementation

[0026] Some embodiments of this disclosure will be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, embodiments of this disclosure. In fact, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to enable this disclosure to meet applicable legal requirements. Throughout this document, similar reference numerals refer to similar elements.

[0027] The phrases “in one embodiment,” “according to one embodiment,” “for example,” “in some examples,” “as an example,” etc., generally mean that the specific feature, structure, or characteristic following the phrase can be included in at least one embodiment of this disclosure, and can be included in more than one embodiment of this disclosure (such phrases do not necessarily refer to the same embodiment).

[0028] The terms “example” or “exemplary” as used herein mean “serving as an example, instance, or illustration.” Any specific implementation described herein as an “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other specific implementations.

[0029] If the specification states that a component or feature “may,” “can,” “should,” “will,” “preferably,” “possibly,” “usually,” “optionally,” “for example,” “in some examples,” “often,” or “may” (or other such language) be included or have that characteristic, then the specific component or feature is not necessarily included or has that characteristic. Such components or features may be optionally included in some embodiments or excluded.

[0030] The term "circuit" should be broadly understood to include hardware, and in some embodiments, to include software for configuring the hardware. As used herein, the term "circuit" should therefore be understood to include specific hardware configured to perform functions associated with a particular circuit as described herein, relative to components of a device. For example, in some embodiments, "circuit" may include processing circuitry, storage media, network interfaces, input / output devices, etc.

[0031] The terms “electronically coupled,” “electronically coupling,” “electronically couple,” “communicating with,” “electronically communicating with,” or “connection” in this disclosure refer to two or more components (e.g., but not limited to, client devices, data analysis systems, databases) connected via wired means (e.g., but not limited to, wired Ethernet) and / or wireless means (e.g., but not limited to, Wi-Fi, Bluetooth, ZigBee) such that data and / or information can be transmitted to and / or received from these components.

[0032] The term "data analytics system" can refer to a system or virtual environment configured to generate a data classifier that indicates the meaning of data stored in a database and / or generate mapping specifications based on the data classifier. A data analytics system can take the form of one or more central servers configured to communicate with one or more additional servers running software applications and have access to one or more databases storing digital content items, application-related data, etc. This document at least incorporates... Figure 1 An exemplary data analysis system is described.

[0033] The term "user" should be understood as an individual, group of individuals, business, organization, etc. Users mentioned in this document can access the data analytics system using client devices. The term "client device" refers to computer hardware and / or software configured to access the data analytics system. Client devices may include, but are not limited to, smartphones, tablets, laptops, wearable devices, personal computers, enterprise computers, etc.

[0034] In some examples, data and information (such as electronic requests) may be transmitted to and / or received from a data analytics system. For example, a "data classification request" may indicate an electronic request for generating a data classifier that indicates the meaning of data stored in a database. In some examples, a data classification request may be transmitted from a client device to the data analytics system. As another example, a "user input request" may indicate an electronic request for providing user input. In some examples, a user input request may be sent from the data analytics system to a client device and may include an electronic request for confirming the data classifier, the details of which are described herein.

[0035] The term "data object" refers to a data structure that can represent one or more values ​​associated with data. Data objects can be stored in a database and may include one or more "data fields". In some examples, data fields may be in the form of American Standard Code for Information Interchange (ASCII) text, pointers, memory addresses, etc., and may include at least one value associated with the data object.

[0036] For example, a data object may include a "data table," which may represent values ​​in a tabular or semi-tabular form that may include rows and / or columns. In some examples, a data table may include at least one data field. Exemplary data fields in an exemplary data table are shown below:

[0037] (Name) (Column 1) (Column 2) (Row 1) Data Field 1 Data Field 2 (Row 2) Data Field 3 Data Field 4

[0038] In the example above, the data table may include two rows, two columns, and four data fields (such as Data Field 1, Data Field 2, Data Field 3, and Data Field 4).

[0039] In some examples, one or more data fields of a data table can be associated with row identifiers and / or column identifiers. Row identifiers can be in the form of ASCII text, pointers, memory addresses, etc., and can uniquely identify the row associated with a data field. Column identifiers can be in the form of ASCII text, pointers, memory addresses, etc., and can uniquely identify the column associated with a data field.

[0040] Continuing with the example above, Data Field 1 can be associated with Row identifier 1 and Column identifier 1. Data Field 2 can be associated with Row identifier 1 and Column identifier 2. Data Field 3 can be associated with Row identifier 2 and Column identifier 1. Data Field 4 can be associated with Row identifier 2 and Column identifier 2.

[0041] The term "metadata" refers to data that describes other data, such as data fields of a data object. In some examples, a data object may include one or more metadata and / or be associated with one or more metadata. In some examples, metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may include information associated with the meaning of values ​​in data fields.

[0042] In some examples, a data table may include at least one of name metadata, column metadata, and / or row metadata, as shown in the following examples:

[0043]

[0044]

[0045] In the example above, name metadata (such as Name Metadata 1), column metadata (such as Column Metadata 1 and Column Metadata 2), and row metadata (such as Row Metadata 1 and Row Metadata 2) can be associated with the meaning of data fields (such as Data Field 1, Data Field 2, Data Field 3, and Data Field 4) in the data table.

[0046] In some examples, name metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may include the name of the data table. In some examples, column metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may include column identifiers and / or column names. In some examples, row metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may include row identifiers and / or row names. In some examples, the data analysis system may implement machine learning models to determine the meaning of data fields based on at least one of the name metadata, column metadata, and / or row metadata, the details of which are described herein.

[0047] While the exemplary data described above illustrates exemplary name metadata, exemplary column metadata, and exemplary row metadata, it should be noted that the scope of this disclosure is not limited to these metadata. For example, a data analysis system may determine "relevance metadata" that indicates a relationship between two or more data objects. In some examples, a data analysis system may determine "domain metadata" that indicates a domain associated with a data object. Examples of relevance metadata and domain metadata are described herein.

[0048] The term "data object identifier" refers to an identifier that uniquely identifies and / or locates data objects from multiple data objects and / or from one or more databases. In some examples, data object identifiers may be in the form of ASCII text, memory addresses, network addresses, etc.

[0049] In some examples, data objects stored in a database may be associated with a database schema. As mentioned above, a "database schema" can refer to a database structure that defines how data is organized and / or related to each other within the database. In some examples, a database schema can be a physical implementation of a data model. The term "data model" can refer to an abstract model that organizes data objects and standardizes their relationships. Exemplary data models may include (but are not limited to) a Common Data Model (CDM), which is a shared data model that provides a standardized organization of data objects that will be shared between applications and / or data sources.

[0050] The term "database identifier" refers to an identifier that uniquely identifies and / or locates a database and / or database schema. In some examples, data object identifiers may be in the form of ASCII text, memory addresses, network addresses, etc.

[0051] The term "data classifier" can refer to data that indicates a classification, includes descriptions, and / or provides meaning related to one or more data objects. Data classifiers can be in the form of ASCII text, pointers, memory addresses, etc. For example, a data classifier can be in the form of a text string that may include data categories associated with one or more data fields in one or more data objects. Exemplary data categories may include, but are not limited to, user names, email addresses, battery level values, and measurement results. In some examples, data classifiers may be generated by a data analysis system, the details of which are described herein.

[0052] The term "mapping specification" can refer to data objects that describe and / or specify moves and / or transformations that determine how data associated with one data model can be represented based on another data model. For example, a mapping specification can be configured to transform a first set of data objects associated with a first database schema into a second set of data objects associated with a second database schema. In some examples, mapping specifications can be generated by a data analysis system, with example details described herein.

[0053] As mentioned above, many systems and methods have not overcome the technical challenges and difficulties associated with databases. For example, many systems lack the ability to analyze data stored in traditional databases that may not conform to the latest database models. In some examples, consuming data for analytical purposes can be costly when it may be unknown what insights can be generated from the data. For instance, data stored in traditional databases may include numeric strings, and it may be unknown what these numeric strings might represent. Therefore, understanding this data can be costly and time-consuming before it can be used for analysis. In some examples, understanding this data may not generate useful insights, potentially wasting computational resources.

[0054] In contrast, various examples according to this disclosure overcome these challenges and difficulties. In some examples, machine intelligence solutions (such as those using machine learning and other techniques) can infer the meaning of data in a database by examining data names, data types, and relationships between data. In some examples, when the meaning of data has sufficient confidence, the data can be automatically mapped into a data model for analytical purposes. In some examples, when the confidence in the meaning of data is insufficient, user input regarding the meaning of data can be provided, which can be used to improve the machine intelligence solution. Therefore, various examples of this disclosure can reduce the cost and computational resources required for data mapping and increase the quality of useful insights from data.

[0055] The methods, apparatus, and computer program products of this disclosure may be embodied in any of a variety of devices. For example, the methods, apparatus, and computer program products of the exemplary embodiments may be embodied in a networked device (e.g., a data analysis system) (such as a server or other network entity) configured to communicate with one or more devices (such as one or more client devices). Additionally or alternatively, the computing device may include a fixed computing device, such as a personal computer or a computer workstation. Additionally or alternatively, the exemplary embodiments may be embodied in any of a variety of mobile devices, such as a portable digital assistant (PDA), mobile phone, smartphone, laptop computer, tablet computer, wearable device, or any combination of the above devices.

[0056] Figure 1 An exemplary system architecture 100 in which embodiments of the present disclosure may operate is shown. Users can access the data analysis system 105 via a communication network 103 using client devices 101A, 101B, 101C, ... 101N.

[0057] Client devices 101A to 101N can be any computing device as defined above. Electronic data received by the data analysis system 105 from client devices 101A to 101N can be provided in various forms and via various methods. For example, client devices 101A to 101N may include desktop computers, laptop computers, smartphones, netbooks, tablets, wearable devices, etc. In some examples, one or more of client devices 101A to 101N may each be assigned a client device identifier that uniquely identifies the client device. In some examples, the client device identifier may include ASCII text, pointers, memory addresses, etc.

[0058] In embodiments where the client devices in client computing devices 101A to 101N are mobile devices (such as smartphones or tablets), the client devices can execute "applications" to interact with the data analytics system 105. These applications are typically designed to run on mobile devices (such as tablets or smartphones). For example, they may be available on mobile device operating systems (such as...) or Applications running on these platforms typically provide frameworks that allow applications to communicate with each other and with specific hardware and software components of the mobile device. For example, the mobile operating systems mentioned above each provide frameworks for interacting with location service circuits, wired and wireless network interfaces, user contacts, and other applications. Communication with hardware and software modules running outside of applications is typically provided via application programming interfaces (APIs) provided by the mobile device's operating system. In some examples, the application may provide a user interface that allows the user to interact with the data analytics system 105.

[0059] Alternatively or additionally, client devices 101A to 101N may interact with data analysis system 105 via a web browser. Alternatively or additionally, client devices 101A to 101N may include various hardware or firmware designed to interface with data analysis system 105.

[0060] The communication network 103 may include one or more wired or wireless communication networks, including, for example, wired or wireless local area networks (LANs), personal area networks (PANs), metropolitan area networks (MANs), wide area networks (WANs), etc., and any hardware, software, and / or firmware (such as network routers) required to implement the one or more networks. For example, the communication network 103 may include a General Packet Radio Service (GPRS) network, a Code Division Multiple Access 2000 (CDMA2000) network, a Wideband Code Division Multiple Access (WCDMA) network, a Global System for Mobile Communications (GSM) network, an Evolution of GSM Enhanced Data Rate (EDGE) network, a Time Division Synchronous Code Division Multiple Access (TD-SCDMA) network, a Long Term Evolution (LTE) network, a High-Speed ​​Packet Access (HSPA) network, a High-Speed ​​Downlink Packet Access (HSDPA) network, IEEE 802.11 (Wi-Fi), Wi-Fi Direct, IEEE 802.16 (WiMAX), etc. Additionally or alternatively, the communications network 103 may include a public network (such as the Internet), a private network (such as an intranet), or a combination thereof.

[0061] In some examples, the communication network 103 may utilize networking protocols, including but not limited to Hypertext Transfer Protocol (HTTP), HTTP / REST, networking protocols based on one or more Transmission Control Protocol / Internet Protocol (TCP / IP), Near Field Communication (NFC), Bluetooth, and / or ZigBee. For example, the networking protocol may be customized to suit the needs of the data analysis system 105. In some implementations, the protocol may be a customized protocol for sending JSON objects via a WebSocket channel. In some implementations, the protocol may be RPC-based JSON, REST / HTTP-based JSON, etc.

[0062] See again Figure 1 The data analysis system 105 may be embodied in the aforementioned computing device. For example, the data analysis system 105 may include at least one processor and at least one non-transitory memory storing computer program instructions. These computer program instructions may instruct the data analysis system 105 to function in a particular manner such that the instructions stored in the at least one non-transitory memory can produce an article of art, the execution of which can realize the embodiments of the present disclosure. Thus, in some examples of the present disclosure, the data analysis system 105 may include a database connector 107, a data interpreter 109, and / or a data mapper 111.

[0063] Database connector 107, data interpreter 109, and / or data mapper 111 may be embodied in hardware devices (such as one or more circuits), software devices (such as computer program code), or a combination of hardware and software devices. In some examples, database connector 107 may be configured to access data from one or more databases (including, but not limited to, databases such as…). Figure 1 The databases 113A to 113N shown retrieve one or more data objects. In some examples, the data interpreter 109 may be configured to generate one or more data classifiers. In some examples, the data mapper 111 may be configured to generate one or more mapping specifications. Combined with at least... Figure 2 Example diagrams showing and describing various exemplary components of the data analysis system 105 are shown and described.

[0064] However, it should be noted that the various components in the data analysis system 105 can operate using the same computer or computing device according to the examples of this disclosure. For example, the database connector 107, data interpreter 109, and / or data mapper 111 can utilize the same processor or memory to perform these functions. In some examples, the database connector 107, data interpreter 109, and / or data mapper 111 can utilize separate circuitry.

[0065] In various embodiments of this disclosure, one or more electronic requests may be sent to the data analysis system 105, including but not limited to data classification requests and / or data mapping requests. In some examples, these electronic requests may be in the form of HTTP requests. In some examples, these electronic requests may be sent directly to the data analysis system 105 by client devices 101A to 101N via the communication network 103. Additionally or alternatively, these electronic requests may be sent to the data analysis system 105 via an intermediary.

[0066] In some examples, upon receiving a data classification request, the data analysis system 105 can generate one or more data classifiers. In some examples, based on these one or more data classifiers, the data analysis system 105 can generate one or more mapping specifications.

[0067] See again Figure 1 Databases 113A to 113N may be embodied as one or more data storage devices, such as one or more network attached storage (NAS) devices, or as one or more individual servers. Databases 113A to 113N may include information accessible to data and / or data analysis system 105 and / or client devices 101A to 101N.

[0068] In some examples, databases 113A to 113N may store data, such as, but not limited to, one or more data objects. In some examples, upon receiving a data classification request, data analysis system 105 may transmit an electronic request to databases 113A to 113N to retrieve or obtain one or more data objects from databases 113A to 113N. In some examples, data analysis system 105 may store one or more data objects in databases 113A to 113N.

[0069] However, it should be noted that databases 113A to 113N can perform the above operations using the same computer or computing device. For example, databases 113A to 113N can be integrated into data analysis system 105, making databases 113A to 113N a part of data analysis system 105. In some examples, databases 113A to 113N and data analysis system 105 can utilize separate circuits.

[0070] It can be represented by one or more computing systems. Figure 1 Data analysis system 105, such as Figure 2 The device 200 is shown. Device 200 may include a processor 202, a memory 204, input / output circuitry 206, and / or communication circuitry 208. Device 200 may be configured to perform the above-described... Figure 1 See below Figures 3 to 7 The aforementioned operation.

[0071] While these components are described with respect to functional limitations, it should be understood that a particular implementation will necessarily involve the use of specific hardware. It should also be understood that some of these components may include similar or common hardware. For example, both sets of circuits may use the same processor, network interface, storage medium, etc., to perform their associated functions, so that each set of circuits does not require duplicate hardware.

[0072] In some embodiments, processor 202 (and / or coprocessor or any other processing circuitry assisting or otherwise associated with the processor) may communicate with memory 204 via a bus for transferring information between components of the device. Memory 204 is non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, memory 204 may be an electronic storage device (e.g., a computer-readable storage medium). Memory 204 may be configured to store information, data, content, applications, instructions, etc., for enabling the device to perform various functions according to exemplary embodiments of the invention.

[0073] In such Figure 2In the example shown, memory 204 may store computer program instructions, which may include database connector module 210, data interpreter module 212, and / or data mapper module 214. When database connector module 210 is executed by processor 202, device 200 may be configured to retrieve data from one or more databases (such as, but not limited to, those described above). Figure 1 The databases 113A to 113N retrieve one or more data objects. When the data interpreter module 212 is executed by the processor 202, the device 200 can be configured to generate one or more data classifiers. When the data mapper module 214 is executed by the processor 202, the device 200 can be configured to generate one or more mapping specifications.

[0074] Additionally or alternatively, apparatus 200 may include one or more designated hardware components configured for database connector module 210, data interpreter module 212, and / or data mapper module 214. For example, apparatus 200 may include a separate processor, a specially configured field-programmable gate array (FPGA), or application-specific integrated circuit (ASIC) configured to perform the functions of database connector module 210, data interpreter module 212, and / or data mapper module 214.

[0075] See back Figure 2 The processor 202 can be embodied in a variety of different ways and may include, for example, one or more processing devices configured to execute independently. In some examples, the processor 202 may include one or more processors configured in series via a bus to enable independent execution of instructions, pipelines, and / or multithreading. The use of the terms "processor" or "processing circuitry" can be understood to include a single-core processor, a multi-core processor, multiple processors within a device, and / or a remote or "cloud" processor.

[0076] As described above, processor 202 can be configured to execute instructions stored in memory 204 or otherwise accessible to processor 202. In some preferred and non-limiting embodiments, processor 202 can be configured to perform hard-coded functions. Thus, whether configured by hardware or software methods, or by a combination thereof, processor 202 can represent an entity capable of performing operations and being configured accordingly according to embodiments of this disclosure (e.g., physically embodied in circuit form). Alternatively, for example, when processor 202 embodies an executor of software instructions, these instructions can specifically configure processor 202 to perform the algorithms and / or operations described herein when executing these instructions.

[0077] Communication circuit 208 can be any device, such as a device or circuit embodied in hardware or a combination of hardware and software, configured to communicate with and / or to a network and / or with device 200, or any other device, circuit, or module (such as those described above). Figure 1 The clients 101A to 101N and / or databases 113A to 113N receive and / or transmit data. In this regard, the communication circuitry 208 may include, for example, methods for enabling communication with wired or wireless communication networks (such as those described above). Figure 1 The communication network 208 is the network interface for communication in the communication network 103. For example, the communication circuit 208 may include one or more network interface cards, antennas, buses, switches, routers, modems, and supporting hardware and / or software, or any other device suitable for enabling communication via the network. Additionally or alternatively, the communication circuit 208 may include circuitry for interacting with the antenna to transmit signals via the antenna or process signals received via the antenna.

[0078] In some examples, device 200 may optionally include input / output circuitry 206, which can then communicate with processor 202 to provide output to a user, and in some embodiments, receive indications of user input. Input / output circuitry 206 may include user interface circuitry and may include a display, which may include a web user interface, mobile application, client device, kiosk, etc. In some embodiments, input / output circuitry 206 may also include a keyboard, mouse, joystick, touchscreen, touch area, softkeys, microphone, speaker, or other input / output mechanism. The processor and / or the user interface circuitry including the processor may be configured to control one or more functions of one or more user interface elements via computer program instructions (e.g., software and / or firmware) stored in processor-accessible memory (e.g., memory 204, etc.).

[0079] It should also be noted that all or some of the information discussed herein may be based on data received, generated, and / or maintained by one or more components of device 200. In some embodiments, one or more external systems, such as remote cloud computing and / or data storage systems, may also be utilized to provide at least some of the functions discussed herein.

[0080] In some embodiments, other elements of device 200 may provide or supplement the functionality of a particular circuit. For example, processor 202 may provide processing functionality, memory 204 may provide storage functionality, and communication circuitry 208 may provide network interface functionality, etc. It should be understood that any such computer program instructions and / or other types of code may be loaded onto the circuitry of a computer, processor, or other programmable device to produce a machine on which the computer, processor, or other programmable circuitry executing the code may form means for implementing a variety of functions, including those described herein.

[0081] As described above and based on this disclosure, embodiments of this disclosure can be configured as methods, mobile devices, backend network devices, etc. Therefore, embodiments can include various means, including entirely hardware or any combination of software and hardware.

[0082] Now see Figures 3 to 7 Exemplary methods according to various embodiments of the present disclosure are illustrated. In some examples, each block or step in the flowchart, as well as combinations of blocks and / or steps in the flowchart, may be implemented by various means, such as hardware, circuitry, and / or other means associated with the execution of software including one or more computer program instructions.

[0083] In some examples, one or more programs in the program described in the figures may be embodied by computer program instructions, which may be stored by memory circuitry (such as non-transitory memory) of a device employing embodiments of the present disclosure and executed by processing circuitry (such as a processor) of the device. These computer program instructions may instruct the device to operate in a particular manner such that the instructions stored in the memory circuitry produce an article of art, the execution of which performs the function specified in the flowchart block. Furthermore, the device may include one or more other components, such as, for example, communication circuitry and / or input / output circuitry. The various components of the device may communicate electronically with each other to transmit data to and / or receive data from each other.

[0084] In some examples, the implementation may take the form of a computer program product on a non-transitory computer-readable storage medium that stores computer-readable program instructions (e.g., computer software). Any suitable computer-readable storage medium may be used, including non-transitory hard disks, CD-ROMs, flash memory, optical storage devices, and / or magnetic storage devices.

[0085] Now see Figure 3 The illustration shows an exemplary method 300 according to some embodiments of the present disclosure. Specifically, exemplary method 300 may illustrate an exemplary embodiment of classifying data objects and generating a mapping specification for transforming data objects. In some examples, method 300 may be comprised of processing circuitry (e.g., in conjunction with...). Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 is executed.

[0086] Method 300 begins at box 301.

[0087] At block 303, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can retrieve a first plurality of data objects. In some examples, the first plurality of data objects may be associated with a first database schema.

[0088] In some examples, the processing circuitry can originate from a client device (such as, but not limited to, ... Figure 1 One of the client devices 101A to 101N shown receives a data classification request. The data classification request may include one or more data object identifiers and / or a request to generate a data classifier.

[0089] As described above, data object identifiers can identify and / or locate data objects from one or more databases (such as, but not limited to, databases such as, etc.). Figure 1 The data objects in databases 113A to 113N are shown. Upon receiving a data classification request, the processing circuit can transmit a data retrieval request (which may include, for example, one or more data object identifiers) to one or more databases (such as, but not limited to, databases 113A to 113N). Figure 1 Databases 113A to 113N are shown. The processing circuitry can receive one or more data objects associated with data object identifiers from the one or more databases in response to a data retrieval request.

[0090] In some examples, the processing circuitry may include database connector components (e.g., but not limited to) Figure 1 The database connector 107 shown is an example. The database connector component can obtain data object and / or database schema information from one or more databases, and can transmit the data object and / or database schema information to the data interpreter component (e.g., but not limited to...). Figure 1 The data interpreter 109 shown.

[0091] At block 305, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can determine a first data classifier corresponding to a first database pattern. In some examples, the processing circuitry can determine the first data classifier based at least on a first plurality of data objects retrieved at block 303.

[0092] In some examples, the processing circuitry may include data interpreter components (e.g., but not limited to) Figure 1 The data interpreter 109 shown is available from the database connector component (e.g., but not limited to, such as...). Figure 1 The database connector 107 shown receives data.

[0093] As mentioned above, a data classifier can be in the form of ASCII text, pointers, memory addresses, etc., and can indicate the meaning of a classification, including descriptions and / or providing information associated with one or more data objects. In some examples, the processing circuitry may apply machine learning models, intelligent agents (IA), and / or artificial intelligence (AI) tools to determine the data classifier.

[0094] In some examples, the processing circuitry can implement an artificial neural network to determine a first data classifier. An exemplary artificial neural network may include multiple interconnected nodes, and each node may represent a mathematical function that generates an output (to the node) based on inputs received from the node. These multiple nodes may be divided into layers, such as an input layer, one or more intermediate layers, and an output layer.

[0095] For example, the following data object (in the form of a data table) can be provided as input by the processing circuit to an exemplary artificial neural network:

[0096] John Doe Jr. John.doe@email.com Adam Davis Adam.davis@email.com Richard Roe II Richard.roe@email.com

[0097] In this example, the artificial neural network can generate nodes for each data field in the data table (e.g., "John", "Doe", "Jr.", "John.doe@email.com"). By interconnecting the nodes and their associated mathematical functions, the artificial neural network can output one or more data classifiers that indicate the classification of each row and / or column of the data table. For example, the artificial neural network can generate a data classifier for the data table (which may be, for example, in the form of a text string). In some examples, the data classifier may indicate that the data field associated with the first column is the first name, the data field associated with the second column is the last name, the data field associated with the third column is the first name suffix, and / or the data field associated with the fourth column is the email address.

[0098] In some examples, the processing circuitry may implement a decision tree algorithm to determine a first data classifier. An exemplary decision tree may include one or more leaves, and each leaf may represent, for example, a possible classification of the data. Additionally or alternatively, the decision tree may include one or more branches, which may represent, for example, possible combinations of classifications (i.e., leaves on the decision tree).

[0099] For example, a decision tree algorithm can determine one or more categories for each column of a data table. Continuing from the exemplary data table described above, the processing circuitry can provide the data fields associated with the first column as input to the exemplary decision tree algorithm. The exemplary decision tree algorithm can determine that these data fields represent names (e.g., the "Name" category as the top node in the decision tree), and can further determine whether these data fields represent first names or last names (e.g., the "First Name" subcategory and the "Last Name" subcategory as children of the top node in the decision tree). Based on the data fields associated with the first column, the decision tree algorithm can calculate a first probability that these data fields represent first names, and a second probability that these data fields represent last names. The decision tree algorithm can compare the first probability with the second probability and can determine that the first probability is higher than the second probability. In this example, the decision tree algorithm can generate an output (which may indicate that the first column represents a name), and the processing circuitry can determine a data classifier based on the output from the decision tree algorithm.

[0100] In some examples, the processing circuitry can implement a supervised learning model to determine a first data classifier. In an exemplary supervised learning model, inputs received by the model can be mapped to outputs based on exemplary input-output pairs (e.g., training data). The exemplary supervised learning model can analyze the training data and infer one or more functions based on the exemplary input-output pairs. The exemplary supervised learning model can utilize the inferred functions to generate one or more outputs. The exemplary supervised learning model may include, but is not limited to, support vector machines.

[0101] Continuing from the exemplary data table above, exemplary name suffixes can be provided as training data to the exemplary supervised learning model. For example, the processing circuitry can provide the example supervised learning model with the following exemplary name suffixes: II, III, IV, Jr., Sr., MD, PhD. The supervised learning model can associate each exemplary name suffix with a name suffix classification. When the supervised learning model receives the data field "Jr." from the exemplary data table above, the supervised learning model can generate an output indicating the association between the data field and the name suffix, and the processing circuitry can generate a data classifier based on this output.

[0102] While the examples above illustrate the generation of exemplary data classifiers based on artificial neural networks, decision tree algorithms, and / or supervised learning models, it should be noted that the scope of this disclosure is not limited to these mechanisms. In some examples, additionally or alternatively, other machine learning models, IA, and / or AI tools may be utilized to determine the data classifier, including but not limited to Bayesian networks, genetic algorithms, regression models, and / or random forests.

[0103] In some examples, in addition to or as a substitute for the data fields of the data objects, the processing circuitry may determine a first data classifier based on metadata associated with the data objects. For example, the processing circuitry may determine the first data classifier based at least in part on name metadata, column metadata, row metadata, relevance metadata, and / or domain metadata associated with a first plurality of data objects, exemplary details of which are combined with at least Figure 4 , Figure 5 and Figure 6 To describe.

[0104] At block 307, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can generate a mapping specification. In some examples, the mapping specification can be configured to transform a first plurality of data objects associated with a first database schema into a second plurality of data objects associated with a second database schema.

[0105] In some examples, the processing circuitry can originate from a client device (such as, but not limited to, ... Figure 1 One of the client devices 101A to 101N shown receives a data mapping request. The data mapping request may include: a first database identifier associated with a first database schema, a second database identifier associated with a second database schema, and a request to convert a data object from the first database schema to the second database schema.

[0106] As described above, database identifiers can identify and / or locate databases (such as, but not limited to, databases). Figure 1 The databases 113A to 113N shown are associated with database schemas. Upon receiving a data mapping request, the processing circuitry may, for example, retrieve data objects associated with the first database schema and determine a data classifier, similar to those described above in conjunction with boxes 303 and 305. In some examples, the data mapping request may be combined with a data classification request, allowing the client device to transmit an electronic request to trigger the processing circuitry to generate a data classifier and mapping specification.

[0107] In some examples, the processing circuitry may generate a mapping specification at least in part based on a first plurality of data objects (retrieved at box 303) and a first data classifier (determined at box 305). As described above, the term "mapping specification" may refer to data objects that describe and / or specify moves and / or transformations that determine how data associated with a data model can be represented based on another data model.

[0108] Continuing with an example related to the following data table:

[0109]

[0110] The exemplary data classifier generated at box 305 may indicate that the data field associated with Column 1 is a first name, the data field associated with Column 2 is a last name, the data field associated with Column 3 is a first name suffix, and / or the data field associated with Column 4 is an email address. For example, a data mapping request received by the processing circuit may indicate a request to transform data from the aforementioned source data tables into different types of data tables and / or data objects based on different database schemas. For example, the data mapping request may include a request to transform the aforementioned source data tables into target data objects conforming to a common data model.

[0111] As described above, the processing circuitry can generate a mapping specification at least in part based on a data classifier (e.g., the data classifier determined at box 305). Continuing from the example above, the data classifier can indicate that the data field associated with Column 4 of the source data table is an email address. The processing circuitry can determine that the target data object (e.g., a data table based on a common data model) may include a data field for email addresses. The processing circuitry can generate rule statements in the mapping specification to transform Column 1 of the source data table into the corresponding data field for email addresses in the target data object.

[0112] In some examples, the processing circuitry can generate one or more rule statements in the mapping specification to combine one or more data fields based on a data classifier. In the example above, the data classifier can indicate that Column 1, Column 2, and Column 3 are associated with a name. Based on the data classifier, the processing circuitry can combine the data fields of Column 1, Column 2, and Column 3 for each row and provide these data fields to the corresponding data fields for the name in the target data table.

[0113] In some examples, the processing circuitry may generate one or more rule statements within the mapping specification to split a data field into multiple data fields or copy it into multiple data fields based on the data classifier. For instance, if the data classifier indicates that a data field includes dates in YYY-MM-DD format, and the processing circuitry determines that the target data table includes separate columns for year, month, and day, the processing circuitry may generate one or more rule statements to split the data field, thereby separating the year, month, and day values.

[0114] While the examples above illustrate exemplary mapping specifications associated with transforming and converting data fields between data tables, it should be noted that the scope of this disclosure is not limited to data tables. In some examples, processing circuitry may generate mapping specifications based on other types of data objects.

[0115] For example, the plurality of data objects may include a text document, which may include one or more ASCII characters. As described in conjunction with box 305 above, the processing circuitry may generate a first data classifier based on the text document. For example, the processing circuitry may implement a machine learning model to perform natural language processing on the text document. Based on the results of the natural language processing, the processing circuitry may generate a data classifier that indicates the meaning of the text document.

[0116] As a non-limiting example, a data classifier may indicate that a text document describes the battery level values ​​of one or more devices. Based at least in part on the data classifier, processing circuitry may generate a mapping specification that can be configured to transform the battery level values ​​in the source text document into data fields in a target data object.

[0117] In some examples, the processing circuitry may implement machine learning models, IA (Integrated Automation), and / or AI tools to generate mapping specifications. For instance, the processing circuitry may implement an artificial neural network to determine the most efficient mapping specification. In this example, the processing circuitry may generate nodes in an exemplary artificial neural network based on data operations specified in a rule statement. The processing circuitry may then compute the path in the exemplary artificial neural network that may require the fewest operations to determine the most efficient mapping specification. In some examples, the processing circuitry may utilize other machine learning models, IA, and / or AI tools to generate the mapping specification.

[0118] At box 309, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can generate a second plurality of data objects.

[0119] In some examples, the processing circuitry may generate a second set of data objects based at least in part on a first set of data objects and a mapping specification. For example, the processing circuitry may transform data fields in the source data objects into data fields of the target data objects based on rule statements of the mapping specification.

[0120] Continuing from the exemplary source data table related to names and email addresses described above, the processing circuitry can transform the source data table into a target data table based on rule statements in the mapping specification. As mentioned above, the rule statements can be generated based on a data classifier. In this example, based on the rule statements, the processing circuitry can generate combined data fields associated with Column 1, Column 2, and Column 3 for each row from the source data table (based on, for example, row identifiers), and populate the combined data fields into multiple first data fields of the target data object. Additionally or alternatively, the processing circuitry can transform the data field of Column 4 from the source data table into multiple second data fields of the target data object.

[0121] While the examples above represent exemplary data as exemplary data objects, it should be noted that the scope of this disclosure is not limited to data tables. In some examples, multiple data objects may include logs, numeric strings, etc.

[0122] Method 300 ends at box 311.

[0123] Now see Figure 4 , Figure 5 and Figure 6 Exemplary methods according to some embodiments of this disclosure are shown. Specifically, these exemplary methods may illustrate determining a data classifier (which may be used, for example, as described above). Figure 3 An exemplary implementation (related to box 305).

[0124] Now see Figure 4 The present disclosure illustrates an exemplary method 400 according to some embodiments thereof. Specifically, exemplary method 400 may illustrate an exemplary embodiment of determining a data classifier based at least on metadata associated with data objects in a database.

[0125] In some examples, method 400 may be handled by processing circuitry (e.g., in conjunction with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 is executed.

[0126] Method 400 can begin with box A. See again. Figure 3 Box A can be used after retrieving the first plurality of data objects (box 303).

[0127] At block 402, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the data analysis system 105 Figure 2 The processor 202 of the device 200 can retrieve data from multiple data objects (e.g., in conjunction with the above). Figure 3Metadata associated with the first plurality of data objects.

[0128] In such Figure 4 In the example shown, the metadata retrieved by the processing circuitry may include at least one of name metadata 404, column metadata 406, or row metadata 408 associated with a first plurality of data objects. For example, name metadata, column metadata, and / or row metadata may be associated with a data table.

[0129] In some examples, the name metadata 404 may be in the form of ASCII text, pointers, memory addresses, etc., and may include the name of a data object (such as a data table). For example, the name metadata 404 may indicate that a data table is associated with consumer information.

[0130] In some examples, column metadata 406 may be in the form of ASCII text, pointers, memory addresses, etc., and may include column identifiers and / or column names. For example, column metadata 406 may indicate that the data field associated with the column is related to a consumer's name.

[0131] In some examples, row metadata 408 may be in the form of ASCII text, pointers, memory addresses, etc., and may include a row identifier and / or the name of the row. For example, row metadata 408 may indicate that the data field associated with the row is related to a consumer identifier.

[0132] At block 410, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can determine a first data classifier based on metadata. For example, the processing circuitry can determine the first data classifier based on at least one of name metadata 404, column metadata 406, and / or row metadata 408.

[0133] As described above, the processing circuitry can apply machine learning models, IA, and / or AI tools to determine a data classifier. In some examples, the metadata retrieved at box 402 can be provided as input data to the machine learning model, IA, and / or AI tool. For example, name metadata 404 can indicate that a data table is associated with consumer information, column metadata 406 can indicate that a data field associated with a column is associated with a consumer's name, and / or row metadata 408 can indicate that a data field associated with a row is associated with a consumer identifier. The processing circuitry can implement machine learning models, IA, and / or AI tools to determine a first data classifier based at least on at least one of name metadata 404, column metadata 406, and / or row metadata 408, similar to the combination described above. Figure 4 Those mentioned above.

[0134] After box 410, method 400 can return to box B. For example... Figure 3 As shown, box B can be used before generating the mapping specification at box 307. In some examples, the processing circuitry can be at least partially based on the combination Figure 4 The data classifier is used to generate the mapping specification.

[0135] Now see Figure 5 The present disclosure illustrates an exemplary method 500 according to some embodiments thereof. Specifically, exemplary method 500 may illustrate an exemplary embodiment of determining a data classifier based at least on relevance metadata associated with data objects in a database.

[0136] In some examples, method 500 may be handled by processing circuitry (e.g., in conjunction with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 is executed.

[0137] Method 500 can begin with box A. See again. Figure 3 Box A can be used after retrieving the first plurality of data objects (box 303).

[0138] At block 501, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can determine relevant metadata. In some examples, the relevant metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may indicate the relationship between two or more data objects.

[0139] In some examples, relevance metadata may indicate the relationship between a first data table and a second data table. For example, relevance metadata may indicate that both the first and second data tables are associated with consumer information. Additionally or alternatively, relevance metadata may indicate that the first data table may include a data field associated with a consumer's name, and the second data table may include a data field associated with a consumer's email address.

[0140] In some examples, the processing circuitry can implement machine learning models, information automation (IA) tools, and / or AI tools to determine relevance in the data. For instance, the processing circuitry can implement a supervised learning model to determine the relationship between a first data table and a second data table.

[0141] At block 503, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can determine a first data classifier based on relevant metadata.

[0142] As described above, the processing circuitry can apply machine learning models, IA, and / or AI tools to determine a data classifier. In some examples, the relevance metadata determined at box 501 can be provided as input data by the processing circuitry to the machine learning model, IA, and / or AI tool. The processing circuitry can enable the machine learning model, IA, and / or AI tool to determine a first data classifier based at least in part on the relevance metadata, similar to the combination described above. Figure 4 Those mentioned above.

[0143] After box 503, method 500 can return to box B. For example... Figure 3 As shown, box B can be used before generating the mapping specification at box 307. In some examples, the processing circuitry can be at least partially based on the combination Figure 5 The data classifier is used to generate the mapping specification.

[0144] Now see Figure 6 The present disclosure illustrates an exemplary method 600 according to some embodiments thereof. Specifically, exemplary method 600 may illustrate an exemplary embodiment for determining a data classifier based at least on domain metadata.

[0145] In some examples, method 600 may be handled by processing circuitry (e.g., in conjunction with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 is executed.

[0146] Method 600 can begin with box A. See again. Figure 3 Box A can be used after retrieving the first plurality of data objects (box 303).

[0147] At block 602, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can determine domain metadata. In some examples, the relevant metadata may be in the form of ASCII text, pointers, memory addresses, etc., and may indicate the subject domain associated with a data object (such as, but not limited to, a first data table).

[0148] The term "domain" or "subject domain" can refer to a set of common attributes and / or functions among multiple data objects. In some examples, data objects associated with the same domain may represent a scope of knowledge or activities associated with a common entity. For example, data objects representing names, email addresses, and phone numbers could be categorized under the consumer information domain.

[0149] In some examples, the processing circuitry may implement machine learning models, IA (Information Technology), and / or AI tools to determine domain data. For instance, the processing circuitry may implement an artificial neural network to determine the corresponding domain associated with a first data table.

[0150] At block 604, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can determine a first data classifier based on domain metadata.

[0151] As described above, the processing circuitry can apply machine learning models, IA, and / or AI tools to determine a data classifier. In some examples, the domain metadata determined at box 602 can be provided as input data by the processing circuitry to the machine learning model, IA, and / or AI tool. The processing circuitry can enable the machine learning model, IA, and / or AI tool to determine a first data classifier based at least in part on the domain metadata, similar to the combination described above. Figure 4 Those mentioned above.

[0152] After box 604, method 600 can return to box B. For example... Figure 3 As shown, box B can be used before generating the mapping specification at box 307. In some examples, the processing circuitry can be at least partially based on the combination Figure 6 The data classifier is used to generate the mapping specification.

[0153] Although Figure 4 , Figure 5 and Figure 6 Exemplary methods for determining a data classifier based on name metadata, column metadata, row metadata, relevance metadata, and / or domain metadata are illustrated, but it should be noted that the scope of this disclosure is not limited to these metadata. In some examples, other metadata may be used in addition to or as an alternative to the metadata described above to determine the data classifier.

[0154] Now see Figure 7 The illustration shows an exemplary method 700 according to some embodiments of the present disclosure. Specifically, the exemplary method 700 may illustrate an exemplary embodiment of classifying data objects and generating a mapping specification for transforming the data objects. In some examples, method 700 may be comprised of processing circuitry (e.g., in conjunction with...). Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 is executed.

[0155] Method 700 begins at box 701.

[0156] At block 703, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can determine a first data classifier.

[0157] In some examples, the first data classifier may be associated with a first plurality of data objects. In some examples, the processing circuitry may be based on a combination similar to... Figure 3 , Figure 4 , Figure 5 and / or Figure 6 The method described above is used to generate the first data classifier.

[0158] At box 705, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can calculate a confidence score associated with the first data classifier.

[0159] A "confidence score" can refer to a mathematical value that indicates the probability that the corresponding data is correct and / or represents the true state. For example, a confidence score associated with a data classifier indicates the probability that the data classifier includes the correct classification and / or correct description of the data objects.

[0160] As mentioned above Figure 3 , Figure 4 , Figure 5 and Figure 6 The processing circuitry may determine a first classifier based on, for example, a machine learning model, an IA (Information Technology) and / or an AI (Artificial Intelligence) tool. In some examples, the processing circuitry may use the same or different machine learning model, IA, and / or AI tool as used to determine the data classifier to generate confidence scores.

[0161] For example, when processing circuitry uses a supervised learning model to determine a first data classifier, it compares the data fields of a data object with input-output pairs in the training data. The processing circuitry can calculate a proximity value, which indicates the level of similarity between the data fields and the training data. The higher the similarity, the greater the likelihood that the supervised learning model will generate a correct classification of the data object. Therefore, the processing circuitry can generate a confidence score based on the proximity value.

[0162] Additionally or alternatively, the processing circuitry may utilize other machine learning models, IA, and / or AI tools to calculate confidence scores.

[0163] At block 707, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2The processor 202 of the described device 200 can determine whether the confidence score meets a predetermined threshold.

[0164] In some examples, the processing circuitry may determine a predetermined threshold based on, for example, system requirements. For instance, exemplary system requirements may indicate the required level of precision for the data mapping. In this example, the higher the required level of precision, the higher the value of the predetermined threshold.

[0165] In some examples, the predetermined threshold may be set by the user. For instance, a user (while operating a client device) may provide user input to the processing circuitry. The user input may include values ​​that correspond to the predetermined threshold.

[0166] See again Figure 7 If the processing circuit determines at block 707 that the confidence score meets a predetermined threshold, then method 700 may proceed to block 709.

[0167] For example, if the confidence score calculated at box 705 is 0.8 and the predetermined threshold is 0.6, the processing circuit can determine that the confidence score is higher than the threshold, and therefore the confidence score meets the predetermined threshold.

[0168] At box 709, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can generate a mapping specification in response to determining that the confidence score meets a predetermined threshold.

[0169] In some examples, the processing circuitry may generate the mapping specification at least in part based on the first data classifier, similar to the combination above. Figure 3 Those mentioned above.

[0170] See again Figure 7 If the processing circuit determines at block 707 that the confidence score does not meet the threshold, then method 700 may proceed to block 711.

[0171] For example, if the confidence score calculated at box 705 is 0.7 and the predetermined threshold is 0.9, the processing circuit can determine that the confidence score is below the threshold and therefore the confidence score does not meet the predetermined threshold.

[0172] At block 711, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described apparatus 200 can generate user input requests and can transmit user input requests to client devices (such as those described above). Figure 1The client devices 101A to 101N are mentioned.

[0173] In some examples, the user input request may be associated with a first data classifier. For instance, the user input request may include an electronic request to confirm whether the first data classifier is correct.

[0174] In some examples, user input requests may be transmitted to a client device associated with an expert. This client device may present the user input request for display, which may include a sample of data objects and a data classifier determined at box 703. For example, the client device may display a data table and a data classifier indicating that the data table is identified as associated with consumer information. The expert can then select whether the classification is correct or incorrect.

[0175] At block 713, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can receive user input. This user input can respond to a user input request at box 711.

[0176] In some examples, user input may include indications from the user regarding whether the data classifier determined at box 703 is correct. For example, user input may include confirmation of the data classifier, which may indicate, for example, expert confirmation that the data classifier represents the correct classification of the data objects. Alternatively, user input may include modifications to the data classifier, which may indicate, for example, expert determination that the data classifier determined at box 703 is incorrect or inaccurate.

[0177] At box 715, the processing circuitry (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can determine whether the user input received at block 713 indicates that the first data classifier determined at block 703 is correct.

[0178] At block 715, if the processing circuitry determines that the user input confirms the data classifier is correct, method 700 may proceed to block 709. For example, if the user input includes confirmation of the first data classifier, at block 709, the processing circuitry may generate a mapping specification in response to the confirmation of the first data classifier.

[0179] At block 715, if the processing circuitry determines that the user input confirmation data classifier is incorrect, method 700 may proceed to block 719. At block 719, the processing circuitry (e.g., in conjunction with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2The processor 202 of the described device 200 can modify the first data classifier based on user input.

[0180] In some examples, user input may include modifications to the first data classifier. For example, if the first data classifier determined at box 703 indicates that a data object is identified as associated with “first name”, then user input may indicate that the data object is associated with “last name”.

[0181] In some examples, the processing circuitry can modify the first data classifier based on user input. Continuing with the previous example, the processing circuitry can change the data classifier from indicating "first name" to indicating "last name" based on user input.

[0182] After modifying the first data classifier at box 719, the processing circuitry can proceed to box 709. At box 709, the processing circuitry can generate a mapping specification based on the modified first data classifier.

[0183] See again Figure 7 At box 717, the processing circuit (e.g., combined with...) Figure 1 The processing circuitry and / or combination of the described data analysis system 105 Figure 2 The processor 202 of the described device 200 can generate feedback data based on user input.

[0184] In some examples, feedback data can be provided to machine learning models, IA, and / or AI tools to improve the accuracy of the generated data classifier. For example, processing circuitry can retrieve multiple additional data objects associated with the same or different database schemas (compared to the database schemas of a first set of multiple data objects) and can generate a data classifier based on the data objects and the feedback data.

[0185] For example, when the processing circuitry uses a supervised learning model to determine a first data classifier, it can generate feedback data in the form of input-output pairs (i.e., training data for the supervised learning model) based on user input. For instance, when user input indicates that a data object is associated with a surname, the processing circuitry can generate a pair of inputs (data object) and outputs (surname classification). The processing circuitry can provide this pair of inputs and outputs to train the supervised learning model, enabling the model to improve the accuracy of determining data classifiers for other data objects similar to the data objects in this input-output pair.

[0186] Method 700 ends at box 721.

[0187] It should be understood that this disclosure is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terminology is used herein, it is used in a general and descriptive sense only, and not for purposes of limitation, unless otherwise stated.

Claims

1. An apparatus comprising at least one processor and at least one non-transitory memory, the at least one non-transitory memory including program code, wherein the at least one non-transitory memory and the program code are configured to utilize the at least one processor such that the apparatus at least: Receive a request to execute a data mapping process for the first database schema; In response to the request, Retrieve from the database a first plurality of data objects for a first data model associated with the first database schema; A first data classifier corresponding to the first database pattern is determined by using a first machine learning model that processes the first plurality of data objects and relevance metadata as input, wherein the relevance metadata indicates the relationship between the first database pattern and at least one other database pattern in the database. Calculate the confidence score associated with the first data classifier; and In response to determining that the confidence score meets a predetermined threshold, Generate a mapping specification, including a set of rules for data operations associated with the mapping specification, the set of rules being at least partially based on the first data classifier and metadata associated with the first plurality of data objects, wherein the mapping specification includes one or more rule statements specifying the movement and / or transformation of the representation of a second plurality of data objects associated with a second database schema based on the first plurality of data objects associated with the first database schema; Based on the set of rules associated with the mapping specification, the first plurality of data objects associated with the first database schema are transformed into a second plurality of data objects for a second data model associated with the second database schema; and Provide one or more insights about the second plurality of data objects.

2. The apparatus of claim 1, wherein the first plurality of data objects includes a first data table, wherein the first data table includes at least one data field.

3. The apparatus of claim 2, wherein the first data table includes at least one of name metadata, column metadata, or row metadata.

4. The apparatus according to claim 3, wherein, When the first data classifier corresponding to the first database pattern is determined, the at least one non-transitory memory and the program code are configured to utilize the at least one processor to cause the apparatus to: Retrieve at least one of the name metadata, column metadata, or row metadata associated with the first plurality of data objects; and The first data classifier is further determined based on at least one of the name metadata, the column metadata, or the row metadata.

5. The apparatus of claim 2, wherein the first plurality of data objects includes a second data table, wherein the at least one non-transitory memory and the program code are configured to utilize the at least one processor to cause the apparatus to: Determine the relevant metadata associated with the first data table and the second data table; and The first data classifier is further determined based on the relevance metadata.

6. The apparatus according to claim 2, wherein, When the first data classifier corresponding to the first database pattern is determined, the at least one non-transitory memory and the program code are configured to utilize the at least one processor to cause the apparatus to: Determine the domain metadata associated with the first data table; and The first data classifier is further determined based on the domain metadata.

7. The apparatus of claim 1, wherein the at least one non-transitory memory and the program code are configured to utilize the at least one processor to cause the apparatus to calculate the confidence score based on the similarity between the first plurality of data objects and data associated with the supervised learning model.

8. A computer-implemented method, comprising: Receive a request to execute a data mapping process for the first database schema; In response to the request, Retrieve from the database a first plurality of data objects for a first data model associated with the first database schema; A first data classifier corresponding to the first database pattern is determined by using a first machine learning model that processes the first plurality of data objects and relevance metadata as input, wherein the relevance metadata indicates the relationship between the first database pattern and at least one other database pattern in the database. Calculate the confidence score associated with the first data classifier; and In response to determining that the confidence score meets a predetermined threshold, Generate a mapping specification, including a set of rules for data operations associated with the mapping specification, the set of rules being at least partially based on the first data classifier and metadata associated with the first plurality of data objects, wherein the mapping specification includes one or more rule statements specifying the movement and / or transformation of the representation of a second plurality of data objects associated with a second database schema based on the first plurality of data objects associated with the first database schema; Based on the set of rules associated with the mapping specification, the first plurality of data objects associated with the first database schema are transformed into a second plurality of data objects for a second data model associated with the second database schema; and Provide one or more insights about the second plurality of data objects.

9. The computer-implemented method of claim 8, wherein the first plurality of data objects includes a first data table, wherein the first data table includes at least one data field.

10. The computer-implemented method of claim 9, wherein the first data table includes at least one of name metadata, column metadata, or row metadata.

11. The computer-implemented method of claim 10, wherein determining the first data classifier corresponding to the first database pattern further comprises: Retrieve at least one of the name metadata, column metadata, or row metadata associated with the first plurality of data objects; and The first data classifier is further determined based on at least one of the name metadata, the column metadata, or the row metadata.

12. The computer-implemented method of claim 9, wherein the first plurality of data objects includes a second data table, and wherein the computer-implemented method further comprises: Determine the relevant metadata associated with the first data table and the second data table; as well as The first data classifier is further determined based on the relevance metadata.

13. The computer-implemented method according to claim 9, further comprising: Determine the domain metadata associated with the first data table; as well as The first data classifier is further determined based on the domain metadata.

14. The computer-implemented method of claim 8, wherein calculating the confidence score comprises calculating the confidence score based on the similarity between the first plurality of data objects and the data associated with the supervised learning model.

15. A computer program product comprising at least one non-transitory computer-readable storage medium storing a portion of computer-readable program code, the computer-readable program code portion including an executable portion configured to: Receive a request to execute a data mapping process for the first database schema; In response to the request, Retrieve from the database a first plurality of data objects for a first data model associated with the first database schema; A first data classifier corresponding to the first database pattern is determined by using a first machine learning model that processes the first plurality of data objects and relevance metadata as input, wherein the relevance metadata indicates the relationship between the first database pattern and at least one other database pattern in the database. Calculate the confidence score associated with the first data classifier; and In response to determining that the confidence score meets a predetermined threshold, Generate a mapping specification, including a set of rules for data operations associated with the mapping specification, the set of rules being at least partially based on the first data classifier and metadata associated with the first plurality of data objects, wherein the mapping specification includes one or more rule statements specifying the movement and / or transformation of the representation of a second plurality of data objects associated with a second database schema based on the first plurality of data objects associated with the first database schema; Based on the set of rules associated with the mapping specification, the first plurality of data objects associated with the first database schema are transformed into a second plurality of data objects for a second data model associated with the second database schema; and Provide one or more insights about the second plurality of data objects.

16. The computer program product of claim 15, wherein the first plurality of data objects includes a first data table, wherein the first data table includes at least one data field.

17. The computer program product of claim 16, wherein the first data table includes at least one of name metadata, column metadata, or row metadata.

18. The computer program product of claim 17, wherein when the first data classifier corresponding to the first database pattern is determined, the executable portion is configured to: Retrieve at least one of the name metadata, column metadata, or row metadata associated with the first plurality of data objects; and The first data classifier is further determined based on at least one of the name metadata, the column metadata, or the row metadata.

19. The computer program product of claim 16, wherein the first plurality of data objects includes a second data table, and wherein the executable portion is configured to: Determine the relevant metadata associated with the first data table and the second data table; and The first data classifier is further determined based on the relevance metadata.

20. The computer program product of claim 16, wherein when the first data classifier corresponding to the first database pattern is determined, the executable portion is configured to: Determine the domain metadata associated with the first data table; and The first data classifier is further determined based on the domain metadata.

Images