Metadata index for information management

Abstract

A method, apparatus, computer system, and computer program product for managing information. A set of bucketed hashes and comparison information for a data record is identified by a computer system. The set of bucketed hashes is generated from the comparison information, wherein the set of bucketed hashes and the comparison information form a metadata record. A plurality of candidate metadata records in a metadata database is identified by the computer system using the set of bucketed hashes, wherein the plurality of candidate metadata records includes a set of candidate bucketed hashes and candidate comparison information. An entity membership for the data record is identified by the computer system from a comparison of the comparison information in the metadata record to the candidate comparison information in the plurality of candidate metadata records.

Description

Background Technology

[0001] 1. Field:

[0002] This disclosure generally relates to an improved computer system, and more specifically to a method, apparatus, system, and computer program product for managing data sets in a computer system using metadata indexes.

[0003] 2. Relevant technical descriptions:

[0004] The amount of data collected and stored by organizations is constantly increasing. For example, businesses may store data about products, services, customers, transactions, and other types of information. The goal is to ensure that the most up-to-date and accurate versions of this information are available. This information may reside in different databases or other collections of information.

[0005] For large amounts of information, master data management solutions can be used to work with this data to perform indexing, matching, and linking of data from different sources. A master data management system can be used as a central data management system to provide access to a single, trusted view of the data. Master data management provides accurate and comprehensive matching of data, helping to overcome discrepancies and errors that may occur across different data sources. In matching data, pairs of records from different sources can be compared to determine if they match and whether they should be linked together. Record attributes within the records are compared to determine if they match. These record attributes can be, for example, names, addresses, phone numbers, or other suitable information. Different comparison functions can be used to compare records based on matching criteria, such as exact match, distance, speech, partial match, n-gram, or some other criterion. Scores can be generated based on the results of these types of comparisons. For example, scores can be assigned to each attribute and combined based on statistically determined relative weights to determine whether records are sufficiently matched to be linked. Summary of the Invention

[0006] According to one embodiment of the present invention, a method for managing information. A computer system identifies a set of bucket hashes and comparison information for data records. The set of bucket hashes is generated based on the comparison information, wherein the set of bucket hashes and comparison information form metadata records. The computer system uses the set of bucket hashes to identify multiple candidate metadata records in a metadata database, wherein the multiple candidate metadata records include the set of candidate bucket hashes and candidate comparison information. The computer system identifies the entity membership of the data records by comparing the comparison information in the data records with the candidate comparison information in the multiple candidate metadata records.

[0007] According to another embodiment of the present invention, the information management system includes a computer system and a metadata database stored in the computer system. Each metadata record in the metadata database includes a set of bucket hashes, comparison information, and entity membership, wherein the set of bucket hashes is generated using the comparison information, and the entity membership identifies a set of related metadata records.

[0008] According to another embodiment of the present invention, a computer program product for managing information includes a computer-readable storage medium having first program code, second program code, and third program code stored on a computer-readable storage medium. The first program code is executable by a computer system to enable the computer system to identify a set of bucket hashes and comparison information for a data record, wherein the set of bucket hashes is generated from the comparison information, and wherein the set of bucket hashes and comparison information forms a metadata record. The second program code is executable by a computer system to enable the computer system to use the set of bucket hashes to identify multiple candidate metadata records in a metadata database, wherein the multiple candidate metadata records include candidate bucket hashes and candidate comparison information. The third program code is executable by a computer system to enable the computer system to determine the entity membership of a data record based on a comparison of the comparison information in the metadata record with the candidate comparison information in the multiple candidate metadata records. Attached Figure Description

[0009] Figure 1 This is a cloud computing environment based on an illustrative embodiment;

[0010] Figure 2 This is a diagram of an abstract model layer based on an illustrative embodiment;

[0011] Figure 3 It is a graphical representation of a data processing system network that can implement the illustrative embodiments;

[0012] Figure 4 This is a block diagram of an information management environment according to an illustrative embodiment;

[0013] Figure 5 This is a diagram illustrating the data flow of matching records using an information manager according to an illustrative embodiment;

[0014] Figure 6 This is an illustration of metadata records according to an illustrative embodiment;

[0015] Figure 7 This is a flowchart of a process for managing information according to an illustrative embodiment;

[0016] Figure 8 This is a flowchart of a process for managing information according to an illustrative embodiment;

[0017] Figure 9This is a flowchart of a process for determining entity membership according to an illustrative embodiment;

[0018] Figure 10 This is a flowchart of a process for managing information according to an illustrative embodiment;

[0019] Figure 11 This is a flowchart of a process for generating a metadata database for linking data records, according to an illustrative embodiment; and

[0020] Figure 12 This is a block diagram of a data processing system according to an illustrative embodiment. Detailed Implementation

[0021] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0022] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (e.g., punched cards or protrusions in slots with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0023] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0024] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0025] This document describes various aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0026] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that these instructions, which execute via the computer processor or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, thereby comprising an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0027] These computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing the instructions to be executed on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0028] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, two consecutively shown blocks may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0029] These illustrative embodiments recognize and take into account several different considerations. For example, the illustrative embodiments recognize and take into account that, during the matching of information, records undergo a candidate selection process. The selected candidate records are those suspected of being duplicate records. The illustrative embodiments recognize and take into account comparing candidate records after candidate selection to make a decision on whether to link or not to link the records.

[0030] The illustrative embodiments recognize and consider that, historically, candidate selection has been a very expensive task because the process involves scanning large amounts of data in the form of bucket hashes. The illustrative embodiments recognize and consider that obtaining comparison data for the selected candidate records is necessary in this process and can be a significantly expensive process. The illustrative embodiments recognize and consider that the efficiency of the matching process depends on the efficiency of candidate selection and the retrieval of comparison data for these candidates. Therefore, the illustrative embodiments recognize and consider that each step involves searching and retrieving records from the database.

[0031] Therefore, illustrative embodiments recognize and consider that reducing many database accesses is desirable when processing data records to identify matches and generate links. Illustrative embodiments recognize and consider that access to three types of columns in a row of a database can be used to reduce access, where one column contains a bucket hash, the second column contains a comparison string, and the third column contains linking information for the records. In one illustrative example, a single index is used to facilitate faster matching capabilities. This index is a collection of data in the form of a database containing metadata records, and the index may also be referred to as the database. Each metadata record includes a set of bucket hashes and comparison strings. The metadata record may also include entity membership. Entity membership is a field that includes an identifier for a data record. Other data records with the same identifier are considered to be linked.

[0032] As used in this article, "a set" when referring to items means one or more items. For example, "a set of bucket hashes" is one or more bucket hashes.

[0033] Therefore, illustrative embodiments recognize and contemplate the desire to have a system with an improved indexing mechanism for matching metadata. In one illustrative example, a method, apparatus, computer system, and computer program manage product information. The computer system identifies a set of bucket hashes and comparison information for data records. The set of bucket hashes is generated based on the comparison information, wherein the set of bucket hashes and comparison information forms metadata records. The computer system uses the bucket hashes to identify multiple candidate metadata records in a metadata database, wherein the multiple candidate metadata records include the set of candidate bucket hashes and candidate comparison information. The computer system identifies the entity membership of the data records based on a comparison of the comparison information in the metadata records with the candidate comparison information of the multiple candidate metadata records.

[0034] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings stated herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0035] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0036] The features are as follows:

[0037] On-demand self-service: Cloud consumers can automatically and unilaterally configure computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0038] Extensive network access: Capabilities are available through the network and accessed via standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0039] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0040] Rapid elasticity: Capacity can be provided quickly and flexibly (in some cases, automatically) to shrink rapidly and expand rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0041] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0042] The service model is as follows:

[0043] Software as a Service (SaaS): This provides consumers with the ability to use a provider's applications running on cloud infrastructure. These applications can be accessed from different client devices via a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0044] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created by the consumer or acquired using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0045] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0046] The deployment model is as follows:

[0047] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0048] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0049] Public cloud: Cloud infrastructure that is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0050] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public clouds) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0051] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes.

[0052] See now Figure 1The diagram illustrates a cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The cloud computing nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 1 The types of computing devices 54A-N shown are intended to be illustrative only, and the cloud computing node 10 in the cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0053] See now Figure 2 This shows the result of Figure 1 The cloud computing environment in 50 provides a set of functional abstraction layers. It should be understood beforehand that... Figure 2 The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided.

[0054] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0055] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0056] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources based on anticipated future needs according to the SLA.

[0057] Workload tier 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this tier include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and master data management 96. Master data management 96 provides for... Figure 1 Management information or access in the cloud computing environment 50 Figure 1 The cloud computing environment 50 provides services for the network in the physical location of the cloud. For example, Master Data Management 96 can be used to manage multiple copies of information that may reside in different repositories. Master Data Management 96 can operate to maintain a single version of the truth across all copies of information for an entity. In this example, Single Version Authenticity (SVOT) involves storing the entity's data in a consistent, non-redundant form.

[0058] See now Figure 3 This image depicts a graphical representation of a data processing system network in which illustrative embodiments can be implemented. The network data processing system 300 is a computer network in which illustrative embodiments can be implemented. The network data processing system 300 includes a network 302, which is a medium for providing communication links between different devices and computers connected together within the network data processing system 300. The network 302 may include connections such as wired, wireless communication links, or fiber optic cables.

[0059] In the depicted example, server computers 304 and 306 are connected to network 302 along with data repositories 307 and 308. Additionally, client device 310 is connected to network 302. As described, client device 310 includes client computers 312, 314, and 316. Client device 310 can be, for example, a computer, workstation, or network computer. In the depicted example, server computer 304 provides information such as boot files, operating system images, and applications to client device 310. Furthermore, client device 310 may also include other types of client devices, such as mobile phone 318, tablet computer 320, and smart glasses 322. In this illustrative example, server computer 304, server computer 306, data repository 307, data repository 308, and client device 310 are network devices connected to network 302, where network 302 is the communication medium for these network devices. Some or all of the client devices 310 can form an Internet of Things (IoT), in which these physical devices can connect to network 302 and exchange information with each other through network 302.

[0060] Data repositories 307 and 308 are hardware systems and act as repositories for storing, managing, or distributing at least one of the information. These data stores can store information such as customer information, transactions, inventory information, portfolios, business plans, and other information. Information can be stored in databases located in data repositories 307 and 308. These databases can take different forms. For example, the database can be selected from at least one of a graph database, a non-relational distributed database, a relational database, or other types of databases.

[0061] In this example, client device 310 is a client of server computer 304. Network data processing system 300 may include additional server computer, client computer, and other devices not shown. Client device 310 is connected to network 302 using at least one of a wired connection, a fiber optic connection, or a wireless connection.

[0062] The program code located in the network data processing system 300 can be stored on a computer-recordable storage medium and downloaded to the data processing system or other devices for use. For example, the program code can be stored on a computer-recordable storage medium on a server computer 304 and downloaded to a client device 310 via network 302 for use on the client device 310.

[0063] In the illustrated example, network data processing system 300 is the Internet, where network 302 represents a global collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. The core of the Internet is the backbone of high-speed data communication lines between master nodes or host computers, which consist of thousands of commercial, government, educational, and other computer systems routing data and messages. Of course, network data processing system 300 can also be implemented using many different types of networks. For example, network 302 can include at least one of the following: the Internet, intranet, local area network (LAN), metropolitan area network (MAN), or wide area network (WAN). Figure 3 This is intended as an example, not as an architectural limitation for different illustrative embodiments.

[0064] As used in this article, "multiple" when referring to a project means one or more projects. For example, "multiple different types of networks" means one or more different types of networks.

[0065] Furthermore, when the phrase "at least one" is used with a list of items, it means that different combinations of one or more of the listed items can be used, and it is possible that only one of each item in the list is needed. In other words, "at least one" means that any combination of items and multiple items in the list can be used, but not all items in the list. Items can be specific objects, things, or categories.

[0066] For example, but not limited to, "at least one of Item A, Item B, or Item C" can include Item A, Item A and Item B, or Item B. The example could also include Item A, Item B and Item C, or Item B and Item C. Of course, any combination of these items can exist. In some illustrative examples, "at least one" can be, for example, but not limited to, two Item A; one Item B; and ten Item C; four Item B and seven Item C; or other suitable combinations.

[0067] As depicted, information manager 321 is operable to manage information located in data repositories 307 and 308. In this illustrative example, information manager 321 can identify duplicate copies of information, perform data deduplication, or verify the accuracy between different copies of information. For example, information manager 321 can compare and match information in these different locations to perform data deduplication and other functions when managing information. This information may be stored in records in a database or other types of information collections. In this illustrative example, information manager 321 uses metadata database 323 to perform steps such as identifying candidate data records, performing comparisons of candidate data records, and determining whether to link data records.

[0068] In this illustrative example, metadata record 324 is configured to increase the speed and efficiency of these steps. For example, metadata record 324 is designed to reduce the number of tables or databases accessed during the execution of steps such as candidate selection, comparison, and linked or unlinked steps. For example, metadata record 326 within metadata record 324 includes bucket membership 328, comparison string 330, and entity membership 332. Bucket membership 328 includes a bucket hash generated using comparison string 330. Entity membership 332 is a unique identifier that identifies records linked to each other.

[0069] In other words, all metadata records linked to each other in metadata record 324 have the same unique identifier for entity membership 332. Linked records can be for entities such as people, organizations, companies, government entities, households, accounts, or some other type of entity.

[0070] In this illustrated example, the information needed to identify candidates, compare candidates, and link or not link data records is found in a single metadata record within metadata record 324. This avoids multiple accesses to different databases. Therefore, the reduction in accesses increases the speed at which data records can be processed. Time savings increase with the number of data records. For example, significant time savings can be achieved when processing a large number of records (such as 3,000,000,000 records).

[0071] Figure 3 The illustrations in the examples are not intended to limit the ways in which other illustrative instances can be implemented. For example, information may be located in locations other than or in place of data repositories 307 and 308. For example, information managed by information manager 321 may also be located in at least one of server computer 304, server computer 306, or a group of client devices 310.

[0072] See now Figure 4 A block diagram of an information management environment according to an illustrative embodiment is depicted. In this illustrative example, the information management environment 400 includes hardware (such as...) Figure 1 The components implemented in the hardware shown in the network data processing system 100.

[0073] As depicted, the information management system 402 is operable to manage information 404 in a data repository 406. In this example, the data repository 406 is a hardware system for persistently storing and managing the information collection 404. The data repository 406 may include an information repository selected from at least one of databases, files, documents, spreadsheets, or some other suitable form. The data repository 406 may be located in a single location or may be distributed across many locations on different computers or storage systems.

[0074] In an illustrative example, information 404, in the form of data record 408, is stored in a set of databases 410 within a data repository 406. As depicted, a database is a data structure that stores information in an organized manner. The database may include multiple tables. Different tables may represent different types of information, such as products, financial records, employees, or other information. In this example, data record 408 is stored in different tables within the set of databases 410.

[0075] As depicted, the information management system 402 includes a computer system 412 and an information manager 414. The information manager 414 is located within the computer system 412.

[0076] The information manager 414 can be implemented in software, hardware, firmware, or a combination thereof. When software is used, the operations performed by the information manager 414 can be implemented in program code configured to run on hardware such as a processor unit. When firmware is used, the operations performed by the information manager 414 can be implemented in program code and data and stored in persistent memory for execution on a processor unit. When hardware is used, the hardware may include circuitry that operates to perform the operations in the information manager 414.

[0077] In the illustrative examples, the hardware may take the form of at least one of the following: a circuit system, an integrated circuit, an application-specific integrated circuit (ASIC), a programmable logic device, or some other suitable type of hardware configured to perform multiple operations. Using a programmable logic device, the device can be configured to perform multiple operations. The device can be reconfigured later or can be permanently configured to perform multiple operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field-programmable logic arrays, field-programmable gate arrays, and other suitable hardware devices. Furthermore, the process can be implemented in organic components integrated with inorganic components and can consist entirely of organic components other than human-made ones. For example, these methods can be implemented as circuits in organic semiconductors.

[0078] Computer system 412 is a physical hardware system and includes one or more data processing systems. When more than one data processing system exists in computer system 412, those data processing systems communicate with each other using a communication medium. The communication medium may be a network. The data processing systems may be selected from at least one of a computer, a server computer, a tablet computer, or some other suitable data processing system.

[0079] In this illustrative example, the information manager 414 can perform several different steps within the management information 404 (such as data record 408 in the group database 410). The management of data record 408 can be formed using the metadata database 416.

[0080] For example, the information manager 414 can identify a set of bucket hashes 418 and comparison information 420 for data record 422 in data record 408. In this example, the set of bucket hashes 418 and comparison information 420 for data record 422 forms metadata record 424 in metadata record 426 of metadata database 416. In this illustrative example, the information manager 414 can identify this information by receiving the set of bucket hashes 418 and comparison information 420 for data record 422 received from index service 428.

[0081] In this illustrative example, indexing service 428 may be a large-scale indexing service capable of generating bucket hashes 418 for a large number of data records 408. Indexing service 428 may also identify comparison information 420 from the data records 408. As shown in the figure, in one illustrative example, indexing service 428 may be a function or process in a currently available master data management system.

[0082] In this example, the set of bucket hashes 418 is generated from comparison information 420. The bucket hashes in the bucket hashes 418 are values ​​generated by applying a hash function to one or more comparison information entries 420. In this example, the set of bucket hashes 418 may be based on comparison information 420, which is selected from at least one of the following: first name, last name, address, street name, state, zip code, city of residence, telephone number, occupation, business name, date of birth, account number, occupation, business telephone number, business address, or tax identification number.

[0083] Furthermore, multiple bucket hashes can be generated from this set of bucket hashes 418 for a single data record, and each of these multiple bucket hashes can represent only a portion of a field or attribute in the data record. For example, using the postal code 12345, three bucket hashes can be generated. For instance, the first bucket hash can be generated by applying a hash function to "123", the second bucket hash can be generated by applying the same hash function to "234", and the third bucket hash can be generated by applying the same hash function to "345". This type of bucket hash generation can be referred to as bucketing based on n-gram syntax.

[0084] Therefore, multiple bucket hashes can be generated within this set of bucket hashes 418 for fields or attributes in a single data record, where each bucket hash represents only a portion of the fields or attributes in the data record. In this way, partial matches can be identified to account for potential data entry errors. This type of multi-bucket hash generation can be applied to other types of information in data records, such as phone numbers, birthdays, or other suitable information.

[0085] The comparison information 420 can take many different forms. For example, comparison information 420 can be at least one of a comparison string 430 or a text string 432 representing information 404 in data record 422. In this illustrative example, comparison string 430 can be text with added formatting or syntax, which is used or recognized by a specific comparison or matching process. For example, the position or orientation of the text within comparison string 430 can indicate a specific type of information in data record 422. This specific type of information can also be referred to as an attribute.

[0086] Furthermore, syntax desired by the specific comparison or matching process can be added. As depicted, text string 432 can be text obtained from data record 422 without formatting or any specific syntax.

[0087] In this illustrative example, the bucket hash 418 and comparison information 420 form a metadata record 424 that can be stored in a metadata database 416. The metadata database 416 can also be referred to as an index. Therefore, each data record in the data records of interest can be processed to generate a metadata record 426 in the metadata database 416.

[0088] As depicted, the information manager 414 can use the set of bucket hashes 418 to identify multiple candidate metadata records 434 in the metadata database 416. For example, the information manager 414 can search metadata records 426 that include bucket hashes in metadata records 424 to identify multiple candidate metadata records 434. This type of search can be performed against each of the set of bucket hashes 418 in metadata records 424 to find multiple candidate metadata records 434. In this way, a search for data records 408 that can be associated with data record 422 can be performed using metadata record 426. The multiple candidate metadata records 434 may include a set of candidate bucket hashes 436 and candidate comparison information 438.

[0089] Furthermore, the information manager 414 can determine the entity membership 440 of the data record 422 from a comparison 442 between comparison information 420 in the metadata record 424 and candidate comparison information 438 in multiple candidate metadata records 434. The comparison 442 may include a score indicating the probability of a match between the metadata record 424 and other metadata records in the metadata record 426.

[0090] In this way, it is possible to determine the probability that data record 422, represented by data record 422, is the same as or sufficiently similar to other data records in data record 408. When the probability is that data record 422 is the same as or sufficiently similar to other data records in data record 408, a match is considered to exist.

[0091] The rating may include sub-parts or sub-ratings that indicate the probability of a match between different parts of data record 422. These different parts may be fields or attributes in data record 422. For example, a part may be at least one of the following: first name, last name, address, street name, state, zip code, city of residence, phone number, occupation, business name, date of birth, account number, occupation, business phone number, business address, tax identification number, or some other suitable information found in a part of the data record.

[0092] For example, in determining entity membership 440, the information manager 414 can compare the comparison information 420 in metadata record 424 with candidate comparison information 438 in multiple candidate metadata records 434 to form a comparison 442. The information manager 414 can generate a set of comparison scores 444 from the comparison 442. The information manager 414 can determine entity membership 440 based on a comparison of the set of comparison scores 444 with a set of thresholds 446.

[0093] In this illustrative example, the set of thresholds 446 can be a set of values, such as a set of percentages. If the comparison score 444 has a percentage indicating how close a match is to a match for a particular attribute or for all comparison information, then the comparison information 420 in metadata record 424 can be considered a match with each of the multiple candidate metadata records in the multiple candidate metadata records 434, where the set of comparison scores 444 satisfies or exceeds the set of thresholds 446.

[0094] In this illustrative example, entity membership 440 in metadata record 424 of data record 422 can be updated to have the same unique identifier as one or more other metadata records in metadata record 426 of data record 408 that are considered a match. In another illustrative example, entity membership 440 in metadata record 424 may have a value indicating that data record 422 corresponding to metadata record 424 is unique and does not match another data record in data record 408. In this illustrative example, such linking of data record 408 can be performed by information manager 414, which updates entity membership in metadata record 426 to indicate the link between data records 408. For example, information manager 414 may update metadata record 424 using entity membership 440 determined for data record 422 from comparison 442 with candidate metadata record 434. The update may include a unique identifier for candidate metadata record 434 that matches metadata record 424.

[0095] Using metadata records 426 that have already been processed to determine entity membership in metadata database 416, this database can be used to search for data records 408 in the same set of databases 410 within data repository 406. A similar process can be performed using input data records or search items in a search request.

[0096] For example, information manager 414 may receive search request 448 from requester 450. In this instance, requester 450 may be a user in the form of a human operator or a process or program running on a computer.

[0097] Information manager 414 can use the search terms 456 received from the search request 448 from requester 450 to create a set of search bucket hashes 452 and search comparison information 454. Information manager 414 can use this set of search bucket hashes 452 to search the metadata database 416 to find the set of candidate metadata records 434.

[0098] Information manager 414 can determine whether candidate metadata record 458 from the set of candidate metadata records 434 is a match for search term 456 based on the set comparison score 444, where the set comparison score 444 indicates how similar candidate comparison information 438 in the set of candidate metadata records 434 is to search comparison information 454. Information manager 414 can use the source identifier 462 in candidate metadata record 458 to return the identified data record 460 in data record 408 corresponding to metadata record 424 to requester 450.

[0099] In one illustrative example, features that overcome the time and processing resources required to analyze information (such as data records) may exist in one or more databases of other types of repositories. In one illustrative example, information manager 414 may be operable to perform a process for deduplicating data to match information (such as information 404 located in data record 408). Metadata database 416 may be used to reduce the amount of searching in the database or table to match data record 408.

[0100] In the illustrative example, metadata (such as bucket hash 418 and comparison information 420) resides in metadata record 426 within metadata database 416. In the depicted example, metadata record 426 is generated for data records 408 to be compared. In this instance, each metadata record in metadata record 426 includes a set of bucket hash 418 and comparison information 420 that has already been generated for the corresponding data record in data record 408.

[0101] In the illustrative example, a search can be performed in metadata record 426 to match the bucket hashes 418 within metadata record 426 to identify candidate metadata records 434 that may potentially be linked to each other. Each candidate metadata record includes a set of bucket hashes 418 and comparison information 420. When comparing the comparison information 420, additional queries to the database are unnecessary because the comparison information 420 exists along with the set of bucket hashes 418 in data record 422.

[0102] Furthermore, when the comparison result shows a match between the comparison information 420 of data records 408, the entity membership 440 of these data records in the corresponding metadata record 426 can be updated to link the two data records in data record 408 to each other. Furthermore, this process can also be used to update the entity membership 440 in the metadata record 426 to unlink the corresponding data records in data record 408.

[0103] In this way, the number of database accesses is reduced compared to currently used technologies. Therefore, metadata records 426 in the metadata database 416 form an index that can be used to match the metadata of data record 408.

[0104] These metadata records can be used to search data record 408. For example, when a search request 448 is received, the information manager 414 can generate a search bucket hash 452 and search comparison information 454 from the search term 456 in the search request 448. This information can be used to find candidate metadata records 434, and then a comparison is performed using the search comparison information 454 and the candidate comparison information 438. When a match is found with one or more of the candidate metadata records 434, the source identifiers in those candidate metadata records can be used by the information manager 414 to perform a location or return of at least one of the data records 408 in response to the search request 448.

[0105] Computer system 412 can be configured to perform at least one of the steps, operations, or actions described in various illustrative examples using software, hardware, firmware, or a combination thereof. Thus, computer system 412 operates as a dedicated computer system, wherein information manager 414 within computer system 412 allows for the management of information 404, including updating entity membership to link or unlink data records 408 using processes that reduce the time and resources required to perform comparisons. Specifically, information manager 414 transforms computer system 412 into a dedicated computer system compared to currently available general-purpose computer systems that do not have information manager 414 that uses metadata database 416 to compare information 404 and data records 408.

[0106] In this illustrative example, the use of an information manager 414 in computer system 412 integrates processing into a practical application for managing information 404 that enhances the performance of computer system 412. In other words, the information manager 414, using a metadata database 416 within computer system 412, addresses a practical application of a process integrated into the information manager 414 within computer system 412, which uses metadata records 426 in the metadata database 416 to identify data records 408 that are considered to match each other. In this illustrative example, the information manager 414 in computer system 412 provides a practical application of information management, thereby improving the functionality of computer system 412 compared to current technologies.

[0107] Therefore, the illustrative example provides a metadata architecture that facilitates more efficient selection and comparison of candidates. Furthermore, the structure of the metadata records reduces the large-bucket problem of current techniques and enables the acquisition of data with correlations between multiple buckets. Moreover, this design also enables the retrieval of candidate metadata records from multiple buckets based on bucket hashing in a single pass, rather than performing this step one by one as in current techniques.

[0108] Turn now Figure 5 The illustration depicts a data flow for matching records using an information manager, according to an illustrative embodiment. In this illustrative example, the same reference numerals may be used in more than one drawing. This reuse of reference numerals in different drawings indicates the same elements in different drawings.

[0109] In this illustrative example, the information manager 414 includes several different components. As depicted, the information manager 414 includes a metadata record manager 500 and a matching and linking service 502.

[0110] In this illustrative example, data records 408 in data repository 406 can be processed by index service 428. Index service 428 can be a bulk index service capable of processing most or all of data records 408. In this illustrative example, index service 428 includes a normalizer 504, a comparison information generator 506, and a bucket hash generator 508.

[0111] Normalizer 504 can generate comparison information and, as part of the comparison information generation, can normalize the information in data record 408. For example, names can be normalized by normalizer 504. In one example, the names “Bob,” “Robert,” and “William” can be normalized to “Bob,” such that each instance of “Robert” and “William” is changed to “Bob” to generate bucket hashes and comparison information. As another example, speech normalization can be performed, where “Richard” and “Richard” can be normalized by normalizer 504 to the speech representation of “r IH ch er d.” As depicted, comparison information generator 506 can generate comparison information for each data record, such as a comparison string including at least one of the formatting or syntax used by the comparison process.

[0112] In this illustrative example, the bucket hash generator 508 can generate a bucket hash based on comparison information 420. For example, the bucket hash generator 508 can use normalized terms generated by the normalizer 504 to generate a bucket hash 418 for each data record, and the comparison information generator 506 can be used to generate a comparison string in the comparison information 420.

[0113] Index service 428 sends bucket hash 418 and comparison information 420 to metadata record manager 500 in information manager 414. As depicted, metadata record manager 500 can use bucket hash 418 and comparison information 420 to generate metadata record 426. Therefore, through index service 428, metadata records in metadata record 426 in metadata database 416 exist for each corresponding data record process.

[0114] In this illustrative example, the metadata record manager 500 can be implemented using several different types of processes. For example, the metadata record manager 500 may include a search engine such as ElasticSearch. This type of search engine can generate an index of metadata records 426 in the metadata database 416. This indexing of metadata can increase the speed and efficiency of performing searches on candidate metadata records.

[0115] In this illustrative example, the matching and linking service 502 includes several different components. As shown in the figure, the matching and linking service 502 includes a candidate selector 510, a comparator 512, and a connector 514.

[0116] Candidate selector 510 can search for candidate metadata records 434, which are candidates used to link to metadata records processed to identify matching metadata records. This search can be performed using bucket hashes 418. For example, each bucket hash in a metadata record can be used to search for other metadata records that include that bucket hash. Data records with matching bucket hashes are candidate metadata records 434 returned from metadata records 426 in the metadata database 416.

[0117] As depicted, comparator 512 can compare candidate comparison information 438 in candidate metadata record 434 with comparison information 420 in metadata record 424 of the data record 422 being processed. This comparison is performed without requiring another access to metadata database 416 or another database to obtain the comparison information. In this example, candidate comparison information 438 is located in candidate metadata record 434 along with candidate bucket hash 436. Therefore, in this example, an additional database access using metadata record 426 to search for comparison information is unnecessary. In this example, comparator 512 can consider syntactic similarity between different data records with the same meaning. This comparison can be performed using an equivalence function.

[0118] When one or more matches are identified, linker 514 can update the entity membership of one or more candidate metadata records that match the metadata record being processed. This update can... Figure 4 The update is performed on the candidate metadata records 434 returned during the initial search of the bucket hash 418. As a result, additional access to the database for entity membership is unnecessary. The update can be performed by linking or delinking the data records 408 to each other by updating the entity membership in the metadata record being processed.

[0119] In this way, data records 408 can be processed and linked in a manner that reduces at least one of the number of database accesses or the time required to process data records 408. When duplicate data records are identified, data deduplication or other steps can then be performed.

[0120] Figure 4 and Figure 5 The depiction of the information management environment 400 and its various components does not imply any physical or architectural limitations on the manner in which the illustrative embodiments can be implemented. Other components may be used in addition to or in place of the illustrated components. Some components may be unnecessary. Furthermore, boxes are presented to illustrate certain functional components. When implemented in the illustrative embodiments, one or more of these boxes may be combined, divided, or combined and divided into different boxes.

[0121] For example, metadata record 424 may include additional information from fields other than the bucket hash 418 and comparison information 420. For instance, metadata record 424 may include an attribute or field of at least one of entity membership, customer identifier, record type, probability self-rating, timestamp, or other suitable information. Entity membership is an identifier that can be used to link records. The customer identifier identifies a customer. In this example, record type is an identifier of the type or category of a record (such as a person or organization). Different comparison functions can be used based on the record type. As a result, the record type can be used to determine how to perform a match on data of that type.

[0122] In the illustrated example, the probabilistic self-rating score is a self-comparative score of the record. This score indicates the strength of the record. For example, if a record has more attributes, its self-comparative score may be higher than a record with fewer attributes. This score can be used to perform some analysis on the quality of the source data. Sources with records having higher self-rating scores are richer than sources with records having lower self-rating scores. The timestamp indicates when the metadata record 424 was last updated.

[0123] In another illustrative example, information manager 414 can identify bucket hash 418 and comparison information 420 by generating information from data record 408. As another example, data repository 406 may reside in a separate computer system from computer system 412.

[0124] See Figure 6 An illustration of a metadata record is depicted according to an illustrative embodiment. Metadata record 600 is achievable... Figure 4 An example of one way to use metadata record 424 in the database.

[0125] In the illustrative example, metadata record 600 includes several different attributes. As shown in the figure, metadata record 600 includes internal record identifier 602, source identifier 604, bucket hash 606, comparison string 608, entity membership 610, and text search 612.

[0126] Internal record identifier 602 is a unique identifier that uniquely identifies metadata record 600 within the metadata database. Source identifier 604 can be used to locate the data record corresponding to metadata record 600. For example, source identifier 604 may include, for example, a source identifier and a record identifier. Record identifier and other information (such as repository identifier, database identifier, IP address, or other suitable information) can be used to locate the data record corresponding to metadata record 600.

[0127] Bucket hash 606 is a bucket hash generated using comparison information (such as at least one of comparison string 608 or text search 612). As shown, bucket hash 606 in metadata record 600 can be searched against bucket hashes in other metadata records.

[0128] In the illustrated example, comparison string 608 is an example of comparison information. Comparison string 608 contains text having at least one of formatting or syntax, which is used by a process that can be used to compare data records. Comparison string 608 can be a concatenated text representation of a data record. Using comparison string 608, text-based search functionality in a search engine for a metadata database can be utilized to perform faster identification of candidate metadata records with a sufficient level of matching. In the illustrative example herein, a match can be formed such that a comparison score is generated and used to determine whether a sufficient number of word matches are sufficient to consider a metadata record as a match with a candidate metadata record.

[0129] In this example, text search 612 contains words that are not in any particular format or grammar. Text search 612 can be used by search engines such as ElasticSearch.

[0130] In this illustrative example, entity membership 610 includes identifiers that link metadata records to each other and, consequently, to data records represented by those metadata records. Each metadata record of a matching data record has the same value for entity membership 610. In some cases, a data record may be unique and may not have any matching data records. In that case, entity membership 610 in the metadata record corresponding to the data record is unique within the metadata record.

[0131] Next, turn to Figure 7 A flowchart depicts a process for managing information according to an illustrative embodiment. Figure 7 The process can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is executed by one or more processor units in one or more hardware devices located in one or more computer systems. For example, the process can be implemented in... Figure 4 It is implemented in the information manager 414 of the computer system 412.

[0132] The process begins by identifying a set of bucket hashes and comparison information for a data record, wherein the set of bucket hashes is generated from the comparison information, and wherein the set of bucket hashes and comparison information forms a metadata record (step 700). The process uses the set of bucket hashes to identify multiple candidate metadata records in the metadata database, wherein the multiple candidate metadata records include a set of candidate bucket hashes and candidate comparison information (step 702). The process determines the entity membership of the data record by comparing the comparison information in the metadata record with the candidate comparison information in the multiple candidate metadata records (step 704). The process then terminates.

[0133] See Figure 8 A flowchart depicts a process for managing information according to an illustrative embodiment. Figure 8 The process described in the text is possible Figure 7 The process execution includes additional steps for managing information.

[0134] This process uses a set of bucket hashes and comparison information to create metadata records for the data records (step 800). Step 800 can be performed at any time. Figure 7 It can be executed at any time during different steps in the process.

[0135] This process updates the metadata record using the entity membership determined for the data record (step 802). The process then terminates. Step 802 can be... Figure 7 This is executed after determining the entity's membership in step 704.

[0136] Turn now Figure 9The diagram depicts a flowchart of a process for determining entity membership according to an illustrative embodiment. The process shown in the diagram is implementable. Figure 7 An example of one way of step 704 in the process.

[0137] The process begins by comparing the comparison information in the metadata record with candidate comparison information in multiple candidate metadata records to form a comparison (step 900). The process generates a set of comparison scores based on the comparisons (step 902). In this illustrative example, a comparison score can be generated for each identified candidate record. Further, the set of comparison scores can include sub-comparison scores, where each sub-comparison score is for a specific attribute or field that can be identified in the candidate comparison information. For example, the attribute or field used for comparison can be identified using a syntax in which the order of strings or tags can exist, to generate sub-comparison scores in the comparison score. The scores can then be aggregated by weighting to generate a comparison score for the entire metadata record. Weights can be assigned to give more weight to some attributes when determining whether a match exists.

[0138] The process determines entity membership based on a comparison of the set of scores with a set of thresholds (step 904). The process then terminates.

[0139] See Figure 10 A flowchart depicts a process for managing information according to an illustrative embodiment. Figure 10 The process can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is executed by one or more processor units in one or more hardware devices located in one or more computer systems. For example, the process can be implemented in... Figure 4 This is implemented in the information manager 414 of the computer system 412. In this flowchart, the process can be used to search for data records using metadata records in the metadata database.

[0140] The process begins by receiving a search request from the requester (step 1000). The process then uses the search terms received in the search request from the requester to create a set of search bucket hashes and search comparison information (step 1002). The set of search bucket hashes is generated using the search terms in the search request. Similarly, the comparison information can be generated using the search terms in the search request.

[0141] The process uses a set of search bucket hashes to search a set of candidate metadata records in the metadata database (step 1004). The process selects candidate metadata records from the set of candidate metadata records based on a set of comparison scores, which indicate how similar the candidate comparison information in the set of candidate metadata records is to the search comparison information (step 1006).

[0142] The process uses the source identifier in the candidate metadata record to return the identified data record corresponding to the candidate metadata record to the requester (step 1008). The process then terminates. The identified data record can be identified using information in the candidate metadata record. For example, the candidate metadata record may include a source identifier. The source identifier includes information that can be used to identify the location of the data record to be returned. This information may include, for example, a repository identifier, a record identifier for the database in which the data record resides, or other suitable information.

[0143] Next turn Figure 11 The diagram depicts a flowchart, according to an illustrative embodiment, of a process for generating a metadata database to link data records. This metadata database may also be referred to as a matching index.

[0144] Figure 11 The process can be implemented in hardware, software, or both. When implemented in software, the process can take the form of program code that is executed by one or more processor units in one or more hardware devices located in one or more computer systems. For example, the process can be implemented in... Figure 4 It is implemented in the information manager 414 of the computer system 412.

[0145] The process begins by receiving incoming data records from a data repository (step 1100). The data repository can be any suitable backend storage or database. For example, the data repository could be Janus Graph, Cloudant, HBase, or a DB2 database. In the example described herein, step 1100 can be performed by a web application 1120. The web application 1120 can operate as a persistence layer and is independent of the indexing service 1130 and the batch matching service 1150.

[0146] This process generates bucket hashes and comparison information for the incoming data records (step 1102). In this example, step 1102 is performed via an indexing service 1130, which receives data records from a web application 1120. For example, the web application 1120 may make a Presentational State Transfer (REST) ​​application interface (API) call to request a bulk indexing service to generate metadata for the data records. In this example, the indexing service 1130 may be a scalable, batch-driven process capable of reading incoming records, normalizing the data, creating comparison strings, and creating bucket hashes for each incoming data record to support downstream statistical matching. Furthermore, the indexing service 1130 may also create text representations of the incoming data records for text-based indexing or fast searching.

[0147] The process stores the bucket hash and comparison information as metadata records in a metadata database (step 1104). In this example, step 1104 can be performed by a data repository manager 1140, which is operable to manage the metadata records in the metadata database. The metadata database can be a data repository that follows a specific data storage format (such as a column-based or row-based format). In this illustrative example, a column-based database can be used with index support, and the resulting information for the incoming data records is stored in the same row. As described, the data repository manager 1140 can perform searches for similar data records against the incoming data records before downstream analysis to improve efficiency. In this illustrative example, the initial search can be based on any search engine, such as Elasticsearch or SOLR.

[0148] In the illustrative example, the batch matching service 1150 performs steps 1106 to 1116. The batch matching service 1150 may be a scalable Spark-based matching engine deployed in a containerized environment.

[0149] The process searches the metadata database for metadata records that contain a bucket hash that matches the bucket hash generated from the incoming data record to identify candidate metadata records (step 1106). In step 1106, the candidate metadata record contains metadata for the corresponding data record. For example, the candidate metadata record contains a bucket hash and comparison information that can be compared with the bucket hash in the comparison information generated from the input data record.

[0150] The process compares the comparison information in the candidate metadata record with the comparison information generated from the incoming data record (step 1108). In this illustrative example, a pairwise comparison can be performed between the candidate's comparison information and the data record in the comparison information generated from the incoming data record. The comparison information in the incoming data record and the comparison information in the metadata record can be compared to form a comparison.

[0151] The process generates a comparison score based on this comparison (step 1110). The process then compares the score to a set of thresholds (step 1112). In this illustrative example, the thresholds in step 1112 can be predefined thresholds. These thresholds can be configurable values ​​defined at the system level. The optimal value for the threshold can be selected based on different trials and tests conducted by the data administrator.

[0152] Using this comparison, candidate metadata records whose comparison scores meet or exceed a predefined threshold can be considered a match, which is used to link the data record corresponding to the candidate metadata record with the incoming data record being processed. The process uses the candidate metadata records identified as matches to identify a unique set of identifiers for the entity membership used to match the data records (step 1114).

[0153] The process then generates a metadata record for the processed incoming data record (step 1116). In step 1116, the metadata record includes a bulk hash and comparison information generated from the incoming data record. Further, the metadata record also includes a unique identifier for entity membership, wherein the unique identifier is the same as the unique identifier in the candidate metadata record, which has been identified as a match from the comparison. The metadata record may also include a source identifier identifying the source of the incoming data record. This source identifier may be information used to retrieve the data record from the source.

[0154] This process stores the metadata record of the incoming data record in the metadata database (step 1118). The process then terminates. This step can be performed by the data repository manager 1140.

[0155] The flowcharts and block diagrams in the various depicted embodiments illustrate the architecture, functionality, and operation of some possible implementations of the devices and methods in the illustrative embodiments. In this regard, each block in a flowchart or block diagram may represent at least one of a module, segment, function, or operation or step. For example, one or more blocks may be implemented as program code, hardware, or a combination of program code and hardware. When implemented in hardware, the hardware may, for example, take the form of an integrated circuit manufactured or configured to perform one or more operations in the flowchart or block diagram. When implemented as a combination of program code and hardware, the implementation may take the form of firmware. Each block in a flowchart or block diagram may be implemented using a dedicated hardware system performing different operations or a combination of dedicated hardware and program code executed by the dedicated hardware.

[0156] In some alternative implementations of the illustrative embodiments, one or more functions indicated in the boxes may not occur in the order shown in the figures. For example, in some cases, depending on the functions involved, two boxes shown consecutively may be executed substantially simultaneously, or these boxes may sometimes be executed in reverse order. Furthermore, additional boxes may be added besides those shown in the flowcharts or block diagrams.

[0157] Turn now Figure 12 A block diagram of a data processing system according to an illustrative embodiment is depicted. The data processing system 1200 can be used to implement... Figure 1The system includes server computer 104, server computer 106, and client device 110. Data processing system 1200 can also be used to implement computer system 412. In this illustrative example, data processing system 1200 includes a communication framework 1202 that provides communication between processor unit 1204, memory 1206, persistent storage 1208, communication unit 1210, input / output (I / O) unit 1212, and display 1214. In this example, communication framework 1202 takes the form of a bus system.

[0158] Processor unit 1204 is used to execute instructions for software that can be loaded into memory 1206. Processor unit 1204 includes one or more processors. For example, processor unit 1204 may be selected from at least one of a multi-core processor, a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a network processor, or some other suitable type of processor. Furthermore, processor unit 1204 may be implemented using one or more heterogeneous processor systems, in which the main processor and secondary processor reside on a single chip. As another illustrative example, processor unit 1204 may be a symmetric multiprocessor system, which contains multiple processors of the same type on a single chip.

[0159] Memory 1206 and persistent storage device 1208 are examples of storage device 1216. A storage device is any hardware capable of storing information such as, for example, but not limited to, data, program code in the form of functions, or at least other suitable information on a temporary, permanent, or both temporary and permanent basis. In these illustrative examples, storage device 1216 may also be referred to as a computer-readable storage device. In these examples, memory 1206 may be, for example, random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 1208 may take various forms depending on the specific implementation.

[0160] For example, persistent storage 1208 may include one or more components or devices. For example, persistent storage 1208 may be a hard disk drive, a solid-state drive (SSD), flash memory, a rewritable optical disc, a rewritable magnetic tape, or a combination of the above. The media used in persistent storage 1208 may also be removable. For example, a removable hard disk drive may be used in persistent storage 1208.

[0161] In these illustrative examples, communication unit 1210 provides communication with other data processing systems or devices. In these exemplary instances, communication unit 1210 is a network interface card.

[0162] The input / output unit 1212 allows data input and output to other devices that can be connected to the data processing system 1200. For example, the input / output unit 1212 can provide a connection for user input via at least one of a keyboard, mouse, or some other suitable input device. Furthermore, the input / output unit 1212 can send output to a printer. The display 1214 provides a mechanism for displaying information to the user.

[0163] Instructions for at least one of the operating system, application, or program may be located in storage device 1216, which communicates with processor unit 1204 via communication frame 1202. Processes in different embodiments may be executed by processor unit 1204 using computer-implemented instructions, which may be located in memory (such as memory 1206).

[0164] These instructions, referred to as program code, computer-usable program code, or computer-readable program code, can be read and executed by a processor in processor unit 1204. The program code in different embodiments may be implemented on different physical or computer-readable storage media, such as memory 1206 or persistent storage 1208.

[0165] Program code 1218 is functionally located on a computer-readable medium 1220, which is selectively removable and can be loaded onto or transferred to a data processing system 1200 for execution by a processor unit 1204. In these illustrative examples, program code 1218 and computer-readable medium 1220 form a computer program product 1222. In the illustrative examples, computer-readable medium 1220 is a computer-readable storage medium 1224.

[0166] In these illustrative examples, computer-readable storage medium 1224 is a physical or tangible storage device for storing program code 1218, and not a medium for propagating or transmitting program code 1218. As used herein, computer-readable storage medium 1224 should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0167] Alternatively, program code 1218 can be transferred to data processing system 1200 using a computer-readable signal medium. The computer-readable signal medium is a signal and can be, for example, a propagated data signal containing program code 1218. For example, the computer-readable signal medium can be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals can be transmitted via a connection (such as a wireless connection, fiber optic cable, coaxial cable, wire, or any other suitable type of connection).

[0168] Furthermore, as used herein, "computer-readable medium 1220" can be singular or plural. For example, program code 1218 may be located in computer-readable medium 1220 in the form of a single storage device or system. In another example, program code 1218 may be located in computer-readable medium 1220 distributed across multiple data processing systems. In other words, some instructions in program code 1218 may be located in one data processing system, while other instructions in program code 1218 may be located in a data processing system. For example, a portion of program code 1218 may be located in computer-readable medium 1220 in a server computer, while another portion of program code 1218 may be located in computer-readable medium 1220 in a group of client computers.

[0169] The different components shown for data processing system 1200 do not imply an architectural limitation on how different embodiments can be implemented. In some illustrative examples, one or more components may be incorporated into or otherwise formed part of another component. For example, in some illustrative examples, memory 1206 or a portion thereof may be incorporated into processor unit 1204. Different illustrative embodiments may be implemented in data processing systems that include components other than or in lieu of those shown for data processing system 1200. Figure 12 Other components shown may differ from the illustrative example shown. Different embodiments can be implemented using any hardware device or system capable of running program code 1218.

[0170] Therefore, illustrative embodiments of the present invention provide a computer-implemented method, computer system, and computer program product for managing information. In one illustrative example, a computer system identifies a set of bucket hashes and comparison information for a data record. The set of bucket hashes is generated based on the comparison information, wherein the set of bucket hashes and comparison information forms a metadata record. The computer system uses the set of bucket hashes to identify multiple candidate metadata records in a metadata database, wherein the multiple candidate metadata records include the set of candidate bucket hashes and candidate comparison information. The computer system identifies the entity membership of the data record by comparing the comparison information in the metadata record with the candidate comparison information in the multiple candidate metadata records.

[0171] Metadata records in a metadata database enable at least one of faster searches or fewer database accesses when comparing data records to determine if a record is a match. In the illustrative example, the metadata record includes a bucket hash and comparison information. Additionally, the metadata record may include entity membership information. Therefore, a search can be performed to identify metadata records with a bucket hash that matches the metadata record being processed. The returned metadata records are candidate metadata records for comparison. In the illustrative example, these metadata records may also include comparison information. As a result, comparisons can be made with results already returned in the initial search without performing another search to obtain the comparison information. Furthermore, when a match is identified, the metadata record being processed or the candidate metadata records that have been returned can be updated to link the metadata records without forming another search.

[0172] Furthermore, illustrative examples can include information such as source identifiers and source record identifiers in metadata records to improve the speed at which the data record can be located. This reduces the number of searches and saves time required to process many data records.

[0173] For illustrative and descriptive purposes, descriptions of various illustrative embodiments have been presented and are not intended to be exhaustive or limited to the embodiments disclosed. The various illustrative examples describe components that perform actions or operations. In the illustrative embodiments, components may be configured to perform the described actions or operations. For example, a component may have a configuration or design that provides the component with the ability to perform actions or operations described in the illustrative examples as being performed by the component. Furthermore, within the scope of the terms “comprising,” “including,” “having,” “containing,” and variations thereof used herein, these terms are intended to be inclusive in a manner similar to the term “comprising” as an open-ended transitional term, without excluding any additional or other elements.

[0174] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Not all implementations will include all features described in the illustrative examples. Furthermore, different illustrative embodiments may provide different features compared to other illustrative embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for managing information, the method comprising: A set of bucket hashes and comparison information for identifying data records by a computer system, wherein the set of bucket hashes is generated from the comparison information, and wherein the set of bucket hashes and the comparison information form a metadata record; The computer system uses this set of bucket hashes to identify multiple candidate metadata records in the metadata database, wherein the multiple candidate metadata records include the set of candidate bucket hashes and candidate comparison information; as well as The computer system determines the entity membership of the data record by comparing the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records. The computer system receives the search request from the requester; The computer system uses the search terms in the search request received from the requester to create a set of search bucket hashes and search comparison information; The computer system uses this set of search bucket hashes to search for a set of candidate metadata records in the metadata database; The computer system selects candidate metadata records from the set of candidate metadata records based on the set of comparison scores, where the set of comparison scores indicates how similar the candidate comparison information in the set of candidate metadata records is to the search comparison information; as well as The computer system uses the source identifier in the candidate metadata record to return the identified data record corresponding to the candidate metadata record to the requester, wherein the bucket hash of the metadata record of the data record and the comparison information are received from the index service that generated the bucket hash of the data record and the comparison information.

2. The method according to claim 1, further comprising: The computer system uses this set of bucket hashes and the comparison information to create metadata records for the data records; as well as The computer system updates the metadata record using the entity membership determined for the data record.

3. The method according to claim 1, wherein, The computer system determines the entity membership of the metadata record by comparing the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records, including: The computer system compares the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records to form a comparison; The computer system generates a set of comparison scores based on the comparison; and The computer system determines the entity's membership based on a comparison of the set of scores with a set of thresholds.

4. The method according to claim 1, wherein, It is possible to generate multiple bucket hashes in the set of bucket hashes for a single data record, and each of the multiple bucket hashes represents only a portion of the attributes in the data record.

5. The method according to claim 1, wherein, The comparison information includes at least one of a comparison string or a text string representing information in the data record.

6. The method according to claim 1, wherein, The bucket hash is based on at least one of the following: first name, last name, address, street name, state, zip code, city of residence, telephone number, occupation, business name, date of birth, account number, occupation, business telephone number, business address, or tax identification number.

7. The method according to claim 1, wherein, The metadata record further includes at least one of the following: entity membership, customer identifier, record type, probability self-rating, or timestamp.

8. An information management system, comprising: Computer systems; as well as A metadata database, stored in the computer system, wherein each metadata record in the metadata database includes a set of bucket hashes, comparison information, and entity membership, wherein the set of bucket hashes is generated using the comparison information, and the entity membership identifies a set of related metadata records. The computer system receives a search request from a requester; creates a set of search bucket hashes and search comparison information using the search terms received in the search request from the requester; searches a set of candidate metadata records in the metadata database using the set of search bucket hashes; selects candidate metadata records from the set of candidate metadata records based on a set of comparison scores indicating how similar the candidate comparison information in the set of candidate metadata records is to the search comparison information; and returns an identified data record corresponding to the candidate metadata record to the requester using a source identifier in the metadata record, wherein the set of bucket hashes and the comparison information of the metadata record of the data record are received from an indexing service that generated the set of bucket hashes and the comparison information of the data record.

9. The information management system according to claim 8, wherein, The computer system identifies the set of bucket hashes and the comparison information of the data record; generates the set of bucket hashes from the comparison information in the data record; and forms a metadata record with the set of bucket hashes and the comparison information. It uses the set of bucket hashes to identify multiple candidate metadata records in the metadata database, wherein the multiple candidate metadata records include candidate bucket hashes and candidate comparison information. It then determines the entity membership of the data record based on a comparison between the comparison information in the metadata record and the candidate comparison information in the multiple candidate metadata records.

10. The information management system according to claim 9, wherein, The computer system uses the set of bucket hashes and the comparison information to create a metadata record for the data record, and uses the entity membership determined for the data record to update the metadata record.

11. The information management system according to claim 9, wherein, In determining the entity membership of a metadata record by comparing the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records, the computer system compares the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records to form a comparison; and generates a set of comparison scores from the comparison. And determine the entity's membership based on a comparison of the set of comparative scores with a set of thresholds.

12. The information management system according to claim 8, wherein, The comparison information includes at least one of a comparison string or a text string representing information.

13. The information management system according to claim 9, wherein, The metadata record further includes at least one of the following: entity membership, customer identifier, record type, probability self-rating, or timestamp.

14. A computer program product for managing information, the computer program product comprising: Computer-readable storage medium; First program code, stored on the computer-readable storage medium, is executable by a computer system to enable the computer system to identify a set of bucket hashes and comparison information for a data record, wherein the set of bucket hashes is generated from the comparison information, and wherein the set of bucket hashes and the comparison information form a metadata record; The second program code, stored on the computer-readable storage medium and executable by the computer system, enables the computer system to use the set of bucket hashes to identify multiple candidate metadata records in the metadata database, wherein the multiple candidate metadata records include the set of candidate bucket hashes and candidate comparison information; as well as Third program code, stored on the computer-readable storage medium and executable by the computer system, enables the computer system to determine the entity membership of the data record based on a comparison of the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records; The fourth program code, stored on a computer-readable storage medium, can be executed by a computer system to enable the computer system to receive a search request from the requester; Fifth program code, stored on a computer-readable storage medium, executable by a computer system to enable the computer system to create a set of search bucket hashes and search comparison information using search terms from the search request received from the requester; The sixth program code, stored on the computer-readable storage medium and executable by the computer system, enables the computer system to use the set of search bucket hashes to search for a set of candidate metadata records in the metadata database; A seventh program code, stored on the computer-readable storage medium, executable by the computer system, causes the computer system to select candidate metadata records from the set of candidate metadata records based on a set of comparison scores, the set of comparison scores indicating how similar the candidate comparison information in the set of candidate metadata records is to the search comparison information; as well as The eighth program code, stored on the computer-readable storage medium, is executable by the computer system to cause the computer system to return a data record corresponding to the identified candidate metadata record to the requester using the source identifier in the candidate metadata record, wherein the bucket hash of the metadata record of the data record and the comparison information are received from an indexing service that generated the bucket hash of the data record and the comparison information.

15. The computer program product of claim 14, further comprising: A fourth piece of program code, stored on the computer-readable storage medium and executable by the computer system, enables the computer system to create a metadata record for the data record using the set of bucket hashes and the comparison information; as well as A fifth piece of program code, stored on the computer-readable storage medium, is executable by the computer system to cause the computer system to update the metadata record using the entity membership determined for the data record.

16. The computer program product according to claim 14, wherein, The third program code includes: Program code stored on the computer-readable storage medium, the program code being executable by the computer system to cause the computer system to compare the comparison information in the metadata record with the candidate comparison information in the plurality of candidate metadata records to form a comparison; Program code stored on the computer-readable storage medium, the program code being executable by the computer system to cause the computer system to generate a set of comparison scores based on the comparison; and Program code stored on the computer-readable storage medium, the program code being executable by the computer system to enable the computer system to determine the entity's membership based on comparing the set of comparative scores with a set of thresholds.

Images