Method and system for processing data records

Abstract

The present disclosure relates to a method comprising: providing one or more sets of records, each record of the set of records having a set of one or more attributes, inputting values of the set of attributes of the set of records to a trained data representation learning model, receiving as output of the trained data representation learning model a set of feature vectors respectively representing the set of records, and storing the set of feature vectors.

Description

Background Technology

[0001] This invention relates to the field of digital computer systems, and more specifically, to a method for processing data records.

[0002] Storing and processing data can be a prerequisite for the successful operation of a data management system. For example, in the data cleaning process, removing duplicate records or finding matches in the database is a critical step, as duplicates can severely impact the results of any subsequent data processing or data mining. With the increasing complexity involved in the large number of different attributes required across different geographical regions, countries, etc., the matching process for record linking becomes more complex and constitutes one of the main challenges for record linking algorithms. Summary of the Invention

[0003] Various embodiments provide methods, computer systems, and computer program products for processing data records as described in the independent claims. Advantageous embodiments are described in the dependent claims. Embodiments of the invention may be freely combined with each other if they are not mutually exclusive.

[0004] In one aspect, the present invention relates to a method comprising: providing a set of one or more records, each record in the set of records having a set of one or more attributes; inputting the values ​​of the set of attributes of the set of records into a trained data representation learning model; thereby receiving a set of feature vectors, each representing the set of records, as the output of the trained data representation learning model; and storing the set of feature vectors.

[0005] In another aspect, the present invention relates to a computer program product comprising a computer-readable storage medium having computer-readable program code configured to implement all steps of the method according to the foregoing embodiments.

[0006] In another aspect, the present invention relates to a computer system configured to: input the values ​​of a set of attributes of a set of one or more records into a trained data representation learning model, thereby receiving a set of feature vectors representing the set of records as the output of the trained data representation learning model, and storing the set of feature vectors.

[0007] This topic enables efficient storage and representation of data records in a database. The resulting set of feature vectors can offer advantages such as saving storage resources and providing compact or alternative storage solutions for storing data records. Furthermore, the generated set of feature vectors can optimize data record processing by using data records instead of records. For example, matching feature vectors instead of records can be fast, efficient, and reliable. Using a trained data representation learning model can provide an accurate and controllable definition of the vector space used to represent records. For example, a vector space can be defined such that repeating feature vectors can be identified through simplified vector operations within the vector space. Example vector operations could be distance functions. Attached Figure Description

[0008] The embodiments of the present invention will now be explained in more detail by way of example and with reference to the accompanying drawings, wherein:

[0009] Figure 1 This is a block diagram of a computer system based on examples from this topic.

[0010] Figure 2 This is a flowchart illustrating a method for storing data, based on an example from this topic.

[0011] Figure 3 This is a flowchart of a method for matching records, based on an example from this topic.

[0012] Figure 4 This is a flowchart illustrating the inference method based on examples from this topic.

[0013] Figure 5 This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic.

[0014] Figure 6A This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic.

[0015] Figure 6B This is a diagram illustrating the training of an example data representation learning model.

[0016] Figure 7A This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic.

[0017] Figure 7B This is a graph showing the training of an example of a data representation learning model.

[0018] Figure 8 This is a diagram illustrating a method for storing feature vectors, based on an example of this topic.

[0019] Figure 9 This is a diagram illustrating the prediction process based on an example of this topic.

[0020] Figure 10 This is a flowchart of a method for matching records, based on an example from this topic.

[0021] Figure 11 This is a diagram of a system for matching records, based on an example from this topic.

[0022] Figure 12 This refers to a computerized system adapted to implement one or more method steps as described in this disclosure. Detailed Implementation

[0023] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0024] A data record, or record, is a collection of related data items such as a specific user's name, date of birth, and category. A record represents an entity, where an entity refers to a user, object, or concept, and information about it is stored in the record. The terms "data record" and "record" are used interchangeably.

[0025] A dataset can include a collection of records processed by this topic. For example, a dataset can be provided as a collection of related records contained in a file; for instance, a dataset could be a file containing records of all students in a class. A dataset can be, for example, a table in a database or a file in a Hadoop file system. In another example, a dataset can include one or more documents such as HTML pages or other document types.

[0026] The dataset may be stored, for example, in a central repository. A computer system may be configured to access the central repository, which may or may not be part of the computer system. The central repository may be a storage device or storage unit that stores data received from multiple client systems. The dataset may include a subset of existing records in the central repository, which are identified or selected for processing by this method. For example, a subset of records may be selected based on the values ​​of one or more attributes of the records; for example, the subset of records may represent entities belonging to a given country or region. The records of the dataset may be preprocessed, for example, before being processed by this method. Preprocessing may include, for example, converting the format of the attribute values ​​of the records in the dataset. For example, attribute values ​​may be uppercase, their interfering characters (e.g., - / characters) may be removed, and anonymous attribute values ​​(e.g., city=nowhere or name=Test) may be removed.

[0027] A data representation learning model is configured to output feature vectors of records such that the distance between two feature vectors generated by the model based on two records is a measure of the similarity of the attributes of the two records. For example, a data representation learning model can be trained to take records as input and produce a vector space typically d-dimensional, where each unique record is assigned a corresponding vector in the space. Feature vectors can lie in a vector space such that records with similar or shared attribute values ​​are located close to each other in the space. Feature vectors can be ordered arrays of single digits. A feature vector can be an example of a first-order tensor.

[0028] In one example, the trained data representation learning model could be a similarity encoder configured to receive a set of one or more attributes of a record and generate a feature vector representing the record. For example, the feature vector could be obtained by collectively processing the values ​​of the set of attributes, or it could be obtained as a combination of individual feature vectors associated with the set of attributes, where the similarity encoder is configured to continuously receive the values ​​of the set of attributes and generate associated individual feature vectors.

[0029] In another example, the trained data representation learning model may include a similarity encoder for each attribute in the set of attributes of a record, wherein each similarity encoder is configured to receive the corresponding attribute in the set of attributes and generate a corresponding individual feature vector. Individual feature vectors can be combined to obtain a feature vector representing the record.

[0030] In another example, the trained data representation learning model can be a neural network configured to receive a set of one or more attributes of a record and generate a feature vector representing the record. In yet another example, the trained data representation learning model can include a neural network for each attribute in the set of attributes of a record, wherein each neural network is configured to receive the corresponding attribute in the set of attributes and generate a corresponding individual feature vector. Individual feature vectors can be combined to obtain a feature vector representing the record.

[0031] Combining individual feature vectors can be achieved by concatenating or appending elements of individual feature vectors to a feature vector, so that the feature vector includes elements of the individual feature vectors.

[0032] This approach can be deployed and used, for example, in big data solutions (e.g., IBM BigMatch technology running on IBM BigInsight, Cloudera, and Hortonworks) and incorporates information integration software (e.g., Informatica Power Center, IBM Information Server). The computer system can be, for example, a Master Data Management (MDM) system.

[0033] According to one embodiment, the method further includes: receiving records having a set of attributes. The values ​​of the set of attributes of the received records can be input into a trained data representation learning model to obtain feature vectors of the received records from the trained data representation learning model. The obtained feature vectors can be compared with at least a portion of the set of feature vectors to determine a matching level between the obtained feature vectors and the set of feature vectors. The obtained feature vectors and / or the received records can be stored based on the matching level. In one example, the obtained feature vectors can be compared with each feature vector in the set of feature vectors. In another example, the obtained feature vectors can be compared with each feature vector of a selected subset of the set of feature vectors.

[0034] Conventional matching algorithms can use bucketing features. However, if the buckets are too large (exceeding 1000-1500 records), matching performance can degrade to the point where matching may become unusable in real time. Furthermore, creating good bucketing strategies to avoid such large buckets may require deep, hard-to-find human subject matter experts. Additionally, re-bucketing may require human expertise and several tuning iterations (e.g., 4-6 weeks). This embodiment addresses these problems by using a vector space that makes it easy to match feature vectors in large buckets. Matching according to this embodiment is not limited to a low threshold number of attributes, as distance computation at the vector level allows for matching a high number of attributes. Another advantage of using feature vectors is as follows. Comparison functions such as edit distance and speech distance work well on simple attributes like name attributes. However, these comparison functions may not work for free text fields like a 200-word product description. This can be solved using feature vectors. For this purpose, the content of the free text description is encoded in vectors, and close vectors refer to similar free text descriptions.

[0035] According to one embodiment, storing the set of feature vectors includes clustering the set of feature vectors into clusters and associating each stored feature vector with clustering information indicating the corresponding cluster. Storing feature vectors as clusters can be advantageous because it allows for optimal access to the stored data based on the clusters. For example, criteria regarding clustering can be used to refine queries for accessing the stored data. This can accelerate data access.

[0036] According to one embodiment, storing a set of feature vectors includes clustering the set of feature vectors into clusters. The method further includes determining the distance between the obtained feature vectors and each cluster of the clusters, wherein at least a portion of the set of feature vectors includes clusters of feature vectors having the closest distance to the obtained feature vectors. That is, a selected subset of the set of feature vectors includes feature vectors of clusters of feature vectors having the closest distance to the obtained feature vectors. The distance between a cluster and the obtained feature vectors can be the distance between the obtained feature vectors and the vectors representing the clusters of feature vectors. The vector representing the cluster can be, for example, the centroid of the cluster of feature vectors. In another example, any feature vector of a cluster can be the vector representing the cluster.

[0037] This embodiment can accelerate the matching process and thus save processing resources that would otherwise be needed to match each stored feature vector. Therefore, the matching process implemented by this embodiment can be used, for example, in real time during the operation of the MDM system.

[0038] According to one embodiment, the method is executed in real time, wherein a record is received as part of a create or update operation. For example, the method can be executed while the MDM system is running. For example, in response to receiving a data record, the method can be executed automatically on that record. For example, to run a record matching process in real time within a create / update operation, the matching process might need to complete in 100 milliseconds or less. Using feature vectors according to this subject can make it possible to reduce the matching processing time to the level required for real-time processing.

[0039] According to one embodiment, the trained data representation learning model is configured to output a feature vector from the set of feature vectors by generating individual feature vectors for each attribute in the set of attributes and combining the individual feature vectors to obtain a feature vector. By processing attributes at the individual level, this embodiment enables more detailed access to the features of a record and thus provides a reliable representation of the record.

[0040] Data representation learning models may include, for example, a similarity encoder for each attribute in a set of attributes, wherein the similarity encoder is configured to output individual feature vectors for associated attributes in the set of attributes, such that the distance between two individual feature vectors generated by the similarity encoder based on two attributes is a measure of the similarity between the two attributes.

[0041] According to one embodiment, the trained data representation learning model is configured to receive input values ​​and output a set of feature vectors in parallel. That is, the data representation learning model is configured to process the values ​​of a set of attributes in parallel (or concurrently) and generate outputs in parallel (or concurrently). For example, this can further accelerate the matching process based on the topic.

[0042] According to one embodiment, a trained data representation learning model includes a set of attribute-level trained data representation models associated with a set of attributes, wherein the output of each feature vector in the set of feature vectors includes: inputting the value of each attribute of the set of attributes into the associated attribute-level trained data representation model; receiving an individual feature vector from each of the attribute-level trained data representation models in response to the input; and combining the individual feature vectors to obtain the feature vector. The combination of individual feature vectors may be an element-wise concatenation or appending of individual feature vectors to the feature vector, such that the feature vector includes elements of the individual feature vectors.

[0043] This embodiment enables the construction of data representation learning models as pipelines. Each attribute being compared belongs to a pipeline. Each pipeline predicts the similarity of its attributes. This can be advantageous because it provides a scalable structure. This allows for the handling of an ever-growing number of attributes by adding new pipelines. For example, a pipelined approach could allow a customer to add new pipelines to predict attributes unique to that customer (e.g., a unique ID for a specific customer).

[0044] According to one embodiment, the trained data representation learning model further includes trained weighted reassemblies associated with sets of attributes, wherein the combination includes weighting each of the individual feature vectors using corresponding trained weights in the trained weighted reassemblies, and combining them, for example, by cascading the weighted individual feature vectors. This allows the prediction process at the attribute level to be controlled by changing the set of weights according to the corresponding attribute of the weight. For example, during training, the weights associated with a given attribute can be changed differently compared to other attributes. This weighting process occurs as part of the learning steps in the data representation learning model.

[0045] This embodiment also allows customers to weight the impact of different comparisons. For example, a customer may have high-quality address data. Therefore, comparisons of the address field may have a significant impact on the overall prediction. The customer can therefore change the trained weights associated with the address data.

[0046] According to one embodiment, the trained data representation model for each attribute level is a neural network.

[0047] According to one embodiment, a trained data representation learning model is trained to optimize a loss function. In one example, backpropagation is used to train the trained data representation learning model to optimize the loss function. The loss function can indicate a measure (or a function thereof) of the similarity between feature vectors of a pair of records. The similarity measure can be a combination of individual similarities. Each similarity in the individual similarity indicates the similarity between two individual feature vectors generated for the same attribute in the pair of records. The similarity measure between two feature vectors can be, for example, Euclidean distance, cosine distance, or the difference between pairwise elements of the two feature vectors.

[0048] According to one embodiment, a trained data representation learning model is trained to optimize a loss function, where the loss function is a measure of the similarity between feature vectors of a pair of records. In one example, backpropagation is used to train the trained data representation learning model to optimize the loss function.

[0049] According to one embodiment, the trained data representation learning model includes at least one neural network trained according to a Siamese neural network architecture. For example, the neural network of the trained data representation learning model can be one of two networks of a trained Siamese neural network. This can be advantageous because Siamese neural networks (SiNNs) achieve non-linear embedding of data with the aim of obtaining a semantically meaningful space where relevant patterns (e.g., records of the same entity) are close to each other, while avoiding the closeness of semantically unrelated patterns (e.g., records of different entities). Comparisons between feature vectors can be simplified using distance. In another example, the trained data representation learning model includes an autoencoder.

[0050] According to one embodiment, the trained data representation learning model includes a trained neural network for each attribute in a set of attributes, wherein the output of each feature vector in a set of feature vectors includes: inputting the value of each attribute of the set of attributes into the associated trained neural network, receiving individual feature vectors from each of the trained neural networks in response to the input, and combining the individual feature vectors to obtain a feature vector.

[0051] According to one embodiment, the method further includes: receiving a training dataset comprising pairs of records having attributes, wherein each pair is labeled, and using the training dataset to train a data representation learning model to generate a trained data representation learning model. The data representation learning model can be trained in a supervised manner using the record pairs, wherein each record pair is labeled by a predefined label. The label of the record pair can indicate "same" or "different." "Same" means that the record pair represents the same entity, while "different" means that the two records belong to different physical entities (e.g., people).

[0052] According to one embodiment, the data representation learning model includes a set of attribute-level data representation models, each associated with a set of attributes. Training the data representation learning model includes, for each pair of records and for each attribute in the set of attributes: inputting the attribute value pairs (in the record pair) into the corresponding attribute-level data representation model to obtain individual feature vector pairs; calculating individual similarity levels between individual feature vector pairs; and weighting the individual similarity levels using trainable weights of the attributes. Training the data representation learning model also includes, for each pair of records: determining a measure of similarity between the feature vectors of the record pair as a combination of individual weighted similarity levels; and using the measure to evaluate a loss function. The evaluated loss function can be used in a minimization process during training of the data representation learning model. The combination of individual weighted similarity levels can be the sum of individual weighted similarity levels.

[0053] During training, the trainable parameters of the attribute-level data representation model set and the weight set can be changed to achieve the optimal value of the loss function. For example, the weights can increase the importance of an attribute relative to another. To this end, the weight set can be changed differently for the set of attributes during training. In the first weight-changing configuration, the set of attributes can be ranked according to user-defined priorities, and the weights can be changed using an amount (deltavalue) that depends on their ranking. In another example, the user can change the trained weights during the test / inference phase; for example, the user can increase or decrease one or more of the trained weights. This provides the opportunity to adjust the weights according to user-defined priorities.

[0054] According to one embodiment, the set of attributes includes a first subset of attributes and a second subset of attributes. The method further includes: receiving a first trained data representation learning model, the first trained data representation learning model including a first subset of attribute-level trained data representation learning models associated with the first subset of attributes, wherein the first trained data representation learning model is configured to receive values ​​of the first subset of recorded attributes and output feature vectors of the records by: inputting the value of each attribute of the first subset of attributes into the associated attribute-level trained data representation model of the first subset of attribute-level trained data representation learning models; receiving individual feature vectors from each attribute-level trained data representation model of the first subset of attribute-level trained data representation learning models; and combining the individual feature vectors to obtain a feature vector. A second subset of attribute-level data representation learning models can be provided for the second subset of attributes, wherein each attribute-level data representation learning model of the second subset is configured to generate feature vectors for the corresponding attributes of the second subset of attributes. A data representation learning model can be created such that it includes the first trained data representation learning model and the second subset of attribute-level data representation learning models. The created data representation learning model can be trained to generate a trained data representation learning model.

[0055] The first-trained data representation learning model can be generated centrally by, for example, a service provider. This first-trained data representation learning model can be provided to multiple users or clients. Each user can use the first-trained data representation learning model based on this topic to adapt it to their needs in an effective and controlled manner. For example, a user can add one or more new pipelines associated with new user-specific attributes.

[0056] According to one embodiment, the first trained data representation learning model further includes a first subset of trained weights associated with a first subset of attributes, wherein the first subset of trained weights is used to perform combination, for example, weighting individual feature vectors by corresponding weights and combining the weighted individual feature vectors. The created data representation learning model also includes a second subset of trainable weights associated with a second subset of attributes, wherein the trained data representation learning model is configured to output each feature vector in the set of feature vectors by: inputting the value of each attribute in the set of attributes into the associated attribute-level trained data representation model of the attribute-level trained data representation learning model of the first and second subsets; receiving individual feature vectors from each attribute-level trained data representation model of the first and second subsets of the attribute-level trained data representation learning model; and combining the individual feature vectors using the corresponding weights of the weights of the first and second subsets to obtain a feature vector. The first subset and the second subset of weights can be changed differently during training.

[0057] Figure 1 An exemplary computer system 100 is depicted. Computer system 100 may be configured, for example, to perform master data management and / or data warehousing; for example, computer system 100 may implement a de-duplication system. Computer system 100 includes a data integration system 101 and one or more client systems or data sources 105. Client system 105 may include a computer system (e.g., as described with reference to FIG. 6). Client system 105 may communicate with data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a WAN (wide area network) connection, a LAN (local area network) connection, the Internet, or a combination thereof. Data integration system 101 may control access to a central storage 103 (read and write access, etc.).

[0058] The dataset of records 107 stored in the central repository 103 may have attributes such as first name (a1...a) N The values ​​of a set (N ≥ 1). Although this example is described with respect to a few attributes, more or fewer attributes may be used. The dataset 107 used in this topic may include at least a portion of the records in the central repository 103.

[0059] Data records stored in central repository 103 can be received from client system 105 and processed by data integration system 101 (e.g., to convert them into a uniform structure) before being stored in central repository 103. For example, records received from client system 105 may have a different structure than the records stored in central repository 103. For example, client system 105 may be configured to provide records in XML format, JSON format, or other formats that can associate attributes with corresponding attribute values.

[0060] In another example, data integration system 101 may import data records from client system 105 into central repository 103 using one or more extract-transform-load (ETL) batch processes or via Hypertext Transfer Protocol (“HTTP”) communication or other types of data exchange.

[0061] Data integration system 101 can be configured to process received records using one or more algorithms. Data integration system 101 may, for example, include a trained data representation learning model 120. For example, a trained data representation learning model 120 may be received from a service provider. In another example, a trained data representation learning model 120 may be created at data integration system 101. The trained data representation learning model 120 is configured to generate feature vectors representing a given record. The given record has, for example, a set of attributes. To this end, the trained data representation learning model 120 may be configured to receive values ​​of all or some of the attributes of the record and generate feature vectors.

[0062] In one example, the trained data representation learning model 120 may include trained data representation learning models 121.1-121.N at multiple attribute levels. Each attribute level of the trained data representation learning model 121.1-121.N may be associated with attribute a1...a N The corresponding attributes in the set are associated. The trained data representation learning model 121.1-121.N for each attribute level can be configured to receive the corresponding attribute a1 ... a N The values ​​are then used to generate corresponding individual feature vectors. Individual feature vectors can be combined to obtain a single feature vector representing a record. For example, a feature vector can be generated using N attributes a1... a N The set of associated N trained weights a1-aN is used to perform the combination.

[0063] Figure 2 This is a flowchart illustrating a method for storing data, based on examples from this topic. For illustrative purposes, Figure 2 The method described in [the document] can be used in [the following context] Figure 1 The system shown in the diagram is implemented, but is not limited to this implementation. Figure 2 The method can be performed, for example, by the data integration system 101.

[0064] In step 201, K records R1 ... R can be provided. K A set. For example, a dataset can include K records R1... R K A set of K records. Each record in the set has N attributes a1 ... a N A set of K records, N ≥ 1. In one example, each record in the set of K records may include an attribute (i.e., N = 1). Using an attribute to generate a feature vector based on the topic can be advantageous because the feature vector can be used to describe specific characteristics of the entity being studied. For example, this can make it possible to cluster or match records representing students of the same age or from the same region, etc.

[0065] In another example, each record in the collection of records can include multiple attributes. The collection of records can share attributes a1 ... a N A subset of one or more attributes in a set, and therefore may or may not have the same attribute a1 ... a N The entire set. Using all attributes allows for the description of the entire record; for example, a feature vector can provide a global representation of features. This can be advantageous for detecting duplicate records.

[0066] In one example, the set of records could include all records from an existing database, such as repository 103. This allows for the provision of feature vectors for all existing records. In another example, the set of records could include a single record. This record could, for example, be a newly received record. This could be advantageous, for instance, when creating a new database.

[0067] In step 203, the values ​​of the set of attributes of the record set can be input into the trained data representation learning model 120. And in step 205, the output of the trained data representation learning model 120 can be received. The output includes K feature vectors F1 ... F2 representing the set of records. K The set. Steps 203 and 205 enable the inference of the trained data representation learning model 120. Therefore, the two steps 203 and 205 can be collectively referred to as the inference steps.

[0068] In the first inference example, the trained data representation learning model 120 can be configured to process each record R of the set of records as follows. i (i = 1, 2, ... or K). Record R iThe values ​​of the set of attributes can be input into the trained data representation learning model 120 all at once (e.g., in parallel). The trained data representation learning model 120 can be configured to generate values ​​for the received records R. i The generated feature vector represents the value of the record R. i In this example, the trained data representation learning model 120 may include, for example, a neural network.

[0069] In the second inference example, the trained data representation learning model 120 can be configured to process each record R in the set of records as follows. i Record R i The values ​​of the set of attributes can be fed into a trained data representation learning model 120 at a time. The trained data representation learning model 120 can be configured to generate an individual feature vector for each received value of the set of attributes. The individual feature vectors can be combined using the trained data representation learning model 120 to generate a representation record R. i The feature vectors of an individual can be generated from attribute-level data representation learning models 120, which are each associated with a set of attributes.

[0070] In the third inference example, the trained data representation learning model 120 can be configured to process record R as follows: i Each record in the set. Record R i The values ​​of the set of attributes can be continuously input into the trained data representation learning model 120. The trained data representation learning model 120 can be configured to generate an individual feature vector for each received value. The individual feature vectors can be combined using the trained data representation learning model 120 to generate a representation record R. i The feature vectors. This example can be particularly advantageous when the set of attributes is of the same type. That is, a single trained data representation learning model (e.g., a single neural network) can efficiently generate feature vectors for different attributes of the same type. For example, if the set of attributes includes a person's business and personal phone numbers, the trained data representation learning model 120 can be used to (continuously) generate feature vectors for both attributes. For example, if the entity is a product and the set of attributes is height and width, the trained data representation learning model 120 can be used to (continuously) generate feature vectors for both attributes.

[0071] Combinations of individual feature vectors can be achieved by concatenating or appending elements of individual feature vectors to feature vectors, such that the feature vectors include elements of individual feature vectors.

[0072] The inferences of the trained data representation learning model 120 can generate K feature vectors F1 ... F2 representing the set of records provided in step 201. K The set of eigenvectors. Each eigenvector in the set can be represented in a d-dimensional mathematical space. The set of eigenvectors can be stored in step 207. This allows for improved storage utilization.

[0073] In the first storage example, a set of feature vectors can be stored instead of a set of records. This can provide a lightweight version of the database. This can be particularly advantageous when the database is used for a specific purpose that the feature vectors can satisfy, and not all records are required.

[0074] In the second storage example, the set of feature vectors can be stored in association with the corresponding set of records. This can be particularly advantageous because the set of feature vectors may not require significant storage resources. This makes it possible to provide the set of feature vectors as metadata for the set of records.

[0075] In the third storage example, the set of feature vectors can be clustered into clusters. Clustering information can be determined for each cluster within the clusters. For example, the clustering information can be a cluster ID or the centroid of the cluster, and the set of feature vectors can be stored in association with the clustering information of the clusters to which they belong.

[0076] Figure 3 This is a flowchart illustrating a method for matching records, based on an example from this topic. For illustrative purposes, Figure 3 The method described in [the document] can be used in [the following context] Figure 1 The system shown in the diagram is implemented, but is not limited to this implementation. Figure 3 The method can be performed, for example, by the data integration system 101.

[0077] In step 301, record R can be received. K+l The received records have attributes a1 ... a respectively. N The set corresponds to N attributes A1 ... A N A collection. Attribute A1 ... A N The number of attributes in the set is equal to Figure 2 The number of attributes in the set of N attributes used; however, N attributes A1 ... A N A set can be associated with N attributes a1 ... a N The sets are either completely identical or not completely identical. For example, each attribute A i It can be equivalent to or exactly the same as, for example, an attribute that is both a "name" attribute and a "personal name" attribute, or an attribute that is both a "private phone name" attribute and a "business phone number" attribute. For example, each attribute Ai Can be with a i Equivalent or identical; for example, a1 can be the "Name" attribute and A1 is the "Signature" attribute, or a1 can be the "Private Phone Name" attribute and A1 is the "Business Phone Number" attribute.

[0078] For example, the record R can be received in a data request, for example, from client system 105. K+l The data request can be, for example, an update or creation operation request. The received record can be structured or unstructured. In the case of the received unstructured record, such as an article, step 301 may further include processing the unstructured record to identify attribute values ​​of a set of attributes encoded in the received record. In another example, the request can be a matching request to match a record with a set of records. In both cases, matching a record with a set of records may be necessary, for example, by matching it with existing records before storing the received record to prevent the storage of duplicate records.

[0079] In step 303, the received record R can be... K+l N attributes A1 ... A N The values ​​of the set are input into the trained data representation learning model 120. And in step 305, the output of the trained data representation learning model 120 can be received. The output includes a representation of the received records R. K+l eigenvector F K+l .

[0080] In step 307, the received record R can be represented. K+l eigenvector F K+l With eigenvectors F1 ... F K At least a portion of the set is compared to determine the eigenvector F. K+l With eigenvectors F1 ... F K The matching level of the set. Comparisons can be performed based on a vector space defined by the trained data representation learning model. For example, if the training of the data representation learning model attempts to find a semantically meaningful space where related patterns (e.g., records of the same entity) are close to each other, this can be achieved by computing the feature vector F. K+l With eigenvectors F1 ... F K The comparison is performed by the distance between the feature vectors of the set. This distance can be, for example, Euclidean distance or cosine distance.

[0081] In the first comparison example, the feature vector F can be... K+l With eigenvectors F1 ... F KEach element in the set is compared. This may result in K similarity levels. The highest of the K similarity levels can be provided as the matching level in step 307. This can achieve accurate results.

[0082] In the second comparison example, the closest feature vector F can be identified. K+l Clustering of feature vectors. And the feature vector F can be clustered. K+l The similarity is compared with each feature vector of the identified clusters, resulting in multiple similarity levels, where the matching level is the highest among these similarity levels. This saves resources while still providing reliable comparison results.

[0083] In step 309, the feature vector F can be stored based on the matching level. K+l and / or received records R K+l For example, if the matching level is less than a predetermined threshold, the feature vector F can be stored. K+l and / or received records R K+l Otherwise, the feature vector F can be left unstored. K+l and received record R K+l .

[0084] Figure 3 The method can be used to receive record R K+l It executes automatically. In one example... Figure 3 The method can be executed in real time, for example, recording R. K+l It can be received as part of a create or update operation.

[0085] Figure 4 This is a flowchart illustrating the inference method based on examples from this topic. For illustrative purposes, Figure 4 The method described in [the document] can be used in [the following context] Figure 1 The system shown in the diagram is implemented, but is not limited to this implementation. Figure 4 The method can be performed, for example, by the data integration system 101.

[0086] Figure 4 The method provides Figure 2 An exemplary implementation of steps 203 and 205, specifically, Figure 4 The method enables recording R1 ... R K Each record R in the set i (i = 1, 2, ... or K) Generate feature vector F i .

[0087] In step 401, record R i N attributes a1 ... a N Each attribute a of the set jThe value of (j = 1, 2, ... or N) can be input into the associated attribute-level trained data representation model 121.j of the attribute-level trained data representation model 121.1 to 121.N.

[0088] In step 403, each receiving individual feature vector v from the attribute-level trained data representation model 121.1 to 121.N can be used. ij This could result in each record R i There are N individual feature vectors.

[0089] In step 405, records R can be combined. i N individual feature vectors v ij To obtain the representation record R i A single feature vector F i For example, the combination can be executed as follows: , where a j Is related to attribute a j The associated training weights, and A⊙B means concatenating or appending the elements of two feature vectors A and B to a single vector such that the single vector includes elements of A and B. For example, concatenating vectors [a, b] and [c, d] means making [a, b] and [c, d] into [a, b, c, d].

[0090] Figure 5 This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic.

[0091] In step 501, a training dataset may be received. The training dataset includes record pairs, such as similar record pairs. Each record pair in the training dataset may be associated with a label indicating whether the record pair represents the same entity or different entities. Each record in the training dataset has attributes a1 ... a N A set. For example, a training set can be obtained from one or more sources (e.g., 105).

[0092] The data representation learning model can be trained using the training dataset in step 503. This produces a trained data representation learning model. The data representation learning model can be, for example, an autoencoder or a deep neural network.

[0093] In the first training example, training can be performed to find a semantically meaningful vector space where the feature vectors of related records of the same entity are close to each other. This allows the original pairwise similarity between two records to be preserved in the vector space. This also allows distance to be used to measure the similarity between feature vectors. In this case, the Siamese neural network architecture can be advantageously used to train a data representation learning model as a deep neural network; for example, the deep neural network could be one of the two networks of SiNN.

[0094] In the second training example, training can be performed to find a semantically meaningful vector space, where the feature vectors of related records of the same entity can be identified by using pairwise differences of individual elements of two feature vectors within a predefined range. For example, if the differences are outside the range, this can indicate that the records are not of the same entity.

[0095] Figure 6A This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic. For a simplified description, please refer to... Figure 6B Examples to describe Figure 6A The method.

[0096] Data representation learning models can include N attributes a1 ... a N The set of N attribute-level data represents the model. Each attribute-level data representation model can be a neural network system 611.1-611.N, such as... Figure 6B As shown. Each of the neural network systems 611.1-611.N may include two neural networks, each associated with value pairs of the same attribute. The two networks may share the same weights as the SiNN. In one example, each of the neural network systems 611.1 to 611.N may be a Siamese neural network. Figure 6B As shown, the data representation learning model is structured into pipelines. Each attribute being compared (e.g., first name, last name, ...) belongs to a pipeline. Each pipeline predicts the similarity of its attributes. These pipelines are then weighted.

[0097] The training of data representation learning models can use methods including A pair of marked records The training dataset is used for execution, where m is in 1 and The changes between them. Figure 6B The record pair is shown and Example. Each record pair has N attributes a1 ... a NA set of N attribute value pairs. According to Figure 6B Example, record of It has N attribute value pairs: denoted by reference numerals 610.1 to 610.N respectively. , .

[0098] In step 601, the current record pair (e.g.) Each pair of N attribute values ​​from 610.1 to 610.N is input into the corresponding attribute-level data representation model from 611.1 to 611.N. This input can, for example, be performed in parallel. This can be advantageous because it can accelerate the training process. Figure 6B As shown, the value of attribute a1 is a pair The data input to the attribute level represents model 611.1. The value of attribute a2 is a pair. The data is input into the attribute-level trained data representation model 611.2, etc.

[0099] In response to receiving input, each of the attribute-level data representation models 611.1 to 611.N can output a corresponding individual feature vector pair 612.1 to 612.N in step 602. For example, attribute-level data representation model 611.1 can output the same individual feature vector pair 612.1 because the input values ​​are the same. Attribute-level data representation model 611.2 can output different individual feature vector pairs 612.2 because the input values ​​are different.

[0100] In step 603, each pair of individual feature vectors 612.1 to 612.N can be assigned their respective weights a1 ... a N Weighting is applied. This generates different weighted pairs of individual feature vectors in each pipeline, named v1 and v2. The weights increase the importance of one pipeline relative to another. Weights can be adjusted for specific customers based on customer configuration (number and selection of pipelines).

[0101] In step 604, the individual similarity level 613.1 to 613.N between each pair of weighted individual feature vectors can be calculated. For example, this can be performed by calculating the distance between the two weighted individual feature vectors in each pair. This is in... Figure 6B As shown, the output of each pipeline is determined by the distance... A quantified level of single similarity.

[0102] In step 605, an overall measure of the similarity between the feature vectors of the current record pair can be determined. This can be performed, for example, by a combination of individual weighted similarity levels 613.1 to 613.N.

[0103] For example, two methods can be used to determine the overall measurement. In the first method, individual vectors v1 and v2 can be concatenated for all attributes. This produces concatenated vectors v1 and v2. The Euclidean distance (scalar) between the concatenated vectors v1 and v2 can indicate the overall measurement. In the second method, the individual distance (scalar) between individual vectors v1 and v2 can be calculated for each attribute (e.g., as described in step 604). The sum of the individual distances can indicate the overall measurement.

[0104] Concatenation of vectors can include appending elements of one vector to the concatenated vector, such that the concatenated vector includes the elements of the first vector.

[0105] The loss function 616 can be evaluated in step 606 using the overall measurement determined in step 605. Steps 601 to 606 can be repeated for each record pair of at least a portion of the training dataset using backpropagation until the optimal value of the loss function is achieved. Figure 6B The term "e" shown x "It can be used to convert distance into probability and can be used as part of the loss function. During the training process, attribute-level data representations are learned to trainable parameters of the model from 611.1 to 611.N, as well as the weights a1 ... a of the individual vectors." N In the case that the learning models 611.1 to 611.N for each attribute level data representation are neural networks, they can include network weight reconfiguration, such as network weights from the input layer to the first hidden layer, from the first to the second hidden layer, etc. Before training the learning models 611.1 to 611.N for the attribute level data representations, the network weights can be initialized with random numbers or values. Training can be performed to search for optimized parameters (e.g., for network weights and biases) for the learning models 611.1 to 611.N for the attribute level data representations and to minimize the classification error or residual. For example, the training set can be used as input for feeding forward to each attribute level data representation learning model 611.1 to 611.N, which allows the data loss to be computed via a loss function. The data loss measures the compatibility between the prediction task and the ground truth labels. After obtaining the data loss, it can be minimized by changing the network weights and biases of each attribute level data representation learning model 611.1 to 611.N. For example, this can be performed by backpropagating the loss to each layer and neuron via gradient descent.

[0106] Figure 7A This is a flowchart illustrating a method for training a data representation learning model, based on examples from this topic. For a simplified description, please refer to... Figure 7B Examples to describe Figure 7A The method.

[0107] In step 701, a trained data representation learning model may be received. For example, in step 703, data may be received from... Figure 6A The trained data representation learning model is obtained through this method. Figure 6A The trained data representation learning model can be centrally generated by the service provider and can be used by different clients. In one example, the trained data representation learning model can be used without changing the client system, or it can be used as described above. Figures 7A-7B The trained data representation learning model is updated as described. For example, a user might need to use one or more additional attributes that are not used to generate the trained data representation learning model. Figure 7B As shown, the user may need to add additional attribute a. N+1 The additional attribute is the Employee ID attribute. Therefore, a user-specific data representation learning model can be created in step 703. The user-specific data representation learning model may include trained attribute-level trained data representation models 611.1 to 611.N and the additional attribute a. N+1 An associated additional attribute level is trained in the data representation model 611.N+1. For example, a user wanting to add a custom employee ID to the matching process can add a new pipeline to the structure of the received data representation model. During the training of the user-specific data representation learning model, the trained parameters of the trained data representation models 611.1 to 611.N for the attribute levels are frozen. This can be advantageous because instead of retraining the entire system, the client can add a new pipeline for an additional attribute and only need to train the network for that one attribute. The received trained weights a1 ... a N They are not frozen, and they can be retrained. Additionally, the added weights a... N+1 This is associated with an additional pipeline. Therefore, the user-specific data representation learning model has additional attribute levels. The trained data representation model 611.N+1 trainable parameters and N+1 trainable weights a1 ... a N+1 The trained parameters of the trained data representation model 611.1 to 611.N for the trained attribute levels are frozen and do not need to be changed during the training of the user-specific data representation learning model.

[0108] In step 705, a user-specific data representation learning model can be trained using a training set. The training set consists of pairs of records and associated labels, where each record in the training set has N+1 attributes a1 ... a2. N+1 The training of a learning model based on user-specific data can be done as shown in the reference. Figure 6AThe parameters of the trained data representing the attribute levels 611.1 to 611.N are fixed and do not change during training.

[0109] Figure 8 This is a diagram illustrating a method for storing feature vectors, based on an example of this topic.

[0110] Records 801.1 to 801.4 can be provided. Each of records 801.1 to 801.4 has attributes a1 ... a1 for generating the trained data representation learning model 803. N A set, for example, as referenced Figure 6A As described. Attribute a1 ... a of each record in records 801.1 to 801.4. N The values ​​of the set can be input into a trained data representation learning model 803 to generate a feature vector 804 (named SimVec or similarity vector) representing the record. The attributes a1 ... a N The values ​​of the set are input into the trained data representation learning model 803, which is a machine learning model trained to represent the corresponding attribute levels. For each input record, an individual feature vector is generated using an individual pipeline. Trained weights a1 ... a N Individual feature vectors are weighted. The weighted individual feature vectors are concatenated or combined to create feature vector 804. The resulting feature vector 804, recorded as 801.1 to 801.4, can be clustered using a clustering algorithm such as K-means. For example, the centroid of each cluster in the resulting clusters can be further determined. The feature vector 804, recorded as 801.1 to 801.4, can be stored in association with cluster information 807 indicating the cluster to which the feature vector belongs. For example, the cluster information could be a cluster index and / or the calculated centroids of the clusters. For example, it could be as referenced... Figure 9 Using the stored feature vectors as described, the clustering algorithm can be a trained algorithm. The clustering algorithm can be trained using feature vectors representing training records, where the feature vectors are generated by a data representation learning model trained on the subject.

[0111] Figure 9 This is a diagram illustrating a prediction process based on an example of this topic. For this purpose, references, for example, can be provided. Figure 6A The described trained data representation learning model 903. It can provide attributes a1 ... a1 for generating the trained data representation learning model 903. N The set contains record 901. Record 901 has attributes a1... a NThe values ​​of the set can be input into a trained data representation learning model 903 to generate a feature vector 904 representing record 901. Attributes a1 ... a N The values ​​of the set are input into the trained data representation learning model 903 to obtain the corresponding attribute levels of the trained machine learning model. Individual feature vectors are generated using an individual pipeline. Trained weights a1 ... a N Individual feature vectors are weighted. The weighted individual feature vectors are concatenated or combined to create feature vector 904 (or a similarity vector). Feature vector 904 is then compared with, for example, a reference vector. Figure 8 The centroids 908 of the created clusters 907 are compared to find the closest clusters. For this purpose, a distance 910 can be calculated between the feature vector 904 and each centroid 908, and the cluster associated with the minimum calculated distance can be the cluster associated with feature vector 904. After identifying the clusters associated with feature vector 904, possible matches between feature vector 904 and each feature vector 912 of the clusters associated with feature vector 904 can be determined. For this purpose, a matching metric 914 can be calculated between feature vector 904 and each feature vector 912. This can result in a matching level between the feature vector and the stored feature vectors. The matching level can be, for example, the minimum calculated value of metric 914. If the matching level is above a threshold, this can indicate that record 901 has a matching stored record or a duplicate record. The deduplication system built on this invention can then merge records because they represent the same entity. Merging records is an operation that can be implemented in different ways. For example, merging two records can include creating a gold record as a replacement for records that have been found to be duplicates of each other due to their similar appearance. This is known as data fusion or physical collapse with record or attribute level survival. If the matching level is less than or equal to the threshold, this indicates that record 901 does not match any record in the cluster and can therefore be stored.

[0112] Figure 10 This is a flowchart of method 1000 for matching records, based on an example from this topic. Method 1000 includes a machine learning phase 1001 and an application phase 1002.

[0113] A neural network system (1003) can be trained to generate feature vectors (Sim Vec) representing data records. The trained neural network system (1006) can then be used to generate (1004) feature vectors for all records in the MDM database. A clustering algorithm (1005) can be trained on the generated feature vectors. This results in cluster centroids (1007) for the clusters of the generated feature vectors. Steps 1003, 1004, 1005, and 1007 are part of the machine learning phase (1001).

[0114] During application phase 1002, and after feature vectors have been generated for all records in the MDM database, a request to add a new record can be received (1008). A trained neural network system 1006 can be used to generate feature vectors for the received record (1009). Cluster centroids 1007 can be used to determine (1010) the nearest centroid to the generated feature vectors of the received record. The generated feature vectors belonging to the cluster with the nearest centroid can be queried from the MDM database (1011). The queried feature vectors can be compared with the generated feature vectors of the received record (1012). This comparison can be performed by calculating the distance between the compared feature vectors. If (1013) the distance between two compared feature vectors is below a threshold, this indicates that they are duplicates and the corresponding records can be merged (1014). If (1013) the distance between any two compared feature vectors is not below a threshold, this indicates that they are not duplicates and the received record can be stored in the MDM database.

[0115] Figure 11 This is a diagram of a system 1100 for matching records, based on an example from this topic. An external system (e.g., a customer's database) 1101 may provide new record entries (e.g., individuals, organizations) to the MDM. These entries are sent to the MDM backend 1102, which sends a request to the ML service 1104 to generate a similarity vector 1111 from the new entry 1110. This newly created vector 1111 can then be used to find its corresponding cluster using pre-trained clusters 1108, which are encrypted and stored in a database 1103. Cluster centers 1107 stored in the database 1103 can be used to find the corresponding cluster. After obtaining the cluster, the backend 1102 can then query all existing vectors within that cluster and find possible matches with the received entry. However, homomorphic encryption, such as Paillier encryption, can be used to encrypt the feature vectors stored in the database 1103. Therefore, in order to find a match between the feature vector 1111 of the received entry 1110 and the encrypted feature vector 1112 of the cluster, the feature vector 1111 of the received entry 1110 is not encrypted and can be compared with the encrypted feature vector 1112 in its unencrypted form. For this purpose, the Euclidean distance can be reformulated to benefit from homomorphic encryption. The reformulated Euclidean distance can be as follows:

[0116] .

[0117] This distance is defined by three terms. and The sum of L1 and L3. When SimVec is first created, L1 and L3 can be easily computed and encrypted. L2 can be computed using Paillier homomorphic properties, requiring only that one of the two vectors is plaintext while the other remains encrypted.

[0118] A reformulated distance can be calculated between feature vector 1111 and each encrypted feature vector 1112 of the cluster. This results in an encrypted similarity 1113. The encrypted similarity can be decrypted to obtain a corresponding decrypted similarity 1114. Each decrypted similarity 1114 can be compared with a threshold to determine if a match exists between the new entry 1110 and any stored entries in the database 1103. If no match is found, the generated feature vector 1111 of the new entry can be encrypted first and then stored in the database 1103.

[0119] Figure 12 This refers to a general computerized system 1600 (e.g., a data integration system) suitable for implementing at least a portion of the method steps as described in this disclosure.

[0120] It should be understood that the methods described herein are at least partially non-interactive and are automated through computerized systems such as servers or embedded systems. However, in exemplary embodiments, the methods described herein can be implemented in (partially) interactive systems. These methods can also be implemented in software 1612, 1622 (including firmware 1622), hardware (processor) 1605, or a combination thereof. In exemplary embodiments, the methods described herein are implemented in software as an executable program and executed by a dedicated or general-purpose digital computer such as a personal computer, workstation, minicomputer, or mainframe computer. Thus, the most general system 1600 includes a general-purpose computer 1601.

[0121] In an exemplary embodiment, as shown in FIG6, the computer 1601 includes a processor 1605, a memory (main memory) 1610 coupled to a memory controller 1615, and one or more input and / or output (I / O) devices (or peripherals) 10, 1645 communicatively coupled via a local input / output controller 1635. The input / output controller 1635 may be, but is not limited to, one or more buses or other wired or wireless connections, as known in the art. The input / output controller 1635 may have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication; these elements are omitted for simplicity. Furthermore, the local interface may include address, control, and / or data connections to enable proper communication between the aforementioned components. As described herein, the I / O devices 10, 1645 may generally include any general-purpose encryption card or smart card known in the art.

[0122] Processor 1605 is a hardware device for executing software, specifically software stored in memory 1610. Processor 1605 can be any custom or commercially available processor, central processing unit (CPU), auxiliary processor among several processors associated with computer 1601, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, or any device generally used for executing software instructions.

[0123] The memory 1610 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that the memory 1610 may have a distributed architecture in which the various components are far apart from each other but can be accessed by the processor 1605.

[0124] The software in memory 1610 may include one or more individual programs, each program including an ordered list of executable instructions for implementing logical functions (particularly those involved in embodiments of the invention). In the example of FIG. 6, the software in memory 1610 includes instructions 1612, for example, instructions for managing a database such as a database management system.

[0125] The software in memory 1610 should also typically include a suitable operating system (OS) 411. OS 1611 essentially controls the execution of other computer programs, such as possible software 1612 used to implement the methods described herein.

[0126] The method described herein can be in the form of source program 1612, executable program 1612 (object code), script, or any other entity including instruction set 1612 to be executed. When it is a source program, it needs to be translated by a compiler, assembler, interpreter, etc., which may or may not be included in memory 1610, in order to operate correctly in conjunction with OS 1611. Furthermore, the method can be written in an object-oriented programming language with data classes and method classes, or a procedural programming language with routines, subroutines, and / or functions.

[0127] In an exemplary embodiment, a conventional keyboard 1650 and mouse 1655 may be coupled to an input / output controller 1635. Other output devices, such as I / O device 1645, may include input devices, such as, but not limited to, printers, scanners, microphones, etc. Finally, I / O devices 10, 1645 may also include devices that communicate with both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. I / O devices 10, 1645 may be any general-purpose encryption card or smart card known in the art. System 1600 may further include a display controller 1625 coupled to a display 1630. In an exemplary embodiment, system 1600 may also include a network interface for coupling to a network 1665. Network 1665 may be an IP-based network for communication between computer 1601 and any external servers, clients, etc., via a broadband connection. Network 1665 transmits and receives data between computer 1601 and external system 30, and may be involved in some or all of the steps for performing the methods discussed herein. In an exemplary embodiment, network 1665 may be a managed IP network managed by a service provider. Network 1665 may be implemented wirelessly, for example using wireless protocols and technologies such as WiFi, WiMax, etc. Network 1665 may also be a packet-switched network, such as a local area network, wide area network, metropolitan area network, the Internet, or other similar type of network environment. Network 1665 may be a fixed wireless network, wireless local area network (LAN), wireless wide area network (WAN), personal area network (PAN), virtual private network (VPN), intranet, or other suitable network system, and includes devices for receiving and transmitting signals.

[0128] If the computer 1601 is a PC, workstation, intelligent device, etc., the software in the memory 1610 may also include a Basic Input / Output System (BIOS) 1622. The BIOS is a set of basic software routines that initialize and test the hardware at startup, boot the OS 1611, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 1601 is activated.

[0129] When computer 1601 is running, processor 1605 is configured to execute software 1612 stored in memory 1610, transfer data to and from memory 1610, and generally control the operation of computer 1601 according to the software. The methods described herein and OS 1611, in whole or in part, but usually the latter, are read by processor 1605, possibly buffered within processor 1605, and then executed.

[0130] When the systems and methods described herein are implemented in software 1612, as shown in Figure 6, the methods can be stored on any computer-readable medium, such as storage device 1620, for use by or in conjunction with any computer-related system or method. Storage device 1620 may include disk storage devices such as HDD storage devices.

[0131] This topic includes the following terms.

[0132] Clause 1: A computer-implemented method comprising:

[0133] Provide a collection of one or more records, where each record in the collection has one or more attributes;

[0134] The set of values ​​of the attributes of the set of records is input into the trained data representation learning model, thereby receiving the set of feature vectors representing the set of records as the output of the trained data representation learning model;

[0135] A collection that stores feature vectors.

[0136] Clause 2: The method pursuant to Clause 1 also includes:

[0137] Receive additional records from a collection that has attributes;

[0138] The values ​​of the set of attributes of the other received records are input into the trained data representation learning model, thereby obtaining the feature vector of the other received records from the trained data representation learning model.

[0139] The obtained feature vectors are compared with at least a portion of the set of feature vectors to determine the matching level between the obtained feature vectors and the set of feature vectors.

[0140] The feature vectors obtained are stored based on the matching level, and / or the additional records received.

[0141] Clause 3: The storage of a set of feature vectors according to the method of Clause 1 or 2 includes clustering the set of feature vectors into clusters and associating each of the stored feature vectors with clustering information indicating the corresponding cluster.

[0142] Clause 4: The storage of the set of feature vectors according to the method of Clause 1 or 2 includes clustering the set of feature vectors into clusters of feature vectors, and the method further includes determining the distance between the obtained feature vectors and the vectors representing each cluster in the clusters, wherein at least a portion of the set of feature vectors includes clusters represented by vectors having the closest distance to the obtained feature vectors.

[0143] Clause 5: The method of any one of Clauses 2 to 4 above, wherein a record is received as part of a creation or update operation.

[0144] Clause 6: The method of any one of Clauses 1 to 5 above, wherein each feature vector in the set of output feature vectors comprises generating an individual feature vector for each attribute in the set of attributes and combining the individual feature vectors to obtain the feature vector.

[0145] Clause 7: The trained data representation learning model is configured to process input values ​​in parallel, according to any one of Clauses 1 to 6 above.

[0146] Clause 8: According to the method of any one of Clauses 1 to 7 above, the trained data representation learning model includes a set of attribute-level trained data representation models, wherein each of the attribute-level trained data representation models is associated with a corresponding attribute in a set of attributes, and wherein the output of each feature vector in the set of feature vectors includes:

[0147] The value of each attribute in the set of attributes is fed into the associated attribute level, which is then trained into a data representation model.

[0148] In response to the input, each receiving individual feature vector in the data representation model trained at the attribute level; and

[0149] The feature vector is obtained by combining individual feature vectors.

[0150] Clause 9: According to the method of Clause 8, the trained data representation learning model further includes a set of trained weights, each weight in the set of weights being associated with a corresponding attribute in the set of attributes, wherein the combination includes weighting each of the individual feature vectors using the corresponding trained weights in the set of trained weights.

[0151] Clause 10: According to the method of Clause 8 or 9, each of the attribute levels in the trained data representation model is a neural network.

[0152] Clause 11: The trained data representation learning model is generated by training to optimize a loss function, which is a measure of similarity between feature vectors of a pair of records, according to the method of any one of Clauses 8 to 10 above. The similarity measure is a combination of individual similarities, each of which indicates the similarity between two individual feature vectors generated for the same attribute in the pair of records.

[0153] Clause 12: The trained data representation learning model is trained to optimize a loss function, which is a measure of the similarity between feature vectors of a pair of records, according to any one of Clauses 1 to 11 above.

[0154] Clause 13: The trained data representation learning model according to any one of Clauses 1 to 12 above includes at least one neural network trained according to the Siamese neural network architecture.

[0155] Clause 14: According to the method of Clause 13, the trained data representation learning model comprises a trained neural network for each attribute in the set of attributes, wherein the output of each feature vector in the set of feature vectors includes:

[0156] The value of each attribute in the set of attributes is fed into the associated trained neural network;

[0157] In response to input, receive individual feature vectors from each of the trained neural networks; and

[0158] The feature vector is obtained by combining individual feature vectors.

[0159] Clause 15: The method pursuant to any one of Clauses 1 to 14 above further includes:

[0160] Receive a training dataset consisting of pairs of records with a set of attributes;

[0161] The training dataset is used to train the data representation learning model, thereby generating the trained data representation learning model.

[0162] Clause 16: According to the method of Clause 15, the data representation learning model comprises a set of attribute-level data representation models trained respectively with respect to a set of attributes, wherein training of the data representation learning model comprises, for each similarity record pair:

[0163] For each attribute in the set of attributes:

[0164] The attribute values ​​in the record pair are input into the data representation model trained at the corresponding attribute level to obtain individual feature vector pairs;

[0165] Calculate the individual similarity level between pairs of individual feature vectors;

[0166] The individual similarity level is weighted using trainable weights of attributes;

[0167] The measurement of similarity between feature vectors of record pairs is determined as a combination of individual weighted similarity levels;

[0168] Use measurements to evaluate the loss function used for training.

[0169] Clause 17: The method according to any one of Clauses 1 to 16 above, wherein the set of attributes comprises a first subset of attributes and a second subset of attributes, further comprising:

[0170] The system receives a first trained data representation learning model, which includes a first subset of attribute-level trained data representation learning models associated with a first subset of attributes, wherein the first trained data representation learning model is configured to receive values ​​of the first subset of recorded attributes and output feature vectors of the records by: inputting the value of each attribute of the first subset of attributes into the associated attribute-level trained data representation model of the first subset of attribute-level trained data representation learning models; receiving individual feature vectors from each attribute-level trained data representation model in the attribute-level trained data representation learning models of the first subset; and combining the individual feature vectors to obtain the feature vector.

[0171] Provide a second subset of attribute-level data representation learning models for a second subset of attributes, wherein each attribute-level data representation learning model of the second subset is configured to generate feature vectors for the corresponding attributes of the second subset of attributes;

[0172] Create a second subset of data representation learning models, which includes a first trained data representation learning model and an attribute-level data representation learning model;

[0173] The training data represents the learning model, thereby generating a trained data representation learning model.

[0174] Clause 18: According to the method of Clause 17, the first trained data representation learning model further includes a first subset of trained weights associated with a first subset of attributes, wherein combination is performed using the first subset of trained weights, and the resulting data representation learning model further includes a second subset of trainable weights associated with a second subset of attributes.

[0175] The trained data representation learning model is configured to output feature vectors from the set of feature vectors by:

[0176] The value of each attribute in the set of attributes is input into the associated attribute-level trained data representation learning model of the first and second subsets of the attribute-level trained data representation learning model;

[0177] In response to the input, the model learns the feature vector for each receiving individual from the data representation trained on the attribute levels of the first and second subsets; and

[0178] The individual feature vector is obtained by combining the corresponding weights of the first and second subsets of the weights.

[0179] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0180] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0181] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0182] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0183] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0184] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0185] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A computer-implemented method, comprising: Provide a set of records, each record in the set of records having a set of attributes, wherein the set of attributes includes a first subset of attributes and a second subset of attributes; The system receives a first trained data representation learning model, which includes a first subset of attribute-level trained data representation learning models associated with a first subset of attributes, wherein the first trained data representation learning model is configured to receive the values ​​of the first subset of attributes of a record and output the feature vector of the record by: The value of each attribute in the first subset of attributes is input into the associated attribute-level trained data representation model of the first subset of the attribute-level trained data representation model; Each attribute-level-trained data representation model in the first subset of the attribute-level-trained data representation learning model receives an individual feature vector; as well as Combine individual feature vectors to obtain each feature vector; Provide a second subset of attribute-level data representation learning models for a second subset of attributes, wherein each attribute-level data representation learning model for the second subset of attributes is configured to generate feature vectors for the corresponding attributes of the second subset of attributes; Create a second subset of data representation learning models, which includes a first trained data representation learning model and an attribute-level data representation learning model; The training data represents the learning model, thereby generating a trained data representation learning model; The set of values ​​of the attributes of the recorded set are input into the trained data representation learning model; The set of feature vectors representing the sets of records is received as the output of the trained data representation learning model. Based on a set of feature vectors, access the first record from the set of records.

2. The method according to claim 1, further comprising: Receive additional records from a collection that has attributes; The values ​​of the set of attributes of the other received records are input into the trained data representation learning model, thereby obtaining the feature vector of the other received records from the trained data representation learning model. The obtained feature vectors are compared with at least a portion of the set of feature vectors to determine the matching level between the obtained feature vectors and the set of feature vectors. The feature vectors obtained are stored based on the matching level, and / or the additional records received.

3. The method according to claim 1, further comprising: Cluster the set of feature vectors into clusters of feature vectors; Associate each feature vector in the set with clustering information indicating the cluster from which the feature vectors originate; and Clustering that stores feature vectors.

4. The method according to claim 2, further comprising: Cluster the set of feature vectors into a cluster of feature vectors; Clustering that stores feature vectors; as well as Determine the distance between the obtained feature vector and the vector of each cluster in the cluster representing the feature vector. At least a portion of the set of feature vectors includes clusters represented by vectors that have the closest distance to the obtained feature vectors.

5. The method according to claim 2 is executed in real time, wherein, Records are received as part of a create or update operation.

6. The method of claim 1, wherein the trained data representation learning model is configured to process input values ​​in parallel.

7. The method of claim 1, wherein the trained data representation learning model further comprises a set of trained weights, each weight in the set of weights being associated with a corresponding attribute in the set of attributes, wherein combining comprises weighting each of the individual feature vectors using the corresponding trained weights in the set of trained weights.

8. The method of claim 1, wherein each of the attribute levels in the trained data representation model is a neural network.

9. The method of claim 1, wherein the trained data representation learning model is trained to optimize a loss function, the loss function being a measure of similarity between feature vectors of a pair of records, the similarity measure being a combination of individual similarities, each of the individual similarities indicating the similarity between two individual feature vectors generated for the same attribute in the pair of records.

10. The method of claim 1, wherein the trained data representation learning model is trained to optimize a loss function, the loss function being a measure of the similarity between feature vectors of a record pair.

11. The method of claim 1, wherein the trained data representation learning model comprises at least one neural network trained according to the Siamese neural network architecture.

12. The method of claim 1, wherein the data representation learning model comprises a set of attribute-level data representation models associated with a set of attributes respectively, and training the data representation learning model comprises for each record pair: For each attribute in the set of attributes: The attribute values ​​in the record pair are input into the corresponding attribute level data representation model to obtain individual feature vector pairs; Calculate the individual similarity level between pairs of individual feature vectors; The individual similarity level is weighted using trainable weights of attributes; The measurement of similarity between feature vectors of record pairs is determined as a combination of individual weighted similarity levels; Use measurements to evaluate the loss function used for training.

13. The method of claim 1, wherein the first trained data representation learning model further includes a first subset of trained weights associated with a first subset of attributes, wherein combination is performed using the first subset of trained weights, and the created data representation learning model further includes a second subset of trainable weights associated with a second subset of attributes. in, The trained data representation learning model is configured to output feature vectors from a set of feature vectors by: The value of each attribute in the set of attributes is input into the first subset of the attribute-level trained data representation learning model and the second subset of the first subset of the attribute-level trained data representation learning model, which are associated with the attribute-level trained data representation learning model. In response to the input, the model learns the feature vector of each receiving individual from the attribute levels trained on the first and second subsets. as well as Each feature vector is obtained by combining the individual feature vectors with the corresponding weights of the first subset and the second subset of weights.

14. A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code being configured to implement the method of any one of claims 1-13.

15. A computer system, the system comprising: processor; as well as A memory that stores instructions that cause the processor to perform the method described in any one of claims 1-13.

Images