Methods, systems, and storage media for compliance entity merging and access

Abstract

The disclosed embodiments provide a system for managing data merges. During operation, the system generates matches between a first set of entities in a first data set from a first data provider and a second set of entities in a second data set from a second data provider based on a comparison of fields in the first and second data sets. Next, the system modifies a join query for joining the first and second data sets to include an operator representing a compliance rule for the first or second data set. The system executes the modified join query to produce a joined data set that adheres to the compliance rule, and stores data related to the joined data set within a platform that logically isolates the data from additional data sets. During processing of a query for the data, the system modifies the query to include an additional operator that enforces an access control policy on the data.

Description

Technical Field

[0001] The disclosed embodiments relate to techniques for merging data. More specifically, the disclosed embodiments relate to platforms for managing the merging and access of compliant entities. Background Technology

[0002] Organizations with a large number of users often store and / or manage large amounts of data for their users. For example, an online network with hundreds of millions of members can maintain petabytes (PB) of data related to member profiles and / or activities.

[0003] Sometimes, to comply with regulations and / or policies, it is necessary to update user data and / or other types of data. For example, search data, location data, personally identifiable information (PII), and / or other fields in a dataset need to be obfuscated and / or transformed to conform to the privacy and / or opt-out preferences of the corresponding users. When datasets are shared and / or combined across organizations, the data in the datasets may be subject to multiple sets of privacy and / or compliance rules, which complicates the protection, access, or use of the data by organizational representatives. Summary of the Invention

[0004] According to one aspect of this application, a method for managing compliance entity merging and access includes: generating a match between entities in the first set of the first dataset and entities in the second set of the second dataset based on a comparison of fields in a first set of data from a first data provider and fields in a second set of data from a second data provider; modifying a merging query for combining the first dataset and the second dataset to include one or more operators, the one or more operators representing one or more compliance rules for the first dataset or the second dataset, wherein the merging query includes a merging predicate represented by the generated match; executing the modified merging query to generate a merging dataset from the first dataset and the second dataset that complies with the one or more compliance rules; and storing data related to the merging dataset within a platform, the platform being logically... The combined dataset is isolated from one or more additional datasets not from the first data provider and the second data provider, wherein the storage includes storing a derived dataset generated from one or more fields in the combined dataset within the platform, and includes storing the derived dataset in one or more data stores within the platform based on a pattern for the derived dataset by: identifying searchable fields and unique fields in the derived dataset based on the pattern; storing values ​​of a first set of searchable fields from the derived dataset in a search index; and storing values ​​of a second set of unique fields in the derived dataset in a key-value store; and modifying the query during processing of a query on the stored data to include one or more additional operators that enforce one or more access control policies for the data. Generating the match between entities in the first set and entities in the second set includes: applying a machine learning model to the fields shared by entities in the first set and entities in the second set to generate a match score between pairs of entities selected from the entities in the first set and entities in the second set; and generating a match between entities in the first set and entities in the second set based on the match score. Modifying the combined query to include the one or more operators representing the compliance rules includes: matching fields in the combined query with compliance rules representing privacy settings for the fields; and inserting operators into the relational operator tree representing the combined query, the operators including filters selectively applied to the fields based on the values ​​of the privacy settings. Modifying the query to include the one or more additional operators enforcing the one or more access control policies for the data includes: processing queries from the entity regarding the data based on the amount of data accessed by the query and data access restrictions for the entity.Modifying the query to include the one or more additional operators that enforce the one or more access control policies for the data includes: verifying that the access control policy authorizes the entity to access the data before executing a query by the entity on the data. The platform logically isolates the combined dataset from the one or more additional datasets not from the first data provider and the second data provider by modifying the execution of the query to exclude the additional datasets from the query results. The platform logically isolates the combined dataset from the one or more additional datasets not from the first data provider and the second data provider by: encrypting data in the combined dataset using a first key for the first data provider or the second data provider; encrypting data in the one or more additional datasets using one or more keys for additional data providers excluding the second data provider; and decrypting the data in the combined dataset using the first key during the processing of the query on the data. Entities in the first set and entities in the second set include at least one of the following: users; employees; companies; schools; jobs; or skills. The fields of the first set and the fields of the second set include at least one of the following: name; place; title; location; qualifications; or company.

[0005] In another aspect of this application, a system for managing the merging and access of compliant entities includes: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the above methods to be performed.

[0006] In another aspect of this application, there is a storage medium for one or more storage instructions that, when executed by one or more processors, cause the above method to be performed. Attached Figure Description

[0007] Figure 1 A schematic diagram of a system according to a disclosed embodiment is shown.

[0008] Figure 2 A system for managing data merging according to a disclosed embodiment is shown.

[0009] Figure 3A An exemplary relational operator tree for join queries is shown according to the disclosed embodiments.

[0010] Figure 3B An exemplary relational operator tree for join queries is shown according to the disclosed embodiments.

[0011] Figure 4A flowchart illustrating the process of managing the merging and access of data according to the disclosed embodiments is shown.

[0012] Figure 5 A computer system according to the disclosed embodiments is shown.

[0013] In the accompanying drawings, similar reference numerals refer to the same elements. Detailed Implementation

[0014] The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this disclosure. Therefore, the invention is not limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0015] Overview

[0016] The disclosed embodiments provide methods, apparatus, and systems for managing the merging and access to data associated with multiple data providers. For example, the data may include datasets from two or more organizations or data providers. The data may be joined, merged, or otherwise combined to produce one or more new datasets, which are stored in a distributed environment that allows the datasets to be queried, analyzed, and / or served by users authorized by the data providers.

[0017] More specifically, the disclosed embodiments provide a platform for enforcing compliance, privacy, security, and other data access policies related to multiple data providers during the fusion of datasets from data providers and / or subsequent access to the fused data. These data access policies include, but are not limited to: compliance policies for each organization and / or datasets within an organization; privacy policies or rules for individual entities represented within said datasets (e.g., users, accounts, etc.); and / or different access levels to the datasets and / or fused data.

[0018] To merge input datasets from two or more data providers, the platform matches entities in the datasets based on similarity and / or overlap in fields associated with entities from the datasets. For example, the platform matches records of registered users from an online network in a first dataset with records of company employees from a second dataset based on bindings between user accounts and online networks and corresponding employee accounts and companies, email addresses shared by records across the two datasets, and / or other attributes shared by records across the two datasets (e.g., first name, last name, company, job title, location, etc.).

[0019] Next, the platform executes a join query that connects the input datasets by matching entities in a manner consistent with compliance rules for some or all datasets and / or data providers. For example, the platform adds an operator representing a compliance rule for one or more entities in the datasets to the join query. This operator may be used to remove, obfuscate, encrypt, and / or otherwise transform entity-specific fields to ensure the joined datasets comply with applicable laws, regulations, and / or policies. The platform then uses the join query to generate a joined dataset from two or more input datasets, where each record in the joined dataset represents a match across entities in the input datasets. Each record in the joined dataset also includes one or more fields from each of the input datasets.

[0020] The platform then stores and / or processes queries on connected data and / or additional datasets derived from connected data in a manner that enforces access control policies for the data and / or isolates data from other data providers or entities. For example, the access control policies specify different access levels to connected and / or derived data for various roles (e.g., users, organizations, etc.) and / or rules. Access control policies can be created based on input from administrators of data providers. Such input can be used to update access control policies and dynamically grant and / or revoke access to connected and / or derived data based on roles and / or rules.

[0021] To enforce the access control policy for a given dataset, the platform verifies that the entity requesting access to the dataset (e.g., a user with a platform account or identity) is granted access to the data as stipulated in the access control policy. If compliance rules restrict an entity's access to the dataset exist, the platform modifies the entity's query for the dataset to include additional operators that perform filtering or other transformations on the data based on the compliance rules. The platform also prevents queries from combining the dataset with datasets from other data providers. Such exclusions can be performed by, for example, encrypting and decrypting each dataset using different keys, modifying the query to include "group by" or other filtering clauses in the query results that isolate the datasets from each other, and / or storing datasets in different instances of the platform and / or on different sets of physical resources.

[0022] The disclosed embodiments allow for the separate creation and enforcement of policies by merging data from multiple data providers in a manner that complies with the compliance, privacy, and / or other data access policies of the data providers. Therefore, administrators of the data providers can grant, revoke, and / or modify access to the data as needed, and data from the data providers is merged in a manner that enforces policies on a per-record, per-field, and / or per-query basis. The platform can further manage the creation and access to multiple merged datasets by applying different policies to different combinations of datasets and / or enforcing logical or physical isolation between datasets (e.g., by encrypting and decrypting datasets using different keys and / or modifying queries on datasets).

[0023] Subsequently, the platform reduces the overhead associated with storing, processing, and / or querying sensitive data and / or data related to multiple data access policies. For example, the platform automatically creates storage accounts, security identities, roles, and / or other components for storing and accessing merged datasets based on the compliance and / or access control policies of the corresponding data providers. The platform also modifies queries used to create and / or access the merged data in a manner that enforces the policies. Alternatively, conventional techniques require manual configuration and / or review of roles, accounts, queries, platforms, and / or other components involved in isolating or protecting data. These techniques are prone to errors and / or omissions during configuration and / or review, which reduces data security and / or isolation. Furthermore, the steps for correcting errors and / or omissions involve additional processing due to the creation, deletion, and / or reconfiguration of components, which increases resource consumption and / or overhead on the computer systems on which the components are executed. Because the disclosed embodiments reduce security issues, privacy issues, errors, overhead, and / or processing compared to conventional data processing or merging techniques, the disclosed embodiments are applicable to computer systems, applications, user experiences, tools, platforms, and / or techniques related to protecting, querying, modifying, merging, and / or accessing data in a distributed environment.

[0024] Compliance Entity Merging and Access

[0025] Figure 1 A schematic diagram of a system according to a disclosed embodiment is shown. (As in...) Figure 1 As shown, the system includes an online network 118 and / or other user communities. For example, the online network 118 includes an online professional network of entities (e.g., entity 1 104, entity x 106) that interact with each other in a professional and / or business context.

[0026] The entities include users who use online network 118 to establish and maintain professional connections, list work and community experiences, endorse and / or recommend each other, search for and apply for jobs, and / or perform other actions. The entities also include, or alternatively include, companies, employers, and / or recruiters who use online network 118 to list jobs, search for potential candidates, provide users with business-related updates, advertise, and / or take other actions.

[0027] Online network 118 includes a profile component 126, which allows entities to create and edit profiles containing information related to the entity's professional and / or industry background, experience, summary, job title, projects, skills, etc. Profile component 126 also allows entities to view profiles of other entities in online network 118.

[0028] Profile component 126 also includes, or alternatively includes, mechanisms to assist the entity in completing its profile. For example, profile component 126 may suggest industry, skills, company, school, publications, patents, certificates, and / or other types of attributes as potential additions to the entity profile. The suggestions may be based on predictions of missing fields, such as predicting the entity's industry based on other information in the entity profile. The suggestions may also be used to correct existing fields, such as correcting the spelling of company names in the profile. The suggestions may also be used to clarify existing attributes, such as changing the entity's title of "Manager" to "Engineering Manager" based on the entity's work experience.

[0029] Online network 118 also includes a search component 128, which allows the entity to search for people, companies, jobs, and / or other work or business-related information within online network 118. For example, the entity can enter one or more keywords into the search bar to find profiles, job postings, job candidates, articles, and / or other information that includes and / or otherwise matches one or more keywords. The entity can additionally use the "Advanced Search" function in online network 118 to search profiles, jobs, and / or information by category, such as first name, last name, title, company, school, location, interests, relationships, skills, industry, group, salary, experience level, etc.

[0030] Online network 118 also includes an interaction component 130 that allows the entities to interact with each other on online network 118. For example, interaction component 130 may allow a user, company, school, or other entity to add other entities as connections, follow other entities, send and receive emails or messages with other entities, join groups, and / or interact with posts from other entities (e.g., create, share, reshare, like, and / or comment).

[0031] Those skilled in the art will recognize that online network 118 may include other components and / or features. For example, online network 118 may include a homepage, a landing page, and / or content feeds that provide the entities with the latest posts, articles, and / or updates from the entities' links and / or groups. Similarly, online network 118 may include features or mechanisms for recommending links, job postings, articles, and / or groups to the entities.

[0032] In one or more embodiments, data relating to entity profiles and activities on online network 118 (e.g., data 1 122, data x 124) is aggregated into data repository 134 for subsequent retrieval and use. For example, each profile update, profile view, connection, follow, post, comment, like, share, search, click, message, interaction with a group, address book interaction, response to a recommendation, purchase, and / or other action performed by entities in online network 118 is recorded and stored in a database, data warehouse, cloud storage, and / or other data storage mechanism providing data repository 134.

[0033] More specifically, data related to online network 118 and / or stored in data repository 134 includes (but is not limited to) user data, company data, and / or work data. The user data includes attributes from user profiles of online network 118, such as the user's title, profile, skills, work experience, education, qualifications, industry, location, and / or profile completeness. The user data also includes, or alternatively includes, connections or contacts for each user in online network 118, the user's tenure on online network 118 (e.g., days, weeks, months, or years since the user joined online network 118), the set of groups the user belongs to, awards or honors received by the user, licenses or certificates obtained by the user, and / or patents or publications associated with the user. The user data also includes, or alternatively includes attributes specific to one or more components of online network 118, such as classifying users as job seekers or non-job seekers.

[0034] In some embodiments, the user data characterizes a user's activity using the online network 118. For example, the user data may include an activity level for each user, which may be binary (e.g., dormant or active) or calculated by aggregating different types of activity (e.g., user sessions over a month, page views per session, searches, clicks, etc.) into an overall activity count and / or bucketed activity scores. The user data also records, or alternatively records, impressions, clicks, likes, dislikes, shares, hides, comments, posts, updates, conversions, and / or other user interactions with platform content, as well as other types of activity including connections, messaging, job applications, job seeking, recruiters searching for candidates, interactions between candidates and recruiters, and / or interactions with groups or events.

[0035] In some embodiments, the user data also includes social validation of skills, qualifications, job titles, and / or other profile attributes, such as endorsements, recommendations, ratings, comments, collaborations, discussions, articles, posts, comments, shares, and / or other member interactions related to the profile attributes. The user data also includes, or alternatively includes, the user's schedule, calendar, and / or upcoming availability, which can be used to schedule meetings, interviews, and / or events for the user.

[0036] The company data includes attributes and / or measures associated with the company. For example, company data may include demographic attributes such as the company's location, industry, age, and / or size (e.g., small business, medium / large, global / large, number of employees, etc.). The company data may also include measures of dispersion within the company, such as the multiple unique regions (e.g., metropolitan area, county, city, state, country, etc.) to which employees and / or members from the company's online network 118 belong.

[0037] A portion of the company data may relate to the behavior of the many products provided by or through online network 118. For example, the company data may include recruitment-based data, such as the number of recruiters, the company's potential spending on recruitment solutions within online network 118, the number of recruits in a recent period (e.g., the past 12 months), and / or the same number of recruits divided by the total number of employees and / or members of online network 118 within the company.

[0038] The company data may also indicate the company's level of participation in and / or presence on online network 118. For example, the company data may include the number of employees who are members of online network 118, the number of employees who are members of online network 118 at a specific seniority level (e.g., entry-level, mid-level, managerial, senior, etc.), and / or the number of employees who are members of online network 118 with a specific role (e.g., engineer, manager, sales, marketing, recruiting, executive, etc.). The company data may also include the number of online network 118 members who have connections with employees of online network 118, the number of connections between employees within the company, and / or the number of the company's followers on online network 118. The company data may also record company employees' access to online network 118, such as the number of company employees who accessed online network 118 in a recent time period (e.g., the past 30 days) and / or the same number of visitors divided by the total number of company members in online network 118.

[0039] At least some of the company data may be derived from user data. For example, the company data may include measures of aggregated user activity for specific activity types (e.g., profile viewing, page viewing, jobs, searches, purchases, endorsements, messaging, content viewing, invitations, connections, recommendations, advertising, etc.), user segments (e.g., groups of users sharing one or more common attributes, such as members in the same location and / or industry), and the company itself. This company data can then be used to gather company-level insights or trends from user-level data, perform statistical inference at the company and / or user segment levels, and / or guide decisions related to activities the company can perform within the online network 118.

[0040] The job data includes attributes of jobs posted or described within the online network 118. For example, the job data may include declared or inferred attributes of the job, such as job title, company, required or expected skills, responsibilities, qualifications, role, location, industry, experience, salary range, benefits, and / or member category. At least some job data may also be derived from user data and / or company data. For example, the job data can provide context for each user's impression of a job listing or job description. This context may include the time and location (e.g., geographic location, application, website, webpage, etc.) when a member viewed the job listing or description. In another example, the job data includes counts of views, clicks, applications, and / or other user actions associated with a given job.

[0041] Those skilled in the art will recognize that data repository 134 may store other types of data in relation to online network 118. For example, data repository 134 may include articles, posts, images, videos, slideshows, documents, advertisements, and / or other content created and / or shared within online network 118. Data repository 134 may also include, or alternatively include, content-related metadata such as the author or creator of the content, the time the content was created or shared, and the users who interacted with the content (e.g., view counts, likes, shares, or other types of interaction with the content) and / or the users who interacted with the content.

[0042] In one or more embodiments, the data in data repository 134 is standardized before the data is used by components of the system. For example, skills in user data and / or job data are organized into hierarchical taxonomies stored in data repository 134 and / or another repository. The taxonomies model relationships between skills (e.g., "Java programming" is related to "software engineering" or a subset thereof) and / or standardize similar or highly related skills (e.g., "Java programming," "Java development," "Android development," and "Java programming language" are standardized to "Java").

[0043] In another example, locations in data repository 134 include cities, metropolitan areas, states, countries, continents, and / or other standardized geographic regions. Similar to standardized skills, locations can be organized into hierarchical taxonomy (e.g., cities are organized below states, states below countries, countries below continents, and so on).

[0044] In the third example, data repository 134 includes standardized company names for a set of known and / or verified companies associated with members and / or jobs. In the fourth example, data repository 134 includes standardized titles, qualifications, and / or industries for various jobs, members, and / or companies in an online network. In the fifth example, data repository 134 includes standardized time periods (e.g., daily, weekly, monthly, quarterly, yearly, etc.) that can be used to retrieve profile data 216, user activity data 218, and / or other data represented by time periods (e.g., starting work in a given month or year, graduating from university within five years, job listings posted within two weeks, etc.). In the sixth example, data repository 134 includes standardized job functions such as “accounting,” “consulting,” “education,” “engineering,” “finance,” “healthcare services,” “information technology,” “legal,” “operations,” “real estate,” “research,” and / or “sales.”

[0045] In some embodiments, standardized attributes in data repository 134 are represented by unique identifiers (IDs) in the corresponding taxonomy. For example, each standardized skill is represented by a digital skill ID in data repository 134, each standardized title is represented by a digital title ID in data repository 134, each standardized location is represented by a digital location ID in data repository 134, and / or each standardized company name (e.g., for companies exceeding a certain size and / or exposure level in an online system) is represented by a digital company ID in data repository 134.

[0046] In some embodiments, the data in data repository 134 is combined into a graph. Nodes in the graph represent entities within the platform (e.g., users, schools, companies, locations, jobs, content, etc.). Edges between nodes in the graph represent relationships between corresponding entities, such as connections between user pairs, a user's education at a school, a user's employment at a company, another user's following of a user or company, business relationships and / or partnerships between organizations, and / or a user's residence. The edges also indicate, or alternatively, actions taken by entities, such as creating or sharing articles or posts, sending messages, sending or receiving connection requests, mutual endorsements or recommendations, writing comments, applying for opportunities, joining groups, and / or following other entities.

[0047] Data in data repository 134 can then be used to generate recommendations, search results, and / or other insights related to users, companies, schools, groups, content, jobs, and / or other entities within online network 118. For example, one or more components of online network 118 can track searches, clicks, views, text input, conversions, and / or other feedback during interactions between entities and profile component 126, search component 128, interaction component 130, and / or other components of online network 118. This feedback can be stored in data repository 134 and used as training data for one or more machine learning models, and the output of the machine learning models(s) can be used to display and / or otherwise recommend jobs, advertisements, posts, articles, links, products, companies, groups, and / or other types of content, entities, or actions to members of online network 118.

[0048] Data in data repository 134 can be further merged with datasets from other data providers to generate additional insights and / or support additional functionality related to online network 118 and / or other data providers. In some embodiments, merging data includes joining, merging, or otherwise combining fields or records from two or more datasets. For example, data from data repository 134 relating to career path transitions of users of online network 118 (e.g., a sequence of titles, qualifications, industries, etc. in a user's career history) can be combined with salary or benefits information from employers to provide a more comprehensive set of data or trends related to user career trajectories. In another example, skills listed in user profiles of online network 118 and / or recognized by other users are combined with the company's employee records. The combined data can then be queried and / or analyzed to identify employees with specific skills within the company, shortages or surpluses of various skills within the company, top skills possessed by individual teams within the company, and / or other types of trends or patterns related to the skills of the company's employees.

[0049] Those skilled in the art will recognize that datasets from data repository 134, online network 118, and / or other data providers may be subject to various compliance, privacy, and / or data access policies applied at different granularities and / or different sets of records or fields within the datasets. For example, data in data repository 134 and / or data from external data providers may be subject to local, national, or regional privacy rules. Similarly, different types of data may be associated with different levels of protection or security (e.g., financial data or health records may have the highest level of protection, business records or de-identified health information may have a medium level of protection, and publicly available data may have a low level of protection). Each data provider may also have separate privacy policies relating to the collection, use, or sharing of data. Individual users may additionally specify or customize privacy settings that restrict the collection and / or sharing of user data and / or individual fields within user data.

[0050] Therefore, when creating a merged dataset by combining data from online network 118 and / or other data providers, the merged dataset may be required to comply with data access rules and / or regulations from all data providers and / or for all types of data in the merged dataset. As a result, significant complexity and / or overhead may arise in ensuring that the merged dataset is created, stored, and accessed in accordance with these rules and regulations.

[0051] In one or more embodiments, a platform physically and / or logically isolated from one or more of the data providers is used to create, store, and manage access to merged datasets containing data from the data providers. (As in...) Figure 2 As shown, such a platform 202 includes a merging device 208, a metadata store 224, multiple data stores 234, and an online service 222. Each of these components is described in more detail below.

[0052] The merging device 208 combines two or more input datasets (e.g., dataset 1 204, dataset y 206) into a combined dataset 214. In some embodiments, each input dataset is provided by a different data provider, and the combined dataset 214 includes some or all fields from each input dataset.

[0053] The merging device 208 further includes the function of generating a derived dataset 216 based on the data in the combined dataset 214. More specifically, one or more fields in the derived dataset 216 are generated as transformations of one or more fields in the combined dataset 214.

[0054] For example, merging device 208 will integrate online systems (e.g., Figure 1Skills listed in user profiles of the online network 118 are combined with employee records from the company to produce an exported dataset 216. Each record in the exported dataset 216 includes an employee identifier (ID) for the user with employee records from the company and profiles in the online system. The employee IDs are mapped to skill lists for user teams from the online system and user profiles within the company. The merging device 208 then aggregates the skill counts for teams in the combined dataset 214 to create the exported dataset 216. Each record in the exported dataset 216 includes the name of the team in the company and a list of the five most frequently occurring skills among employees in that team. Therefore, combining dataset 214 and / or exporting dataset 216 allows for the combination and / or transformation of different input datasets from different data providers in ways that improve understanding of the data in the datasets, reveal insights associated with the data, and / or increase the availability or applicability of the data to various types of applications, platforms, and / or distributed systems.

[0055] In some embodiments, the merging device 208 generates a combined dataset 214 and / or an exported dataset 216 based on metadata in metadata storage 224. Metadata storage 224 includes databases, distributed file systems, cloud storage systems, and / or other types of repositories used to create, store, and / or manage access to combined dataset 214 and / or exported dataset 216. This metadata includes compliance policies 236 for some or all data providers of the input datasets. For example, compliance policies 236 include federal, state, local, and / or other government regulations relating to privacy and / or data access.

[0056] The metadata in metadata store 224 also includes, or alternatively includes, privacy policies 238 representing data providers and / or individual users (or other entities) within the input dataset. For example, privacy policy 238 includes corporate standards or rules relating to the collection, storage, use, and / or sharing of data by a given data provider. Privacy policy 238 also includes, or alternatively includes, privacy settings for individual users and / or other entities (e.g., companies, schools, government agencies, etc.) sharing data with the data provider.

[0057] In one or more embodiments, a configurable set of rules is used to specify compliance policy 236 and / or privacy policy 238 in metadata store 224. For example, compliance policy 236 and privacy policy 238 are stored in one or more configuration files created by the data provider of the input datasets and / or entities sharing data with the data provider. Each configuration file includes a set of rules defined using a domain-specific language (DSL) that can be parsed by platform 202, wherein each rule specifies a condition (e.g., a value or range of values ​​for a field in the input datasets) relative to which it evaluates data in one or more input datasets. When the condition evaluates to a given value (e.g., true) for one or more rows, columns, and / or other portions of the input datasets, a corresponding action is performed (e.g., filtering said portions from combined dataset 214 and / or derived dataset 216).

[0058] Metadata storage 224 further includes data processing logic 240 and a schema 242 for generating combined dataset 214 and / or derived dataset 216. In some embodiments, schema 242 describes the structure of data in the input dataset, combined dataset 214, and / or derived dataset 216. For example, schema 242 includes names and / or data types associated with rows and columns in each dataset.

[0059] Data processing logic 240 includes a combine query 212 executed by merging device 208 to generate a combined dataset 214 from the input datasets. For example, the combine query 212 can be specified using Structured Query Language (SQL), GraphQL, SPARQL (SPARQL protocol and RDF query language), and / or another type of query language. The combine query 212 can include inner joins, left outer joins, and / or another type of join operation that combines one or more fields from a first input dataset with one or more fields from a second input dataset. The combine query 212 optionally specifies a combine predicate (e.g., after the "on" or "where" keywords in an SQL query) that specifies how fields from the input datasets are combined into records in the combined dataset 214.

[0060] Those skilled in the art will recognize that entities such as users, companies, schools, jobs, and / or skills can be represented using distinct unique IDs and / or keys in each input dataset. For example, a user can be represented using a unique user ID from a first dataset of an online system and distinct unique employee IDs from a second dataset of companies that employ the user. Therefore, it is not possible to combine user records of users in the first dataset with employee records of users in the second dataset simply by matching the user IDs of users in the first dataset with the employee IDs of users in the second dataset.

[0061] In one or more embodiments, merging device 208 creates a combined dataset 214 by generating matches between (or among) entities in the input dataset based on comparisons of potentially non-unique fields associated with entities in the input dataset. These comparisons may be associated with different confidence levels of the corresponding matches. Continuing the example above, merging device 208 generates a first set of “highest confidence” matches between user records in a first dataset and employee records in a second dataset based on a binding table between user accounts in an online system and employee accounts in a portal for managing company employees. Each record in the binding table is generated after the user, represented by the account, binds his / her online system account to the corresponding account in the portal. To bind the two accounts, the user can provide authentication credentials for his / her online system (or portal) account when logging into the portal (or online system). In response to the provided authentication credentials, the online system (or portal) can perform an “integration” that generates an association between the two accounts and allows access to the functionality of the other account via one account. The record of this association between the accounts is then stored as a binding between accounts in the table.

[0062] Continuing the example above, the merging device 208 may also, or alternatively, generate a second set of "high-confidence" matches between the user records and employee records based on the work email addresses shared by the user records and employee records. When the first name, last name, title, location, position, and / or company are the same in the user records and employee records, the merging device 208 may also, or alternatively, generate a third set of "medium-confidence" matches between the user records and employee records.

[0063] Continuing the example above, the merging device 208 may also, or alternatively, apply a machine learning model to features including fields in user and employee records (e.g., first and last name, title, job title, location, company, etc.) and / or to comparisons between fields in user records and corresponding fields in employee records (e.g., Boolean values ​​indicating whether a field in a user record exactly matches a corresponding field in an employee record, edit distance between two fields, etc.). The machine learning model may include (but is not limited to): regression models, neural networks, tree-based models, and / or deep learning models. In response to input features, the machine learning model outputs a matching score for a given pair of records (comprising user records from a first dataset and employee records from a second dataset), the matching score representing the confidence that the user record and employee record represent the same user. When the matching score exceeds a threshold, the merging device 208 establishes a match between the user record and the employee record.

[0064] In some embodiments, merging device 208 combines records in an input dataset based on configurable requirements or thresholds for the confidence level of matches between entities represented by the records. For example, merging device 208 may obtain a minimum confidence level for matches between entities in the input dataset from data processing logic 240 and / or other metadata in metadata storage 224. This minimum confidence level may be determined by an administrator of one or more data providers of the input dataset based on the data types to be combined in the input dataset and / or the subsequent use of the data. A higher minimum confidence level may be specified for combining sensitive and / or personal data, such as human resources records of individual employees in a company. Alternatively, a lower minimum confidence level may be specified for combining data involving company-level, industry-level, or other data that does not include personally identifiable information (PII). After identifying matching entity pairs or groups that meet the minimum confidence level, merging device 208 executes combination query 212 to combine records representing matching entities. Therefore, in the absence of an explicit combination predicate specified for combining one or more comparisons of records in the input dataset, the matches between entities in the input dataset identified by merging device 208 can be used as implicit combination predicates for combination query 212.

[0065] To ensure that the combined dataset 214 complies with the compliance policy 236 and privacy policy 238 of the corresponding input datasets, the merging device 208 modifies the merging query 212 by 210 to apply the rules in the compliance policy 236 and privacy policy 238 during the creation of the combined dataset 214. For example, the merging device 208 parses the merging query 212 into a relational operator tree, which includes nodes representing projection and selection operators to be applied to each input dataset. Projection operators are used to select columns or fields of the input datasets to which the merging query 212 will be applied, and selection operators are used to select rows or records of the input datasets to be returned by the merging query 212. The merging device 208 also matches the records and / or fields selected by the merging query 212 with one or more rules in the compliance policy 236 and / or privacy policy 238, and adds additional operators representing the rules(s) to the relational operator tree. Each additional operator can be used to filter, obfuscate, or otherwise apply the corresponding rule to the records and / or fields in the corresponding input dataset.

[0066] Subsequently, the merging device 208 executes the modified join query 212 to create a combined dataset 214. Continuing the example above, the merging device 208 converts the modified relational operator tree into a version of join query 212 for execution by an environment that generates the combined dataset 214 from the input dataset. This environment includes, but is not limited to, an offline processing environment that generates the combined dataset 214 from the input dataset using one or more batch operations. The offline processing environment then executes the compiled join query 212 to generate the combined dataset 214 from the input dataset.

[0067] After creating the combined dataset 214, the merging device 208 applies one or more transformation queries 246 specified in the data processing logic 240 to the combined dataset 214 to produce a derived dataset 216. For example, the transformation query 246 is used to perform additional filtering, combining, aggregation, and / or other processing on the rows and / or columns of the combined dataset 214. The derived dataset 216 is then output as the result of the transformation query 246.

[0068] After creating the combined dataset 214 and the exported dataset 216, the merging device 208 and / or another component of the platform 202 store one or both datasets in the key-value store 218, the search index 220, and / or other data stores 234 provided by the platform 202. For example, the component stores the exported dataset 216 in the data store 234 for subsequent querying or use, as described in further detail below. If the combined dataset 214 contains information useful to the data provider and / or allows the creation of additional exported datasets (not shown) from the combined dataset 214, the component may optionally store the combined dataset 214 in one or more data stores 234.

[0069] In one or more embodiments, the component stores the combined dataset 214 and / or the derived dataset 216 in the data storage 234 according to a corresponding pattern 242. For example, the component identifies one or more keys in the pattern of the combined dataset 214 and / or the derived dataset 216 and maps the keys(s) to corresponding values ​​in the key-value store 218. In another example, the component identifies one or more searchable fields in the pattern for use in the combined dataset 214 and / or the derived dataset 216 and stores the values ​​of the searchable fields in the search index 220. The creation and storage of combined datasets and / or derived datasets using modified combined queries will be referenced below. Figures 3A-3B To describe in more detail.

[0070] Then, online service 222 uses data storage 234 to process requests 232 for data 230 from combined dataset 214 and / or exported dataset 216. For example, online service 222 receives a request including one or more keywords that match records in combined dataset 214 and / or exported dataset 216. To process the request, online service 222 translates the request into a query for search index 220, which includes one or more keywords as search parameters, and executes the query against search index 220. Online service 222 receives a set of records that combine keywords from combined dataset 214 and / or exported dataset 216, mapped to search index 220, and returns the records in response to the request as the result of the query. In another example, online service 222 receives a request for records that match a range of employee IDs in combined dataset 214 and / or exported dataset 216. Online service 222 translates the request into a query that specifies a range of employee IDs as key-value pairs in key-value store 218. Online service 222 performs a query relative to key-value store 218 to retrieve a set of records mapped to employee IDs from key-value store 218, and returns the records in response to the request.

[0071] In one or more embodiments, online service 222 receives, processes, and / or responds to request 232 according to an application programming interface (API) of the entity submitting request 232 (e.g., entity 1250, entity z 252). For example, the API exposes requests 232 that can be invoked by the entity to search, retrieve, and / or otherwise generate data for combining dataset 214, exporting dataset 216, and / or other datasets in data store 234. The specification for the API is optionally stored in metadata store 224 to facilitate debugging and / or auditing of online service 222. When a request is received from an entity, online service 222 translates the parameters of the request into a query. Online service 222 then executes the query on one or more data stores 234 containing the data. After the data stores(s) return data 230 in response to the query, online service 222 formats the data 230 into a response to the request (e.g., according to the specification for the API) and transmits the response to the entity. As a result, the API implemented by online service 222 allows entities to access combined dataset 214 and / or derive dataset 216 without knowing where or how the datasets are stored.

[0072] Online service 222 further includes the ability to process request 232 based on compliance policies 236, privacy policies 238, and / or access control policies 244 associated with the combined dataset 214 and / or exported dataset 216 in metadata storage 224. In some embodiments, access control policies 244 are created based on input from administrators and / or other representatives of the data provider of the input dataset. For example, platform 202 includes administrator accounts for administrators of data providers. Access control policies 244 include rules and / or roles that grant different levels of access to the combined and / or exported data to various entities (e.g., users, organizations, etc.). After some or all administrator accounts have approved the rules and / or roles, a given rule and / or role for accessing the combined dataset 214 and / or exported dataset 216 can be added to access control policy 244. At least some of these rules and / or roles can be dynamically added to access control policy 244 to enable additional entities or users to access the combined dataset 214 and / or exported dataset 216 "in a timely manner" (e.g., for debugging or auditing purposes). Alternatively, any administrator among the administrators can revoke access to the combined dataset 214 and / or the exported dataset 216 associated with a given rule and / or role (e.g., by removing the rule and / or role from access control policy 244). In other words, platform 202 supports “shared management” of access by the data provider’s administrators to the combined dataset 214, the exported dataset 216, and / or other data in the data storage 234.

[0073] Upon receiving a request (e.g., request 232) for data 230 from combined dataset 214 and / or derived dataset 216, online service 222 applies rules and / or roles from access control policy 244 to verify that the entity issuing the request is authorized to access the requested data 230. For example, online service 222 matches the identifier of the entity's account (e.g., after the entity provides authentication credentials for the account to online service 222) to an entry in the access control list of access control policy 244. If the entry indicates that the entity is authorized to access the requested data 230, online service 222 continues processing the request. In another example, online service 222 matches attributes of the request (e.g., Internet Protocol (IP) address, portal, request time, request parameters, etc.) to rules in access control policy 244 to verify that the request is permitted by the rules. If access control policy 244 disallows access to the combined dataset 214 and / or the exported dataset 216 based on the attributes of the request and / or the entity issuing the request, then online service 222 discards or rejects the request.

[0074] In some embodiments, access control policy 244 includes data access restrictions associated with rules and / or roles. For example, the rules and / or roles in access control policy 244 include a “quota” of the maximum amount 230 of data that can be retrieved within a given time period (e.g., a day, a week, a month, etc.). When a request matches a rule and / or role, platform 202 determines the amount of data remaining in the corresponding quota. If the amount of data returned in response to the request exceeds the quota, the platform rejects, discards, or delays the request (e.g., until processing the request no longer exceeds the quota).

[0075] After access control policy 244 determines that a request is permitted, online service 222 transforms the request into a query (e.g., query 228) and modifies the query 226 to enforce compliance policy 236, privacy policy 238, and / or access control policy 244. This modification 226 can be made in a similar manner to modification 210 of the combined query 212. For example, online service 222 transforms the query into a relational operator tree including nodes representing projection and selection operators to be applied to the combined dataset 214 and / or the derived dataset 216. Online service 222 then adds additional operators representing the rules in compliance policy 236, privacy policy 238, and / or access control policy 244 to the relational operator tree. Each additional operator can be used to filter, obfuscate, or otherwise apply the corresponding rule to rows and / or columns in the combined dataset 214 and / or the derived dataset 216. Online service 222 can then execute modified queries to retrieve data 230 from combined dataset 214 and / or exported dataset 216 in accordance with compliance policy 236, privacy policy 238 and / or access control policy 244.

[0076] Unlike the operator added as modification 210 to the combining query 212, the operator added as modification 226 to the query 228 by the online service 222 can be specific to the entity making the corresponding request 232 and / or the attributes of request 232. For example, the online service 222 matches the entity making the request and / or one or more attributes of the request with one or more rules and / or roles specified in compliance policy 236, privacy policy 238, and / or access control policy 244. The online service 222 then translates the request into one or more queries 228 in the data storage 234 and makes modification 226 to apply the rules and / or roles to the query 228. In other words, the online service 222 customizes the query 228 to enforce policies specific to the entity and / or the context of the corresponding request. Alternatively, the merging device 208 modifies the combining query 212 by modification 210 to enforce policies specific to the data provider of the corresponding input dataset and / or the entity represented by the data in the input dataset.

[0077] In one or more embodiments, access control policy 244 is used to manage access to combined dataset 214 and / or exported dataset 216 by applications implementing one or more components of platform 202. For example, the application includes data processing logic 240 for generating combined dataset 214 and / or exported dataset 216, and an implementation of an online service 222 for processing queries on combined dataset 214 and / or exported dataset 216. If abuse or misconduct is detected in the application and / or among its users, the data provider's administrator may revoke the application's access to combined dataset 214, exported dataset 216, and / or other components of platform 202.

[0078] In some embodiments, merging device 208, online service 222, data storage 234, and / or other components of platform 202 prevent data associated with different entities from being stored and / or processed in a combined or mixed manner during the processing of request 232 and / or query 228 for combined dataset 214 and / or exported dataset 216. These entities may include, but are not limited to, entities represented by data in combined dataset 214 and / or exported dataset 216 (e.g., users, companies, schools, and / or groups with records in combined dataset 214 and / or exported dataset 216) and / or data providers used to generate input datasets for combined dataset 214 and / or exported dataset 216. These entities may be identified in compliance policy 236, privacy policy 238, access control policy 244, and / or other configurable policies associated with combined dataset 214 and / or exported dataset 216. For example, one or more policies in metadata storage 224 may specify a list of entity identifiers and / or entity types that will be isolated from each other in data storage 234.

[0079] Furthermore, such data isolation can be performed on a logical and / or physical basis. First, platform 202 includes the capability to logically isolate data associated with different entities in combined dataset 214 and / or derived dataset 216. Such logical isolation can be achieved by encrypting data in data store 234 using a separate key for each entity. A request for data 230 associated with a given entity is then processed by the given data store, which decrypts the data 230 using the entity's key and returns the decrypted data. The key can be mapped to an entity ID in metadata store 224 and provided for processing the request (e.g., based on access control policy 244 in metadata store 224) after the request is determined to be valid. Because data associated with other entities is encrypted using a different key, such data will not be returned in a decrypted state even if the request inadvertently retrieves data from one or more data stores 234.

[0080] Such logical isolation is also enforced, or alternatively, through additional modifications 210 to the combination query 212 of the combined dataset 214 and / or the derived dataset 216 and / or modifications 226 to query 228. For example, the merging device 208 may modify a given combination query 212 that includes aggregations of fields in the input dataset (e.g., sum, count, average, minimum, maximum, etc.) to include a "group by" clause that logically isolates the results of combination query 212 by the unique IDs of the entities (e.g., company, customer, etc.). Subsequently, the records in the combined dataset 214 include aggregated values ​​of fields that do not combine or mix data from different entities. In another example, online service 222 processes request 232 for data 230 in data store 234 based on the identity and / or access permissions associated with the entity issuing request 232. If a given entity is authorized to access only a subset of the combined dataset 214 and / or the derived dataset 216, the online service 222 translates the request 232 from the entity into a query 228 to the data store 234 in a manner that restricts the entity's access to other parts of the combined dataset 214 and / or the derived dataset 216. The online service 222 also, or alternatively, transmits the query 228 to the data store 234 using a security identity that reflects the entity's access level to the combined dataset 214 and / or the derived dataset. The data store 234 then returns the data 230 authorized for access by the entity in response to the query 228.

[0081] Such logical isolation can also be provided using multiple instances of platform 202 and / or components within platform 202. For example, different instances of platform 202 may be created for each combined dataset (e.g., combined dataset 214), which combines data from multiple input datasets and / or any derived datasets generated from the combined datasets (e.g., derived dataset 216). Each instance of platform 202 may also reside in different virtual networks within a cloud computing and / or distributed system. As a result, data residing within a given instance of platform 202 is inaccessible to other instances of platform 202.

[0082] Secondly, platform 202 includes the ability to physically isolate and combine data associated with different entities in dataset 214 and / or derived dataset 216. For example, a given instance of platform 202 can be deployed in a resource set and / or in an environment physically separate from other instances of platform 202 (e.g., within the infrastructure associated with the data provider of the input dataset, rather than within an infrastructure shared by multiple data providers). As a result, the compute and storage resources used by this instance of platform 202 are physically isolated from the compute and storage resources used by other instances of platform 202, which further protects the data in the instance from unauthorized and / or unintentional access.

[0083] One or more components of platform 202 additionally include functionality supporting: auditing access to combined dataset 214 and / or exported dataset 216 by the data provider's administrator, and exploration of data 230 in combined dataset 214 and / or exported dataset 216. For example, online service 222 and / or another component of platform 202 maintain access logs to the input dataset, combined dataset 214, exported dataset 216, and / or other data managed by platform 202. Entries in the logs represent reads, writes, and / or transformations of the data. Each entry identifies the entity accessing the data, the data accessed by the entity, the time of access, the type of access (e.g., reading from a dataset, writing to a dataset, creating a dataset from one or more other datasets, etc.), and / or the result of the access (e.g., success, failure, reason for failure, etc.). Subsequently, the administrator account of platform 202 can access the logs to allow the data provider's administrator to monitor the use of data by individual entities and / or detect potential errors or problems associated with access to the data by entities.

[0084] In another example, platform 202 includes a "data browser" application that allows administrators of each data provider to view data from data providers stored within platform 202. If an administrator from a first data provider wishes to view data from a second data provider, the administrator can submit a request to access the data. The request includes the administrator's identity, the data to be accessed, the reason for access, and / or the duration of access. If the request is approved by another administrator of the second data provider, one or more components of platform 202 can update access control policy 244 to reflect the approval, and the administrator from the first data provider can issue query 228 to access the data.

[0085] Those skilled in the art will realize that, Figure 2 The system can be implemented in a variety of ways. First, the merging device 208, online service 222, metadata storage 224, and / or data storage 234 can be provided by a single physical machine, multiple computer systems, one or more virtual machines, a grid, one or more databases, one or more file systems, and / or a cloud computing system. The merging device 208 and online service 222 can also be implemented together and / or separately by one or more hardware and / or software components and / or layers. For example, the merging device 208 can be provided by a cluster that generates a combined dataset 214 and / or an exported dataset 216 on an offline or batch processing basis. On the other hand, the online service 222 includes one or more service endpoints that process requests 232 for data in the combined dataset 214 and / or the exported dataset 216 on a real-time or near-real-time basis.

[0086] Secondly, the system's functionality can be used with various types of data and / or data storage. For example, one or more instances of platform 202 can be used to merge, store, and / or manage access to streaming data, images, audio, video, telemetry data, and / or other types of data. In another example, data storage 234 may include a graph database, a relational database, an insight platform, a distributed streaming platform, and / or another type of system for storing and / or delivering data.

[0087] Figure 3A The disclosed embodiments illustrate a method for combining queries (e.g., Figure 2 An exemplary relational operator tree for a combined query (212). The relational operator tree includes multiple nodes 302-310 representing relational operators in the combined query. Each node in the tree operates on its child nodes, and the root node 302 of the tree represents the result of the combined query.

[0088] Specifically, the relational operator tree includes nodes 308-310 representing table scans of the "Employee" dataset and the "User" dataset, respectively. Node 304 is the parent node of node 308 and specifies the projection from the "Employee" dataset onto the fields named "id", "name", and "title". Node 306 is the parent node of node 306 and specifies the projection from the "User" dataset onto the fields named "id", "skills", and "spam sender".

[0089] Root node 302 is the parent node of nodes 304 and 306 and represents the combination of the "Employee" dataset and the "User" dataset. As mentioned above, this combination may include a combination predicate represented by matches between records in the "Employee" dataset and records in the "User" dataset. Matches are generated based on the bindings between user accounts represented by records in the "Employee" dataset and records and employee accounts represented by records in the "User" dataset. Matches are also, or alternatively, generated based on comparisons of fields in the records, such as first name, last name, email address, location, title, job title, company, and / or other attributes of users and employees.

[0090] In one or more embodiments, nodes 302-310 are used to execute the following GraphQL combined queries:

[0091] Query {

[0092]

[0093] The above combined query applies to records in the "Employees" and "Users" datasets that have the following patterns:

[0094] Empoyee {

[0095] id: String

[0096] name: String

[0097] title: String

[0098] }

[0099] User {

[0100] id: String

[0101] name: String

[0102] title: String

[0103] skills: Skill[]

[0104] spammer: Boolean @ Internal

[0105] }

[0106] Skill {

[0107] name: String

[0108] experiseScore: float

[0109] }

[0110] Therefore, the combination query is used to combine the "id," "name," and "title" fields from records in the "employees" dataset with the "skills" field from corresponding records in the "users" dataset. For example, based on an exact match between the "name" and "title" fields in the records, the combination query is used to combine records from the two datasets (i.e., when a record in the "employees" dataset has the same "name" and "title" as a record in the "users" dataset, the two records are combined). On the other hand, the "id" field in the two datasets represents two different primary keys, which are assigned to records in completely different ways. Because there is no known mapping between the "id" fields in the "users" and "employees" datasets, the combination of records from the two datasets can be performed without comparing or matching the "id" field.

[0111] The "Spam Sender" field in the "Users" dataset is an internal field and is not exposed to end users who access the combined dataset. Instead, it is controlled by the platform (e.g., Figure 2Platform 202) uses the "spam sender" field to enforce compliance policies for the "users" and / or "employees" datasets in processing the combined query, as described below. Figure 3B As discussed further.

[0112] Figure 3B An exemplary relational operator tree for combining queries, according to the disclosed embodiments, is shown. More specifically, Figure 3B This shows the result after node 312 is inserted between node 306 and node 310. Figure 3A Relational operator tree.

[0113] Node 312 represents an operator used to enforce one or more compliance rules on the "Users" dataset and / or on a combination of the "Users" dataset and another dataset. (As in...) Figure 3B As shown, node 312 specifies the filtering of records in the "Users" dataset that have a "true" value for the "Spam Sender" field. Therefore, node 312 can represent a compliance rule used to prevent users identified as spammers from being included in the combined dataset.

[0114] After the relational operator tree is modified to include node 312 and / or other nodes that apply compliance rules to one or both input datasets, the relational operator tree can be compiled into an offline or batch job that performs the combination of the input datasets. The output of the job includes a combined dataset, where rows represent users found in both the "Employee" and "User" datasets. Each row in the combined dataset includes "id," "name," and "title" fields from the "Employee" dataset and a "skill" field from the "User" dataset.

[0115] The dataset can be further processed and combined to generate an exported dataset with the following patterns:

[0116] SkiledEmployee{

[0117] id: String @Unique

[0118] name: String @ Searchable

[0119] title: String

[0120] skills: String[] @Searchable

[0121] }

[0122] For example, an exported dataset can be generated by applying one or more thresholds to the "expertiseScore" component of entries in the "Skills" field of the combined dataset. Then, the "Skills" field in each record of the "SkilledEmployees" exported dataset stores a list of strings representing the skills for the corresponding user whose "expertiseScore" is greater than a threshold.

[0123] The schema of the exported dataset can be further used to store the exported dataset and / or process queries on the exported dataset. For example, the "unique" keyword next to the "id" field indicates that a record in the exported dataset can be uniquely identified by the value of that field. As a result, records in the exported dataset can be stored in a key-value store (e.g., Figure 2 In a key-value store (218), the value of the "id" field can be used as a key to retrieve records from the key-value store. In another example, the "searchable" keyword next to the "name" and "skills" fields indicates that records in the exported dataset can be searched by the values ​​of these fields. Subsequently, records in the exported dataset can be stored in a search index (e.g., […]) in a way that allows searching and / or retrieving records by the values ​​of the "name" and "skills" fields. Figure 2 In the search index 220).

[0124] Figure 4 A flowchart illustrating a process for managing the merging and access of data according to the disclosed embodiments is shown. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Therefore, in Figure 4 The specific arrangement of the steps shown should not be construed as limiting the scope of the embodiments.

[0125] Initially, matches are generated between entities in the first set of the first dataset and entities in the second set of the second dataset based on comparisons of fields from a first set of fields in a first dataset from a first data provider and fields from a second set of fields in a second dataset from a second data provider (operation 402). For example, the entities may include users, employees, companies, schools, skills, and / or jobs. Each entity may be identified by a distinct ID in each dataset, excluding matching entities across datasets based on entity IDs within the datasets. Alternatively, binding tables and / or columns between different types of accounts represented by IDs in the datasets may be used to establish “very high confidence” matches between entities in the two sets. Exact matches between email addresses in the two datasets are also, or alternatively, used to generate “high confidence” matches between entities in the two sets. Exact matches between first name, last name, title, job title, location, qualifications, company, and / or other attributes in the fields of the first and second sets are also, or alternatively, used to generate “medium confidence” matches between entities in the two sets. The matches and corresponding confidence levels can be stored in a table and used to combine the corresponding records in the dataset during the execution of a combined query, as described in further detail below.

[0126] Continuing the example above, the machine learning model may also be applied, or alternatively, to features (e.g., first and last name, title, job title, location, company, etc.) that include the values ​​of fields from both the first and second sets, and / or to comparisons of fields from both sets (e.g., Boolean values ​​indicating whether a field in a record from the first dataset exactly matches a corresponding field in a record from the second dataset, edit distance between two fields, etc.). The machine learning model may include (but is not limited to): regression models, neural networks, tree-based models, and / or deep learning models. In response to the input features, the machine learning model outputs a matching score for a given pair of records (including a first record from the first dataset and a second record from the second dataset), the matching score representing the confidence level that the first and second records represent the same entity. When the matching score exceeds a threshold, a match is generated between the two records.

[0127] Next, the combine query used to combine the first and second datasets is modified to include one or more operators representing compliance rules for the first and / or second datasets (Operation 404). For example, the combine query can be obtained from the metadata used to combine the first and second datasets. The combine query may lack explicitly specified combine predicates; alternatively, it may include implicit (e.g., unstated or unspecified) combine predicates for combining the datasets based on the matches generated in Operation 402. One or more fields in the first and / or second datasets match the compliance rules, and the combine query is resolved into a relational operator tree. Operators representing the compliance rules are then inserted into the relational operator tree, and the modified relational operator tree is compiled into an environment-specific version of the combine query.

[0128] Then, the modified combine query is executed to generate a combined dataset that complies with compliance rules based on the first and second datasets and / or an exported dataset based on one or more fields in the combined dataset (Operation 406). Continuing with the example above, the compiled combine query is executed in the appropriate environment (e.g., an offline or batch system) to generate the combined dataset, and additional transformation queries can then be applied to the combined dataset to generate the exported dataset.

[0129] During the creation of the combined dataset, if the confidence level of the match between the entities represented by the first record in the first dataset and the second record in the second dataset exceeds a threshold, the two records can be combined. This threshold can be selected and / or adjusted based on the sensitivity or importance of the data in the first dataset, the second dataset, and / or the combined dataset (e.g., combining financial data requires a high confidence level of the match, while combining two publicly available datasets requires a lower confidence level).

[0130] During the execution of the modified merging query, the inserted operator is used to filter fields and / or records from the merging dataset so that the merging dataset does not violate compliance rules. For example, the modified merging query may include an inserted operator that enforces a privacy setting represented by a field named "emailVisible," hiding the user's email address when set to false. The operator can be added to the "SELECT" statement in the merging query and includes the following representation: (email Visible ? email : "null")

[0131] When the modified combined query is executed, the operator adds the "email" field to the "select" statement if the condition represented by the "emailVisible" field evaluates to true, and returns null if the condition evaluates to false. As a result, the operator enforces privacy settings by omitting email addresses stored in the "email" field from the combined dataset of users who have enabled privacy settings (e.g., by alternatively returning null values). In other words, the operator selectively filters fields from the combined dataset based on the values ​​of the privacy settings.

[0132] Data related to the combined and / or derived datasets is stored within the platform, which logically isolates the data from additional datasets not originating from the first and second data providers (Operation 408). Finally, during the processing of queries on the stored data, the platform modifies the queries to include one or more additional operators that enforce access control policies on the data (Operation 410).

[0133] More specifically, data from the combined and / or exported datasets can be stored in one or more data stores within the platform, based on patterns used to combine and / or export the datasets. For example, the patterns(s)(s) can specify searchable fields and / or unique fields in the combined and / or exported datasets. The platform stores a mapping of values ​​from a first set of searchable fields to records containing values ​​in a search index, and a mapping of values ​​from a second set of unique fields to records containing values ​​in a key-value store. The platform then processes queries containing values ​​from the unique fields by retrieving the corresponding records from the key-value store. Alternatively, the platform also processes queries representing data searches by matching corresponding search parameters to entries in the search index and retrieving records mapped to those entries from the search index.

[0134] To isolate the combined and / or derived datasets from additional datasets from other data providers (e.g., to ensure the integrity and / or security of each dataset), the platform can encrypt data in the combined and / or derived datasets using a first key for a first data provider and / or a second data provider, and decrypt additional datasets using one or more keys for other data providers. During data queries on the combined and / or derived datasets, the platform can decrypt data using only the first key. Therefore, even if a query unintentionally retrieves data belonging to other data providers, encryption with different keys prevents the underlying values ​​of the data from being returned in the query results.

[0135] The platform may also, or alternatively, modify the execution of the query to exclude additional datasets from the query results. For example, the platform may include a unique ID for one or more data providers associated with each dataset. Thus, the combined dataset and / or derived dataset may include IDs for a first data provider and a second data provider, and each additional dataset may include IDs for a corresponding data provider (one or more) data provider. During processing queries on the combined dataset and / or derived dataset, the platform adds clauses that group the corresponding results by ID for the first and second data providers and / or return only results including IDs for the first and second data providers to prevent aggregation of the combined dataset and / or derived dataset with additional datasets.

[0136] The combined dataset and / or exported dataset can also, or alternatively, be isolated from additional datasets from other data providers by storing the combined dataset and / or exported dataset in one instance of the platform and storing additional datasets in one or more other instances of the platform. Therefore, each instance of the platform is inaccessible to data stored in other instances of the platform. Additional physical isolation of the datasets can be provided by deploying the combined dataset and / or exported dataset in an environment physically separated from the resource collection and / or other datasets.

[0137] To enforce the access control policy, the platform verifies whether an entity requesting data from a combined dataset and / or derived dataset is authorized to access the data before processing one or more queries on behalf of the entity. For example, the platform matches the entity's identity and / or the attributes of the entity's request against one or more rules and / or roles in the access control policy. If the rules and / or roles indicate that the entity's request is valid, the platform continues processing the request. If the roles and / or roles indicate that the entity's request is invalid, the platform discards or rejects the request.

[0138] The platform may also, or alternatively, process data queries based on data access restrictions for the corresponding entity. For example, the platform receives a request from an entity for data in a combined dataset and / or exported dataset. The platform matches the entity's identity with the entity's data access restrictions for a given time period (e.g., the last day, week, month, etc.) specified in the access control policy and / or the amount of data accessed by the entity within the same time period. If the amount of data returned in response to the request exceeds the entity's data access restrictions, the platform rejects, discards, or delays the request (e.g., until processing the request no longer exceeds the data access restrictions).

[0139] The platform may also, or alternatively, transform queries for stored data into a relational operator tree, which includes nodes representing projection and selection operators to be applied to the data. The platform adds operators representing access control policies to the relational operator tree. Each additional operator can be used to filter, obfuscate, or otherwise apply the corresponding rule or role from the access control policy to rows and / or columns in the combined dataset and / or derived dataset. The rule / role may be specific to the entity performing the query (e.g., the entity requesting data to be retrieved by the query). The platform can then execute a modified query to retrieve data from the combined dataset and / or derived dataset in a manner consistent with the access control policy.

[0140] Figure 5 A computer system 500 according to a disclosed embodiment is illustrated. The computer system 500 includes a processor 502, a memory 504, a storage device 506, and / or other components in an electronic computing device. The processor 502 may support parallel processing and / or multithreaded operation with other processors in the computer system 500. The computer system 500 may also include input / output (I / O) devices such as a keyboard 508, a mouse 510, and a display 512.

[0141] Computer system 500 may include the functionality of various components that perform this embodiment. Specifically, computer system 500 may include an operating system (not shown) that coordinates the use of hardware and software resources on computer system 500, and one or more applications that perform specific tasks for users. In order to perform tasks for users, applications may obtain access to hardware resources on computer system 500 from the operating system, and interact with users through hardware and / or software frameworks provided by the operating system.

[0142] In one or more embodiments, computer system 500 provides a system for managing data merging. The system includes a merging device, one or more data storage devices, and / or an online service within a platform. The merging device generates matches between entities in the first set of the first dataset from a first data provider and entities in the second set of the second dataset from a second data provider, based on comparisons of fields from a first set in a first dataset and fields from a second set in a second dataset. The merging device also modifies the merge query used to combine the first and second datasets by generating the matches to include one or more operators representing compliance rules for the first or second dataset. The merging device then executes the modified merge query to generate a compliance-compliant merged dataset from the first and second datasets. One or more data storage devices store data associated with the merged dataset in a manner that prevents data from being combined with additional datasets not from the first and second data providers. During processing of queries on the data, the online service enforces an access control policy on the merged dataset, the access control policy being generated based on input from administrators of the first and second data providers.

[0143] Additionally, one or more components of the computer system 500 can be remotely located and connected to other components via a network. Parts of this embodiment (e.g., platform, online network, merging device, metadata storage, data storage, online services, etc.) can also reside on different nodes of the distributed system implementing this embodiment. For example, this embodiment can be implemented using a cloud computing system that manages the merging and access to datasets from multiple remote data providers.

[0144] The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which can be any device or medium capable of storing code and / or data for use by a computer system. Computer-readable storage media include, but are not limited to: volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tapes, CDs (compressed discs), DVDs (digital versatile discs or digital video discs), or other media capable of storing code and / or data now known or later developed.

[0145] The methods and processes described in the detailed description section can be embodied as code and / or data that can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and code and stored within the computer-readable storage medium.

[0146] Furthermore, the methods and processes described herein can be incorporated into hardware modules or devices. These modules or devices may include, but are not limited to: application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors (including dedicated or shared processor cores) that execute specific software modules or pieces of code at specific times, and / or other programmable logic devices now known or developed later. When a hardware module or device is activated, it executes the methods and processes contained therein.

[0147] The foregoing description of various embodiments is presented for illustrative and descriptive purposes only. It is not intended to be exhaustive or to limit the invention to the forms disclosed. Therefore, many modifications and variations will be apparent to those skilled in the art. Furthermore, the above disclosure is not intended to limit the invention.

Claims

1. A method for managing compliance entity merge and access, comprising: generating matches between entities of a first set in a first dataset from a first data provider and entities of a second set in a second dataset from a second data provider based on a comparison of fields of the first set in the first dataset to fields of the second set in the second dataset; modifying a join query for joining the first dataset and the second dataset to include one or more operators representing one or more compliance rules for the first dataset or the second dataset, wherein the join query includes a join predicate represented by the generated matches; executing the modified join query to produce a joined dataset from the first dataset and the second dataset that complies with the one or more compliance rules; storing data related to the joined dataset within a platform that logically isolates the joined dataset from one or more additional datasets that are not from the first data provider and the second data provider, wherein the storing includes storing, in the platform, a derived dataset produced from one or more fields in the joined dataset and includes storing, within one or more data stores of the platform, the derived dataset based on a schema for the derived dataset by: identifying searchable fields in the derived dataset and unique fields in the derived dataset based on the schema; storing values of a first set of the searchable fields from the derived dataset in a search index; and storing values of a second set of the unique fields in the derived dataset in a key-value store; and during processing of a query on the stored data, modifying the query to include one or more additional operators that enforce one or more access control policies for the data.

2. The method of claim 1, wherein, generating the matches between the entities of the first set and the entities of the second set includes: applying a machine learning model to the fields shared by the entities of the first set and the entities of the second set to produce a match score between pairs of entities selected from the entities of the first set and the entities of the second set; and generating matches between the entities of the first set and the entities of the second set based on the match scores.

3. The method of claim 1, wherein, modifying the join query to include the one or more operators representing the compliance rules includes: matching a field in the join query to a compliance rule representing a privacy setting for the field; and inserting an operator into a relational operator tree representing the join query, the operator including a filter selectively applied to the field based on a value of the privacy setting.

4. The method of claim 1, wherein, modifying the query to include the one or more additional operators that enforce the one or more access control policies for the data includes: processing a query of the data by an entity based on an amount of the data accessed by the query and a data access limit for the entity.

5. The method of claim 1, wherein, modifying the query to include the one or more additional operators that enforce the one or more access control policies for the data includes: verifying, prior to execution of a query of the data by an entity, that the access control policy authorizes the entity to access the data.

6. The method of claim 1, wherein, the platform logically isolates the combined dataset from the one or more additional datasets that are not from the first data provider and the second data provider by: modifying execution of the query to exclude the additional datasets from results of the query.

7. The method of claim 1, wherein, the platform logically isolates the combined dataset from the one or more additional datasets that are not from the first data provider and the second data provider by: encrypting data in the combined dataset with a first key for the first data provider or the second data provider; encrypting data in the one or more additional datasets with one or more keys for additional data providers that do not include the second data provider; and during processing of the query of the data, decrypting the data in the combined dataset using the first key. the first set of entities and the second set of entities include at least one of:

8. The method of claim 1, wherein, a user; an employee; a company; a school; a job; or a skill. the first set of fields and the second set of fields include at least one of:

9. The method of claim 1, wherein, a name; a location; a title; a position; a credential; or a company.

10. A system for managing compliance entity mergers and access, comprising: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause performance of the method of any of claims 1-9.

11. One or more storage media storing instructions that, when executed by one or more processors, cause performance of the method of any of claims 1-9. ​

Images