Data sampling for model exploration

Abstract

The disclosed embodiments provide a system for processing data. During operation, the system obtains a training data set comprising a first set of records associated with a first set of identifier (ID) values ​​and an evaluation data set comprising a second set of records associated with a second set of ID values. Next, the system selects a random subset of ID values ​​from the second set of ID values. The system then generates a sampled evaluation data set, the sampled evaluation data set comprising the first subset of records in the second set of records associated with the subset of random ID values. The system also generates a sampled training data set, the sampled training data set comprising the second subset of records in the first set of records associated with the subset of random ID values. Finally, the system outputs the sampled training data set and the sampled evaluation data set for use in training and evaluating a machine learning model.

Description

Technical Field

[0001] The disclosed embodiments relate to machine learning. More particularly, the disclosed embodiments relate to data sampling for model exploration in machine learning. Background Art

[0002] Analytics is often used to discover trends, patterns, relationships, and / or other properties associated with large, complex, interconnected, and / or multidimensional data sets. This discovered information is then used to gain insights and / or guide decisions or actions related to the data. For example, business analytics can be used to evaluate past performance, guide business planning, and / or identify actions that can improve future performance.

[0003] To gather these insights, large, feature-rich datasets are analyzed using regression models, artificial neural networks, support vector machines, decision trees, naive Bayes classifiers, and / or other types of machine learning models. The discovered information can then be used to guide decisions and / or perform actions related to the data. For example, the output of machine learning models is used to guide marketing decisions, assess risk, detect fraud, predict behavior, and / or customize or optimize the use of an app or website.

[0004] However, during the creation and training of machine learning models for analysis, a significant amount of time, effort, and overhead is spent on feature selection. For example, a dataset for a machine learning model may have thousands to millions of features, including features created from combinations of other features, while only a small fraction of the features and / or combinations may be relevant and / or important to the machine learning model. At the same time, training and / or execution of machine learning models with a large number of features and / or large datasets typically requires more memory, computing resources, and time than machine learning models with a smaller number of features and / or smaller datasets. Overly complex machine learning models that use too many features may also be at risk of overfitting.

[0005] Thus, machine learning and / or analytics may be facilitated through mechanisms for improving the creation, profiling, management, sharing, and reuse of features and / or machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of a system according to the disclosed embodiments is shown.

[0007] Figure 2 A system for processing data according to the disclosed embodiments is shown.

[0008] Figure 3A Example generation of a sampled training dataset and a sampled evaluation dataset according to the disclosed embodiments is shown.

[0009] Figure 3B Example generation of a sampled training dataset and a sampled evaluation dataset according to the disclosed embodiments is shown.

[0010] Figure 4 A flow chart illustrating the processing of data according to the disclosed embodiments is shown.

[0011] Figure 5 A flow chart illustrating a process of performing model exploration in accordance with the disclosed embodiments is shown.

[0012] Figure 6 A computer system according to the disclosed embodiments is shown.

[0013] In the drawings, like reference numerals refer to the same graphical elements. DETAILED DESCRIPTION

[0014] The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown, but is to be accorded the widest scope of the principles and features disclosed herein.

[0015] Overview

[0016] The disclosed embodiments provide methods, devices, and systems for training and evaluating machine learning models. Machine learning models include, but are not limited to, regression models, artificial neural networks, support vector machines, decision trees, random forests, gradient boosting trees, naive Bayes classifiers, Bayesian networks, deep learning models, hierarchical models, and / or ensemble models. Each machine learning model includes a global version and multiple personalized versions. For example, a machine learning model is used to generate recommendations for positions (or other content) for a user. A global version of the machine learning model is created for all users and positions, a user-specific version of the machine learning model is created for each user, and a position-specific version of the machine learning model is created for each position. The outputs of the global version, the user-specific version for a given user, and the position-specific version for a given position are combined to generate a score indicating that the user has applied for the position, clicked on the position, and / or otherwise responded positively to the recommendation for the position.

[0017] More specifically, the disclosed embodiments provide methods, apparatus, and systems for sampling data for training and evaluating machine learning models, including personalized versions for different entities. Each machine learning model is associated with a complete training dataset for training the machine learning model and a complete evaluation dataset for evaluating the performance of the machine learning model after training. As the number of records and / or features in the training dataset and / or evaluation dataset increases, additional processor and memory resources are required to perform the training and subsequent evaluation of the machine learning model.

[0018] In order to accelerate the exploration of different hyperparameters, feature sets and / or other types of training configurations using machine learning models, a random subset of entity IDs is sampled from the evaluation dataset, and a sampled training dataset and a sampled evaluation dataset are generated based on the sampled subset of entity IDs. For example, a specific percentage or proportion of user IDs are randomly selected from the evaluation dataset, and records containing user IDs from the evaluation dataset are included in the sampled evaluation dataset. Records containing the same user ID from the training dataset are similarly included in the sampled training dataset. When records in the training dataset and the evaluation dataset contain position IDs as well as user IDs, a set of position IDs is identified from the records containing the sampled user IDs in the evaluation dataset, and records containing the identified position IDs in the training dataset are also included in the sampled training dataset. Therefore, all entity IDs that appear in the sampled evaluation dataset will also appear in the sampled training dataset.

[0019] The generation of sampled evaluation datasets and sampled training datasets is also repeated multiple times, and each sampled training dataset and corresponding sampled evaluation dataset are used with different training configurations to train and evaluate the machine learning model. For example, each sampled evaluation dataset and corresponding sampled training dataset are generated from a complete evaluation dataset based on a different subset of random entity IDs. The sampled training dataset is used together with the feature set and / or one or more hyperparameters in the training configuration of the machine learning model to train the global version and personalized version of the machine learning model, and the sampled evaluation dataset is then used to evaluate the performance of the trained global version and personalized version. This process is repeated to generate multiple global version sets and multiple personalized version sets of the machine learning model based on different combinations of features and / or hyperparameters, and the highest performing machine learning model is selected for use in a field, production and / or real-world setting or environment. The selected machine learning model is then trained using the complete training dataset, and the machine learning model is evaluated using the complete evaluation dataset before the machine learning model is deployed in the environment.

[0020] By sampling entity IDs from an evaluation dataset that is smaller than the training dataset and using the sampled entity IDs to generate a sampled training dataset and a sampled evaluation dataset, the disclosed embodiments ensure that a personalized version of a machine learning model is created for the sampled entity IDs in the sampled training dataset and can subsequently be evaluated using records associated with the same entity IDs in the evaluation dataset. In addition, generating sampled training datasets and sampled evaluation datasets that are much smaller than the corresponding full training datasets and evaluation datasets speeds up the training and evaluation of machine learning models, which reduces the amount of time and / or resources required to train multiple machine learning models using multiple training configurations. The increase in training speed further enables more thorough exploration of hyperparameters, feature sets, and / or training configurations of machine learning models, thereby obtaining higher performing machine learning models.

[0021] In contrast, conventional techniques use complete training and / or evaluation datasets to train and / or evaluate machine learning models. Consequently, training and evaluating a single machine learning model requires more time and / or computing resources than training and evaluating using sampled training and evaluation datasets. The increased overhead and / or latency associated with training using the complete training dataset can interfere with the exploration and / or optimization of the machine learning model's hyperparameters, feature set, and / or training configuration, potentially leading to reduced performance of the machine learning model.

[0022] Conventional techniques also or alternatively sample individual records from the training dataset and / or evaluation dataset to reduce the size of the dataset. However, this record-based sampling removes data associated with individual entities from each dataset, which results in each personalized version of the machine learning model being trained and / or evaluated using incomplete data for the corresponding entity. As a result, the personalized version is not trained to fit all available behaviors, preferences, and / or other signals for the entities in the training dataset, and is not fully evaluated for additional signals for the entities in the evaluation dataset. In turn, this limited training and / or evaluation of the personalized version results in reduced performance of the personalized version. Therefore, the disclosed embodiments improve computer systems, applications, user experiences, tools, and / or techniques related to training machine learning models, evaluating machine learning models, feature engineering, and / or hyperparameter optimization.

[0023] Data sampling for model exploration

[0024] Figure 1 Schematic diagram of a system according to the disclosed embodiment is shown. Figure 1As shown in , the system includes an online network 118 and / or other user communities. For example, the online network 118 includes an online professional network used by a group of entities (eg, entity 1 104, entity x 106) to interact with each other in a professional and / or business context.

[0025] Entities include users who use online network 118 to establish and maintain professional connections, list work and community experiences, endorse and / or recommend each other, search for and apply for jobs, and / or perform other actions. Entities also or alternatively include companies, employers, and / or recruiters who use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and / or perform other actions.

[0026] The online network 118 includes a profile module 126 that allows an entity to create and edit a profile containing information related to the entity's professional and / or industry background, experience, summary, job titles, projects, skills, etc. The profile module 126 also allows an entity to view the profiles of other entities in the online network 118.

[0027] The profile module 126 may also or alternatively include mechanisms for assisting an entity in completing a profile. For example, the profile module 126 may suggest industries, skills, companies, schools, publications, patents, certifications, and / or other types of attributes to an entity as potential additions to the entity's profile. Suggestions may be based on predictions of missing fields, such as predicting the entity's industry based on other information in the entity's profile. Suggestions may also be used to correct existing fields, such as correcting the spelling of a company name in a profile. Suggestions may also be used to clarify existing attributes, such as changing an entity's title of "Manager" to "Engineering Manager" based on the entity's work experience.

[0028] The online network 118 also includes a search module 128 that allows an entity to search the online network 118 for people, companies, positions, and / or other job or business-related information. For example, an entity can enter one or more keywords into a search bar to find profiles, job postings, job candidates, articles, and / or other information that include and / or otherwise match the keywords. An entity can additionally use an "advanced search" feature in the online network 118 to search for profiles, positions, and / or information by category, such as first name, last name, title, company, school, location, interests, relationships, skills, industry, group, salary, experience level, and the like.

[0029] The online network 118 also includes an interaction module 130 that allows entities to interact with each other on the online network 118. For example, the interaction module 130 may allow entities to add other entities as connections, follow other entities, send and receive emails or messages with other entities, join groups, and / or interact with postings from other entities (e.g., create, share, re-share, like, and / or comment on postings from other entities).

[0030] Those skilled in the art will appreciate that the online network 118 may include other components and / or modules. For example, the online network 118 may include a homepage, a landing page, and / or a content feed that provides the entity with the latest postings, articles, and / or updates from the entity's connections and / or groups. Similarly, the online network 118 may include features or mechanisms for recommending connections, job postings, articles, and / or groups to the entity.

[0031] In one or more embodiments, data related to an entity's profile and activity on online network 118 (e.g., Data 1 122, Data x 124) is aggregated into data repository 134 for subsequent retrieval and use. For example, each profile update, profile view, connection, follow, post, comment, like, share, search, click, message, interaction with a group, address book interaction, response to a recommendation, purchase, and / or other action performed by an entity in online network 118 is tracked and stored in a database, data warehouse, cloud storage device, and / or other data storage mechanism that provides data repository 134.

[0032] The data in the data repository 134 is then used to generate recommendations and / or other insights related to listings of positions or opportunities within the online network 118. For example, one or more components of the online network 118 may track searches, clicks, views, text input, conversions, and / or other feedback during an entity's interaction with a job search tool in the online network 118. The feedback may be stored in the data repository 134 and used as training data for one or more machine learning models, and the output of the machine learning models may be used to display and / or otherwise recommend positions, ads, postings, articles, connections, products, companies, groups, and / or other types of content, entities, or actions to members of the online network 118.

[0033] More specifically, the data in the data repository 134 and one or more machine learning models are used to generate rankings of candidates associated with positions or opportunities listed within or outside of the online network 118. Figure 1As shown, the identification mechanism 108 identifies candidates 116 associated with an opportunity. For example, the identification mechanism 108 can identify the candidates 116 as users who have viewed, searched for, and / or applied for positions, titles, roles, and / or opportunities within or outside of the online network 118. The identification mechanism 108 can also or alternatively identify the candidates 116 as users and / or members of the online network 118 who have skills, work experience, and / or other attributes or qualifications that match the corresponding positions, titles, roles, and / or opportunities.

[0034] After identifying candidates 116, the profile and / or activity data of the candidates 116, along with the characteristics and / or features of the corresponding opportunity (e.g., required or desired skills, education, experience, industry, title, etc.), are input into a machine learning model. The machine learning model, in turn, outputs a score that represents the strengths of the candidate 116 with respect to the opportunity and / or qualifications associated with the opportunity (e.g., skills, current position, previous position, overall qualifications, etc.). For example, the machine learning model generates a score based on the similarity between the candidate's profile data with online networks 118 and the description of the opportunity. The model also adjusts the score based on social and / or other validation of the candidate's profile data (e.g., recognition of skills, recommendations, achievements, awards, patents, publications, reputation scores, etc.). The candidates 116 are then sorted by descending score to generate a ranking.

[0035] In turn, rankings based on the scores and / or associated insights improve the quality of candidates 116, the recommendations of opportunities for candidates 116, and / or the recommendations of candidates 116 for opportunities. Such rankings may also or alternatively enhance user activity utilizing the online network 118 and / or guide the decisions of candidates 116 and / or moderators involved in screening or placing opportunities (e.g., hiring managers, recruiters, human resources professionals, etc.). For example, one or more components of the online network 118 may display and / or otherwise output a member's position in a ranking of candidates for a position (e.g., top 10%, top 20 out of 138, etc.) to encourage members to apply for positions for which the member is highly ranked. In a second example, a component may take a candidate's relative position in a ranking for a set of positions into account during the sorting of positions as a search result in response to a job search conducted by the candidate. In a third example, after a recruiter has performed a search using job qualifications included as search parameters, the component may output the ranking of candidates for a given set of job qualifications as a search result to the recruiter. In a fourth example, the component can recommend positions to candidates based on the predicted relevance of the positions to the candidates or the predicted attractiveness of the positions to the candidates and / or the likelihood that the candidates will apply for the positions.

[0036] In one or more embodiments, some or all of the machine learning models used to generate scores and / or rankings associated with positions and candidates in the online network 118 are personalized to the positions, candidates, and / or other entities in the online network 118. Figure 2 As shown, the data repository 134 and / or another primary data storage device is queried for data 202 including a training data set 216 and an evaluation data set 218 .

[0037] The training dataset 216 includes records for training multiple machine learning models (e.g., machine learning model 1 208, machine learning model n 210), and the evaluation dataset 218 is used to evaluate the performance of each machine learning model after training is completed. For example, the training dataset 216 includes most (e.g., 70%-80%) of the data that can be used with the machine learning model, and the evaluation dataset 218 includes the remainder of the data. The evaluation dataset 218 is optionally divided into a validation dataset for evaluating the performance of individual machine learning models and a test dataset for evaluating the performance of the final model 214 selected from the machine learning models.

[0038] In one or more embodiments, records in the training dataset 216, the evaluation dataset 218, and / or other data 202 in the data repository 134 include standardized, organized, and / or categorized attributes. For example, the skills in the data 202 are organized into a hierarchical taxonomy that is stored in the data repository 134 and / or another repository. The taxonomy models relationships between skills (e.g., "Java programming" is related to or is a subset of "software engineering") and / or standardizes skills that are identical or highly related (e.g., "Java programming," "Java development," "Android development," and "Java programming language" are standardized to "Java").

[0039] In another example, the locations in the data repository 134 include cities, metropolitan areas, states, countries, continents, and / or other standardized geographic regions. Similar to the standardized skills, the locations can be organized into hierarchical classifications (e.g., cities are organized under states, states are organized under countries, countries are organized under continents, etc.).

[0040] In a third example, the data repository 134 includes standardized company names for a set of known and / or verified companies associated with members and / or positions. In a fourth example, the data repository 134 includes standardized titles, seniority, and / or industries for various positions, members, and / or companies in an online network. In a fifth example, the data repository 134 includes standardized time periods (e.g., daily, weekly, monthly, quarterly, annually, etc.) that can be used to retrieve profile data 216, user activity 218, and / or other data 202 represented by time periods (e.g., positions started in a given month or year, college graduations within five years, job listings posted within a two-week period, etc.). In a sixth example, the data repository 134 includes standardized job functions, such as "accounting," "consulting," "education," "engineering," "finance," "health care services," "information technology," "legal," "operations," "real estate," "research," and / or "sales."

[0041] In some embodiments, the standardized attributes in the data repository 134 are represented by unique identifiers (IDs) in the corresponding taxonomies. For example, each standardized skill is represented by a numeric skill ID in the data repository 134, each standardized title is represented by a numeric title ID in the data repository 134, each standardized position is represented by a numeric position ID in the data repository 134, and / or each standardized company name (e.g., for companies above a certain size and / or exposure level) is represented by a numeric company ID in the data repository 134.

[0042] In one or more embodiments, each machine learning model includes a global version (e.g., global versions 232-234) for use with all entities (e.g., members, positions, companies, content, etc.) in the training dataset 216 and the evaluation dataset 218. Each machine learning model also includes a set of personalized versions (e.g., personalized versions 236-238) that are customized or tailored for individual entities.

[0043] For example, the machine learning model created by the discovery device 212 includes a global version, a set of personalized user-specific versions, and a set of personalized position-specific versions. The global version tracks the behavior or preferences of all candidates with respect to all positions in the data repository 134. Each user-specific version of the model is customized to the individual behavior or preferences of the corresponding candidate with respect to certain position characteristics (e.g., the candidate's personal preference for positions that match the candidate's skills). Each position-specific version identifies the relevance of the corresponding position to certain candidate characteristics or the attractiveness of the corresponding position for certain candidate characteristics (e.g., the likelihood that the position will attract candidates whose preferred skills match).

[0044] The outputs of the global version, the user-specific version for a candidate, and / or the job-specific version for a given job are combined to generate a score representing the predicted probability that a user will apply for a job, click on a job, and / or otherwise respond positively to an impression of a job after being presented with the job. Continuing with the example above, the following formula is used to express the generalized linear mixed model for predicting the probability that member m will apply for job j using logistic regression:

[0045] g(E[y mjt ])=x′ mjt b+s′ j α m +q′ m β j

[0046] in, is the link function of the model, b is the coefficient vector of fixed effects representing the global version of the model, and α m is a coefficient vector representing the random effects of the user-specific version of the model for member m, and β j is the coefficient vector representing the random effects of the position-specific version of the model for position j. In addition, x mjt represents the global version of the feature vector, which contains the member features of member m, the position features of position j, derived features and / or features associated with context t. Finally, s j represents the feature vector of position j (i.e., position feature 212), and q m Represents the feature vector of member m (ie, member feature 208 ).

[0047] When member m provides multiple responses to different positions, the member’s personalized coefficient vector α can be accurately estimated m , and the ratings and / or recommendations can be personalized for the member. In contrast, when member m lacks a previous response to a post, α m The posterior mean of is close to 0, and the output of the machine learning model falls back to x′ mjt Similarly, when position j includes multiple responses from members, the personalized coefficient vector β for that position is j can be used to adjust the output of the machine learning model for that position. On the other hand, the lack of response to the position will result in β j The posterior mean of is close to 0, and the global version of the machine learning model overwhelmingly contributes to the rating between the position and a given member.

[0048] A more general formulation of the generalized linear mixed model above includes the following representation:

[0049]

[0050]

[0051] Regarding the above representation, let R denote the set of random effect types being modeled. Also let i(r,n) denote an indexing function that retrieves the index of the random effect type r in the nth training example (e.g., if random effect type r represents a random effect for each position, then i(r,n) returns the position ID associated with example n). Given the indexing function, γ ri(r,n) represents the random effect coefficient vector of random effect type r in the nth training sample, and represents the corresponding feature vector of random effect type r in the nth training sample.

[0052] In one or more embodiments, the exploration device 212 trains each machine learning model using a different training configuration (e.g., training configuration 1 244, training configuration n 246). Each training configuration includes a set of features to be input into the corresponding machine learning model.

[0053] Continuing with the above example, the training configuration includes different feature sets for a global version, a user-specific version, and a position-specific version of the machine learning model that recommends positions to candidates (or recommends candidates to positions). The features used in the global version include, but are not limited to, title, skills, education, qualifications, industry, location, and / or features from an online system (e.g., Figure 1 Features also include declared or inferred attributes of the position, such as the position's title, industry, qualifications, desired skills and experience, salary range, and / or location. Features additionally include the context of the candidate's impression of the position, such as the time and place (e.g., geographic location, application, website, webpage, device or platform used to access the online system, etc.) where the position will be presented or recommended to the candidate. Finally, features include derived features calculated based on member and position attributes, such as vector products, cosine similarities, statistical results, and / or other combinations, aggregations, scaling, and / or transformations of member and / or position attributes.

[0054] Features used by the user-specific version of the machine learning model include the candidate's recent and / or long-term history of interaction with the position, the candidate's stated or inferred position preferences (e.g., preferred title, function, industry, seniority, location, company, etc.), and / or other indications of the candidate's job search behavior or preferences. These features also include some or all of the position's attributes. Features used by the position-specific version of the machine learning model include attributes of the candidate that can be used to predict the candidate's interest in the position.

[0055] Each training configuration also or alternatively includes one or more hyperparameters of the corresponding machine learning model. For example, hyperparameters include convergence parameters that adjust the convergence speed of the machine learning model. In another example, hyperparameters include clustering parameters that control the amount of clustering (e.g., the number of clusters) in clustering techniques and / or classification techniques that utilize clustering. In a third example, hyperparameters specify the feature complexity of features input to the machine learning model, such as the number of topics or items in an n-gram used during natural language processing. In a fourth example, hyperparameters include model training parameters that control the training of the machine learning model, such as the step size or momentum in a gradient descent technique. In a fifth example, hyperparameters include model selection parameters that specify the type of machine learning model (e.g., logistic regression, artificial neural network, support vector machine, decision tree, deep learning model, etc.). In a sixth example, hyperparameters include decay parameters, such as parameters used to determine the annealing schedule in simulated annealing. In a seventh example, hyperparameters include "hyper-hyperparameters," such as a starting position, a default value, and / or other parameters related to exploring the search space of other hyperparameters. In an eighth example, hyperparameters include thresholds, such as a threshold for removing links with low weights in an artificial neural network. In the ninth example, the hyperparameters include

[0056] In one or more embodiments, some or all of the training configurations used by exploration device 212 to train the machine learning model include one or more hyperparameters that control the amount of personalization of the machine learning model. For example, the generalized linear mixed model formulation described above includes the following objective function:

[0057]

[0058] The above objective function includes the model parameters {b, {Γ r}}, where b represents the fixed effect coefficient of the global version, and Γ r ={γ rl} represents the random effect coefficient of random effect type r, where l ranges from 1 to the total number of instances of random effect type r (e.g., when r represents member, the total number of members in the dataset). In the objective function, p(s i ) represents the likelihood function of a given response associated with sample i in the training dataset Ω, given b and Γ.

[0059] The above objective function also includes the hyperparameter λ b and {λ r}.λ b represents the contribution of the fixed effect coefficient to the objective function, and each λ r It represents the contribution of the random effect coefficient of random effect type r to the objective function.

[0060] After the exploration device 212 has trained the global version and the personalized version of a given machine learning model using the training dataset 216 and the corresponding training configuration, the exploration device 212 uses the evaluation dataset 218 to evaluate the performance of the machine learning model (e.g., performance 1 240, performance n 242). For example, the exploration device 212 applies the global version of the machine learning model to each record in the evaluation dataset 218, and applies the personalized version of the machine learning model to the records with the corresponding entity ID in the evaluation dataset 218. For each record in the evaluation dataset 218, the exploration device 212 aggregates the outputs of the global version and the personalized version of the machine learning model into a score, and compares the score with the corresponding annotation of the record. The exploration device 212 then evaluates the performance of the machine learning model by calculating precision, recall, area under the receiver operating characteristic (ROC) curve (AUC), F1 score, observed / expected (OZE) ratio, mean absolute error, mean squared error, and / or another performance metric based on the comparison.

[0061] In one or more embodiments, the exploration device 212 trains and evaluates multiple machine learning models using different training configurations to explore different feature sets and / or hyperparameters of the machine learning models. In turn, the exploration device 212 identifies the feature set and / or hyperparameters of the machine learning model that produces the best performance.

[0062] For example, the exploration device 212 first trains a set of machine learning models using different feature sets from the training dataset 216. The exploration device 212 uses the evaluation dataset 218 to evaluate the performance of the machine learning models, identifies the machine learning model with the best performance, and obtains the feature set input to the machine learning model from the corresponding training configuration. The exploration device 212 trains a second set of machine learning models using the same feature set and different values ​​of the first hyperparameter, and selects the machine learning model with the highest performance from the second set of machine learning models. The exploration device 212 uses the feature set, the value of the first hyperparameter from the highest-performing model in the second set of machine learning models, and different values ​​of the second hyperparameter to train a third set of machine learning models. The exploration device 212 then obtains the value of the second hyperparameter associated with the highest-performing machine learning model in the third set of machine learning models. The exploration device 212 optionally repeats the following process: changing a single attribute (e.g., feature set, hyperparameter, etc.) used to train the machine learning models, identifying the value of the attribute that produces the best performance, fixing that value, and exploring subsequent attributes used to train the machine learning models until optimal values ​​have been identified for all relevant attributes.

[0063] In one or more embodiments, Figure 2The system includes functionality for reducing the size of training data used to train a machine learning model and evaluation data used to evaluate the performance of the trained machine learning model. The smaller amount of data reduces the amount of time, resources, and / or overhead required to train and evaluate the machine learning model, which improves the efficiency and / or thoroughness of selecting hyperparameters, feature sets, and / or other types of training configurations for the machine learning model.

[0064] More specifically, the sampling device 204 generates a sampled training dataset 224 including records 228 sampled from the training dataset 216 and a sampled evaluation dataset 226 including records 230 sampled from the evaluation dataset 218. For example, the records 228 in the sampled training dataset 224 represent a certain proportion or percentage of the records in the training dataset 216. Similarly, the records 230 in the sampled evaluation dataset 226 represent a certain proportion or percentage of the records in the evaluation dataset 218.

[0065] In one or more embodiments, the sampling device 204 samples records 228-230 from the training dataset 216 and the evaluation dataset 218 based on the entity IDs associated with the records 228-230. In these embodiments, the sampling device 204 selects a random subset 222 of the evaluation entity IDs 220 from the evaluation dataset 218 and populates the sampled evaluation dataset 226 with the records 230 associated with the selected entity IDs from the evaluation dataset 218. The sampling device 204 then identifies the records 228 in the training dataset 216 that are associated with the same entity ID and uses the identified records 228 to populate the sampled training dataset 224.

[0066] Because the number of entity IDs in the evaluation dataset 218 is significantly smaller than the number of entity IDs in the training dataset 216, generating the sampled training dataset 224 and the sampled evaluation dataset 226 based on a given random subset 222 of the evaluation entity IDs 220 in the evaluation dataset 218 ensures that the entity IDs in the sampled training dataset 224 are also found in the sampled evaluation dataset 226. In turn, a personalized version of the machine learning model (e.g., personalized versions 236-238) can be generated based on the records 228 associated with the sampled entity IDs in the sampled training dataset 224, and the same personalized version can be evaluated using the records 230 associated with the sampled entity IDs in the sampled evaluation dataset 226.

[0067] use Figure 3A The operation of the sampling device 204 is illustrated with reference to the exemplary sampled training data set 224, sampled evaluation data set 226, training data set 216, and evaluation data set 218 in FIG. Figure 3AAs shown, the evaluation dataset 218 includes records 302-308 grouped under member IDs 2, 3, 5, and 7, respectively. The training dataset 216 includes a larger number of records 310-322 grouped under member IDs 1, 2, 3, 4, 5, 6, and 7, respectively.

[0068] To generate the sampled evaluation data set 226, the sampling device 204 randomly selects member IDs 3 and 7 from the evaluation data set 218. The sampling device 204 copies the records 304 grouped under member ID 3 in the evaluation data set 218 to the corresponding record set 324 in the sampled evaluation data set 226, and copies the records 308 grouped under member ID 7 in the evaluation data set 218 to the corresponding record set 326 in the sampled evaluation data set 226.

[0069] To generate the sampled training dataset 224, the sampling device 204 matches the sampled member ID 3 with a set of records 314 in the training dataset 216, and matches the sampled member ID 7 with another set of records 322 in the training dataset 216. The sampling device 204 then copies the records 314 in the training dataset 216 grouped under the member ID 3 to a corresponding set of records 328 in the sampled training dataset 224, and copies the records 322 in the training dataset 216 grouped under the member ID 7 to a corresponding set of records 330 in the sampled training dataset 224. Therefore, both the sampled evaluation dataset 226 and the sampled training dataset 224 contain records associated with the sampled member IDs 3 and 7 (e.g., records 324-326 and 328-330).

[0070] In one or more embodiments, sampling device 204 includes functionality for generating a sampled training dataset 224 and a sampled evaluation dataset 226 for machine learning models that include multiple types of random effects. For example, each machine learning model includes one set of personalized versions suitable for a set of members and another set of personalized versions suitable for a set of positions. Thus, records in training dataset 216 and evaluation dataset 218 are associated with two sets of entity IDs: a first set of member IDs for members and a second set of entity IDs for positions.

[0071] To account for multiple random effect types in the training dataset 216 and the evaluation dataset 218, the sampling device 204 selects combinations of evaluation entity IDs 220 found in the records of the evaluation dataset 218 to include in the random subset 222. The sampling device 204 populates the sampled evaluation dataset 226 with records 230 associated with the selected combinations of entity IDs from the evaluation dataset 218. The sampling device 204 then identifies records 228 in the training dataset 216 that contain one or more of the selected entity IDs and uses the identified records 228 to populate the sampled training dataset 224. Thus, the sampled training dataset 224 includes all records 230 in the training dataset 216 that are associated with all entity IDs in the random subset 222.

[0072] use Figure 3B The exemplary sampled training dataset 224, sampled evaluation dataset 226, training dataset 216, and evaluation dataset 218 in FIG. 2 are used to illustrate the operation of the sampling device 204 in generating the sampled training dataset 224 and the sampled evaluation dataset 226 in the presence of multiple random effects. Figure 3B As shown, the evaluation data set 218 includes records 332-338 grouped under member ID and position ID pairs. Record 332 is grouped under member ID 2 and position ID 8, record 334 is grouped under member ID 3 and position ID 5, record 336 is grouped under member ID 5 and position ID 2, and record 338 is grouped under member ID 7 and position ID 4.

[0073] Similarly, training data set 216 includes a larger number of records 340-352 grouped under the same and / or different member ID and position ID pairs. Record 340 is grouped under member ID 1 and position ID 5, record 342 is grouped under member ID 2 and position ID 15, record 344 is grouped under member ID 3 and position ID 10, record 346 is grouped under member ID 4 and position ID 7, record 348 is grouped under member ID 5 and position ID 8, record 350 is grouped under member ID 6 and position ID 4, and record 352 is grouped under member ID 7 and position ID 4.

[0074] In the evaluation dataset 218 and the training dataset 216, records associated with a given member ID and position ID pair include information related to the interactions and / or compatibility between the corresponding member and the position. For example, a record containing a member ID and a position ID includes a relevance score between the corresponding member and the position, the context in which the position was recommended to the member (e.g., the time and location of the recommendation), and / or the member's response to the recommendation.

[0075] To generate the sampled evaluation data set 226, the sampling device 204 randomly selects member IDs 3 and 7 and / or position IDs 5 and 4 from the evaluation data set 218. The sampling device 204 copies the records 334 in the evaluation data set 218 that are grouped under member ID 3 and position ID 5 to the corresponding record set 354 in the sampled evaluation data set 226. The sampling device 204 also copies the records 338 in the evaluation data set 218 that are grouped under member ID 7 and position ID 4 to the corresponding record set 356 in the sampled evaluation data set 226.

[0076] To generate sampled training dataset 224, sampling device 204 matches member ID 3 in record 334 to record set 344 in training dataset 216, and matches member ID 7 to another record set 352 in training dataset 216. Sampling device 204 also matches position ID 5 in record 334 to record set 340 in training dataset 216, and matches position ID 4 to two record sets 350 and 352 in training dataset 216. Sampling device 204 then copies record 340 to a corresponding record set 358 in sampled training dataset 224, and copies record 344 to a corresponding record set 360 in sampled training dataset 244. Sampling device 204 also copies record 350 to a corresponding record set 362 in sampled training dataset 224, and copies record 352 to a corresponding record set 364 in sampled training dataset 224. Thus, both the sampled evaluation dataset 226 and the sampled training dataset 224 contain records associated with sampled member IDs 3 and 7 and / or sampled position IDs 4 and 5 (eg, records 324 - 326 and 328 - 330 ).

[0077] After the sampling device 204 generates a sampled training dataset 224 from the training dataset 216 and generates a sampled evaluation dataset 226 from the sampled evaluation dataset 226, the exploration device 212 uses the sampled training dataset 224 and the corresponding training configuration to train the machine learning model. After the training is completed, the exploration device 212 uses the sampled evaluation dataset 226 to evaluate the performance of the machine learning model.

[0078] To explore different training configurations for the machine learning model, the sampling device 204 generates a plurality of sampled training datasets and sampled evaluation datasets, each sampled evaluation dataset 226 and corresponding sampled training dataset 224 containing records (e.g., records 228-230) associated with a different random subset 222 of the evaluation entity IDs 220 from the evaluation dataset 218. The exploration device 212 trains and evaluates a plurality of machine learning models using the sampled training datasets, the sampled evaluation datasets, and the plurality of training configurations.

[0079] Because each sampled training dataset 224 and sampled evaluation dataset 226 includes a portion of the data in the corresponding training dataset 216 and evaluation dataset 218, the exploration device 212 is able to train and evaluate each machine learning model in a corresponding portion of the time and / or with a corresponding portion of the resources required to train and execute the model using the complete training dataset 216 and evaluation dataset 218. For example, the exploration device 212 would need to execute continuously for multiple hours on an allocated set of memory and processor resources to train and evaluate a machine learning model using the billions of records in the training dataset 216 and the evaluation dataset 218. In contrast, the exploration device 212 is able to train and evaluate multiple machine learning models in parallel on the same set of resources using a much smaller number of records (e.g., records 228-230) in the sampled training dataset 224 and the sampled evaluation dataset 226.

[0080] Furthermore, the exploration device 212 uses the accelerated training and evaluation of the machine learning model to explore various feature sets and / or hyperparameters in the training configuration of the machine learning model. As described above, the exploration device 212 includes functionality for training multiple machine learning model sets. Each machine learning model set is used to explore the values ​​of one or more attributes in the training configuration, while the remaining attributes are fixed. After the machine learning model set is trained, the performance of the machine learning models in the set is compared, and the one or more values ​​of the explored attributes used to produce the highest performing machine learning model in the set are used in subsequent training configurations.

[0081] After all attributes in the training configuration have been explored, the exploration device 212 generates the best performing attribute value combination based on the training configuration and provides the attribute value set to the deployment device 206. The deployment device 206 uses the identified attribute values ​​and the training data set 216 to train the global version and the personalized version of the final model 214. The deployment device 206 then uses the evaluation data set 218 and / or the test data set in the evaluation data set 218 to evaluate the performance of the final model 214. After verifying the performance of the final model 214, the deployment device 206 deploys the final model 214 in a field, production and / or real-world environment or setting. In the deployed environment or setting, the deployed final model 214 is used to supplement or perform tasks such as managing the execution of an application, personalizing the user experience, managing relationships, making clinical decisions, executing transactions, operating an autonomous vehicle or machine, and / or analyzing indicators or measurements.

[0082] In one or more embodiments, the deployment device 206 uses a scaled version of the regularization hyperparameter associated with the global version of each machine learning model to train the final model 214. In some embodiments, the regularization hyperparameter is determined by the λ in the objective function of the generalized linear mixed model described above. b During exploration of the regularization hyperparameters, the global version is trained using a sampled training dataset 224 containing records 228 that are a fraction of the total number of records in the training dataset 216. Thus, the values ​​of the regularization hyperparameters selected by the exploration device 212 for the final model 214 reflect the smaller number of records in the sampled training dataset 224.

[0083] To ensure that the regularization hyperparameters achieve similar performance in the final model 214, the exploration device 212, the deployment device 206, and / or another component scales the regularization hyperparameters by the inverse of the ratio of the training dataset 216 represented by the sampled training dataset 224. For example, if the training dataset 216 has N records and the sampled training dataset 224 has M records, where N>>M, then the component calculates the new value of the regularization parameter as N / M*λ b The deployment device 206 then uses the new values ​​to train the final model 214.

[0084] By sampling entity IDs from an evaluation dataset that is smaller than the training dataset and using the sampled entity IDs to generate a sampled training dataset and a sampled evaluation dataset, the disclosed embodiments ensure that a personalized version of a machine learning model is created for the sampled entity IDs from the sampled training dataset and can subsequently be evaluated using records associated with the same entity ID in the evaluation dataset. Furthermore, generating sampled training datasets and sampled evaluation datasets that are much smaller than the corresponding full training datasets and evaluation datasets speeds up the training and evaluation of machine learning models, which reduces the amount of time and / or resources required to train multiple machine learning models using multiple training configurations. The increased training speed further enables more thorough exploration of hyperparameters, feature sets, and / or training configurations of machine learning models, which in turn results in higher performing machine learning models.

[0085] In contrast, conventional techniques use a complete training dataset and / or evaluation dataset to perform training and / or evaluation of a machine learning model. Consequently, training and evaluation of a single machine learning model requires significantly more time and / or computing resources than training and evaluating using a sampled training dataset and a sampled evaluation dataset. The increased overhead and / or latency associated with training using a complete training dataset additionally interferes with the exploration and / or optimization of the machine learning model's hyperparameters, feature set, and / or training configuration, which can result in reduced performance of the machine learning model.

[0086] Conventional techniques also or alternatively sample individual records from the training dataset and / or evaluation dataset to reduce the size of the dataset. However, this record-based sampling removes data associated with individual entities from each dataset, which results in each personalized version of the machine learning model being trained and / or evaluated using incomplete data for the corresponding entity. As a result, the personalized version is not trained to fit all available behaviors, preferences, and / or other signals for the entities in the training dataset, and is not fully evaluated for additional signals for the entities in the evaluation dataset. In turn, this limited training and / or evaluation of the personalized version results in reduced performance of the personalized version. Therefore, the disclosed embodiments improve computer systems, applications, user experiences, tools, and / or techniques related to training machine learning models, evaluating machine learning models, feature engineering, and / or hyperparameter optimization.

[0087] Those skilled in the art will appreciate that this can be accomplished in a variety of ways. Figure 2First, the sampling device 204, the exploration device 212, the deployment device 206, and / or the data repository 134 can be provided by a single physical machine, multiple computer systems, one or more virtual machines, a grid, one or more databases, one or more file systems, and / or a cloud computing system. The sampling device 204, the exploration device 212, and the deployment device 206 can also be implemented together and / or separately by one or more hardware and / or software components and / or layers.

[0088] Secondly, the system can be used with various types of machine learning models and / or tasks. For example, the machine learning models trained by the exploration device 212 include, but are not limited to, regression models, artificial neural networks, support vector machines, decision trees, random forests, gradient boosting trees, naive Bayes classifiers, Bayesian networks, clustering techniques, collaborative filtering techniques, deep learning models, hierarchical models, and / or integrated models. In another example, the exploration device 212 and / or the deployment device 206 include functions for training, evaluating, and deploying machine learning models for performing various types of reasoning, prediction, classification, recommendation, clustering, and / or regression.

[0089] Figure 4 A flow chart illustrating the processing of data according to the disclosed embodiments is shown. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Thus, Figure 4 The specific arrangement of steps shown in should not be construed as limiting the scope of the embodiments.

[0090] First, a training dataset comprising a first set of records associated with a first set of ID values ​​of a first entity ID and an evaluation dataset comprising a second set of records associated with a second set of ID values ​​of the first entity ID are obtained (operation 402). For example, the training dataset and the evaluation dataset contain records associated with an online system, such as Figure 1 The training dataset contains records associated with the member IDs of the members of the online network 118. The training dataset contains a significantly larger number of records than the evaluation dataset and / or a superset of the member IDs in the evaluation dataset.

[0091] A subset of random ID values ​​is selected from the second set of ID values ​​(operation 404), and a sampled evaluation data set is generated that includes a subset of records in the second set of records that are associated with the subset of random ID values ​​(operation 406). For example, the subset of random ID values ​​is generated and / or selected to include a certain proportion (e.g., a fraction, a percentage, etc.) of the ID values ​​in the evaluation data set. Then, the sampled evaluation data set is generated to include the records in the evaluation data set that are associated with the subset of random ID values.

[0092] A third set of ID values ​​for one or more additional entity IDs is identified in the second subset of records (operation 408), and a sampled training dataset is generated that includes a subset of records in the first set of records associated with the random subset of ID values ​​and the third set of ID values ​​(operation 410). For example, the third set of ID values ​​includes position IDs and / or other types of entity IDs associated with records in the sampled evaluation dataset. To ensure that the sampled training dataset includes all entity ID values ​​found in the sampled evaluation dataset, all records in the training dataset that are associated with any of the entity ID values ​​in the sampled evaluation dataset are added to the sampled training dataset.

[0093] Then, the sampled training dataset and the sampled evaluation dataset are output for use in training and evaluating a machine learning model including a global version and a personalized version for the entity represented by the ID value in the sampled training dataset and the sampled evaluation dataset (operation 412). For example, the sampled training dataset and the sampled evaluation dataset are stored in a data storage device and / or sent to a component that performs training and evaluation of the machine learning model. The component then uses the sampled training dataset and the sampled evaluation dataset to explore different training configurations for the machine learning model, as described below for Figure 5 described in further detail.

[0094] Figure 5 A flow chart illustrating a process for performing model exploration according to the disclosed embodiments is shown. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Thus, Figure 5 The specific arrangement of steps shown in should not be construed as limiting the scope of the embodiments.

[0095] First, a sampled training dataset and a sampled evaluation dataset are obtained (operation 502). As described above, the sampled training dataset and the sampled evaluation dataset are generated so that all entity IDs associated with records in the sampled evaluation dataset are also included in the sampled training dataset.

[0096] Next, a global version and a personalized version of the machine learning model are trained using the sampled training dataset and the training configuration (operation 504). For example, the training configuration includes one or more features and / or hyperparameters of the machine learning model. The global version of the machine learning model is trained using records associated with all entities in the training dataset. Each personalized version in a first set of personalized versions of the machine learning model is trained using records associated with a corresponding ID value of a first entity ID (e.g., a member ID), and each personalized version in a second set of personalized versions of the machine learning model is trained using records associated with a corresponding ID value of a second entity ID (e.g., a position ID).

[0097] After all versions of the machine learning model are trained, the performance of the machine learning model is evaluated using the sampled evaluation dataset (operation 506). For example, the machine learning model is applied to the features in the sampled evaluation dataset to generate a set of scores, and one or more performance metrics are calculated based on the set of scores and the annotations of the features in the sampled evaluation dataset.

[0098] Repeat operations 502-506 for the remaining training configurations (operation 508). For example, multiple machine learning models may be trained and evaluated using different sampled training data sets and sampled evaluation data sets. Each machine learning model may be additionally trained using a different set of features and / or hyperparameters specified in the corresponding training configuration.

[0099] The performance of the machine learning models is then compared to identify the highest performing machine learning model (operation 510). For example, the highest performing machine learning model is identified as having the best single performance metric and / or the best value generated by aggregating multiple performance metrics.

[0100] Finally, a final machine learning model is trained using the training configuration for the highest performing machine learning model and the full training dataset (operation 512). For example, the final machine learning model is trained using the hyperparameters and / or features specified in the training configuration for the highest performing machine learning model. The final machine learning model is also trained using the full training dataset, rather than using a much smaller sampled training dataset generated from a subset of records in the full training dataset. During training of the final machine learning model, a regularization hyperparameter associated with the global version of the machine learning model is obtained from the training configuration and scaled by the inverse of the proportion of the training dataset that the sampled training dataset represents. For example, the regularization hyperparameter is scaled by the number of records in the full training dataset divided by the number of records in the sampled training dataset.

[0101] Figure 6A computer system 600 is shown in accordance with the disclosed embodiments. The computer system 600 includes a processor 602, a memory 604, a storage device 606, and / or other components found in electronic computing devices. The processor 602 can support parallel processing and / or multi-threaded operation with other processors in the computer system 600. The computer system 600 can also include input / output (I / O) devices such as a keyboard 608, a mouse 610, and a display 612.

[0102] The computer system 600 may include the functionality of various components for executing the embodiments herein. Specifically, the computer system 600 may include an operating system (not shown) that coordinates the use of hardware and software resources on the computer system 600, and one or more applications that perform specialized tasks for a user. To perform tasks for a user, an application may obtain access to hardware resources on the computer system 600 from the operating system and interact with the user through a hardware and / or software framework provided by the operating system.

[0103] In one or more embodiments, the computer system 600 provides a system for processing data. The system includes a sampling device and an exploration device. The sampling device obtains a training data set including a first set of records associated with a first set of ID values ​​and an evaluation data set including a second set of records associated with a second set of ID values. Next, the sampling device selects a random ID value subset from the second set of ID values. The sampling device then generates a sampled evaluation data set, which includes the first subset of records in the second set of records associated with the random ID value subset. The sampling device also generates a sampled training data set, which includes the second subset of records in the first set of records associated with the random ID value subset. The sampling device outputs the sampled training data set and the sampled evaluation data set for training and evaluating a machine learning model, and the exploration device uses the sampled training data set to train the machine learning model and uses the sampled evaluation data set to evaluate the machine learning model.

[0104] In addition, one or more components of the computer system 600 can be remotely located and connected to other components via a network. Some parts of the embodiments herein (e.g., sampling devices, exploration devices, deployment devices, data repositories, online networks, etc.) can also be located on different nodes of a distributed system implementing the embodiments. For example, the embodiments herein can be implemented using a cloud computing system that samples training and evaluation data sets for use with a remote machine learning model set.

[0105] The data structures and code described in this detailed description are typically stored on a computer-readable storage medium, which can be any device or medium that can store code and / or data for use by a computer system. Computer-readable storage media include, but are not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact disks), DVDs (digital versatile disks or digital video disks), or other media now known or later developed that can store code and / or data.

[0106] The methods and processes described in the detailed description section may be embodied as code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and code and stored in the computer-readable storage medium.

[0107] In addition, the methods and processes described herein may be included in hardware modules or devices. These modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors (including dedicated or shared processor cores), and / or other programmable logic devices now known or developed in the future that execute a specific software module or piece of code at a specific time. When the hardware modules or devices are activated, they execute the methods and processes included therein.

[0108] The foregoing descriptions of various embodiments have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the invention to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. Furthermore, the above disclosure is not intended to limit the present disclosure.

Claims

1. A method comprising: Obtaining a training dataset comprising a first set of records associated with a first set of ID values ​​of a first entity identifier (ID) and an evaluation dataset comprising a second set of records associated with a second set of ID values ​​of the first entity ID, wherein the first entity ID represents a member searching for or applying for a job via an online network, and the training dataset and the evaluation dataset include information related to interactions and / or compatibility between the member and the job; selecting a subset of random ID values ​​of the first entity ID from the second set of ID values; generating a sampled evaluation data set from the second set of records, the sampled evaluation data set comprising a first subset of records associated with the randomly selected subset of ID values ​​from the second set of records; generating a sampled training data set from the first set of records, the sampled training data set comprising a second subset of records associated with the randomly selected subset of ID values ​​from the second set of records; training a global version and a first set of personalized versions of a first machine learning model using the sampled training dataset and a first training configuration; evaluating a first performance of the first machine learning model using the sampled evaluation dataset; comparing the first performance of the first machine learning model to a second performance of a second machine learning model trained using a second training configuration to identify a highest performing machine learning model within the first machine learning model and the second machine learning model; and training a third machine learning model using the training configuration for the highest-performing machine learning model and the training dataset, wherein training the third machine learning model comprises: i) obtaining a regularization hyperparameter from the global version of the training configuration; and ii) scaling the regularization hyperparameter by the inverse of the proportion of the training dataset that is represented by the sampled training dataset, Among them, the first machine learning model, the second machine learning model and the third machine learning model are used to generate the probability that the member will respond positively to the recommended position.

2. The method according to claim 1, further comprising: a third set of ID values ​​identifying a second entity ID in the first subset of records; as well as Additional records in the first set of records that are associated with the third set of ID values ​​are added to the sampled training data set.

3. The method according to claim 2, further comprising: Using the sampled training dataset and the first training configuration, a second set of personalized versions of the first machine learning model is trained for a second set of entities represented by the third set of ID values.

4. The method according to claim 2, wherein: The second entity ID represents a position.

5. The method according to claim 1, wherein The first training configuration includes one or more features.

6. The method according to claim 1, wherein The first training configuration includes one or more hyperparameters.

7. The method according to claim 1, wherein Selecting the subset of random ID values ​​from the second set of ID values ​​includes: The subset of random ID values ​​is generated to include a proportion of the second set of ID values.

8. A system comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to: Obtaining a training dataset comprising a first set of records associated with a first set of ID values ​​of a first entity identifier (ID) and an evaluation dataset comprising a second set of records associated with a second set of ID values ​​of the first entity ID, wherein the first entity ID represents a member searching for or applying for a job via an online network, and the training dataset and the evaluation dataset include information related to interactions and / or compatibility between the member and the job; selecting a subset of random ID values ​​of the first entity ID from the second set of ID values; generating a sampled evaluation data set from the second set of records, the sampled evaluation data set comprising a first subset of records associated with the randomly selected subset of ID values ​​from the second set of records; generating a sampled training data set from the first set of records, the sampled training data set comprising a second subset of records associated with the randomly selected subset of ID values ​​from the second set of records; training a global version and a first set of personalized versions of a first machine learning model using the sampled training dataset and a first training configuration; evaluating a first performance of the first machine learning model using the sampled evaluation dataset; comparing the first performance of the first machine learning model to a second performance of a second machine learning model trained using a second training configuration to identify a highest performing machine learning model within the first machine learning model and the second machine learning model; and training a third machine learning model using the training configuration for the highest-performing machine learning model and the training dataset, wherein training the third machine learning model comprises: i) obtaining a regularization hyperparameter from the global version of the training configuration; and ii) scaling the regularization hyperparameter by the inverse of the proportion of the training dataset that is represented by the sampled training dataset, Among them, the first machine learning model, the second machine learning model and the third machine learning model are used to generate the probability that the member will respond positively to the recommended position.

9. The system according to claim 8, wherein: The memory also stores instructions that, when executed by the one or more processors, cause the system to: a third set of ID values ​​identifying a second entity ID in the first subset of records; and Additional records in the first set of records that are associated with the third set of ID values ​​are added to the sampled training data set.

10. The system according to claim 9, wherein: The second entity ID represents a position.

11. The system according to claim 8, wherein The first training configuration includes one or more features.

12. A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method comprising: Obtaining a training dataset comprising a first set of records associated with a first set of entity identifier (ID) values ​​and an evaluation dataset comprising a second set of records associated with a second set of entity ID values, wherein the first entity IDs represent members searching for or applying for positions via an online network, and the training dataset and the evaluation dataset comprise information related to interactions and / or compatibilities between members and positions; selecting a subset of random entity ID values ​​from the second set of entity ID values; generating a sampled evaluation data set from the second record set, the sampled evaluation data set comprising a first subset of records associated with the subset of entity ID values ​​randomly selected from the second record set; generating a sampled training data set from the first set of records, the sampled training data set comprising a second subset of records associated with the randomly selected subset of ID values ​​from the second set of records; training a global version and a first set of personalized versions of a first machine learning model using the sampled training dataset and a first training configuration; evaluating a first performance of the first machine learning model using the sampled evaluation dataset; comparing the first performance of the first machine learning model to a second performance of a second machine learning model trained using a second training configuration to identify a highest performing machine learning model within the first machine learning model and the second machine learning model; and training a third machine learning model using the training configuration for the highest-performing machine learning model and the training dataset, wherein training the third machine learning model comprises: i) obtaining a regularization hyperparameter from the global version of the training configuration; and ii) scaling the regularization hyperparameter by the inverse of the proportion of the training dataset that is represented by the sampled training dataset, Among them, the first machine learning model, the second machine learning model and the third machine learning model are used to generate the probability that the member will respond positively to the recommended position.

13. The computer-readable storage medium of claim 12, the method further comprising: a third set of ID values ​​identifying a second entity ID in the first subset of records; adding additional records in the first record set associated with the third set of ID values ​​to the sampled training data set; as well as Using the sampled training dataset and the first training configuration, a second set of personalized versions of the first machine learning model is trained for a second set of entities represented by the third set of ID values.

Images