Simplifying feature removal framework for machine learning

Abstract

The disclosed embodiments provide a system for simplifying machine learning. During operation, the system determines the resource overhead for a baseline version of a machine learning model that uses a set of features to generate entity rankings; and determines the number of features to be removed to reduce the resource overhead to a target resource overhead. Next, the system calculates importance scores for the features, where each importance score represents the impact of the corresponding feature on the entity ranking. The system then identifies a first subset of features to be removed as a plurality of features with the lowest importance scores; and trains a simplified version of the machine learning model using a second subset of features excluding the first subset of features. Finally, the system executes the simplified version to generate a new entity ranking.

Description

Technical Field

[0001] The disclosed embodiments relate to machine learning. More particularly, the disclosed embodiments relate to a feature removal framework for simplifying machine learning. Background Art

[0002] Analytics is typically used to discover trends, patterns, relationships, and / or other properties associated with large collections of complex, interconnected, and / or multidimensional data. The discovered information is then used to derive insights and / or guide decisions or actions related to the data.

[0003] To gather these insights, large datasets of features are analyzed using regression models, artificial neural networks, support vector machines, decision trees, naive Bayes classifiers, and / or other types of machine learning models. The discovered information can then be used to guide decisions and / or perform actions related to the data. For example, the output of a machine learning model can be used to guide marketing decisions, assess risk, detect fraud, predict behavior, and / or customize or optimize the use of an application or website.

[0004] However, during the training and execution of machine learning models for analysis, a significant amount of time, effort, and overhead is expended on selecting and processing features. For example, a dataset for a machine learning model may have tens of thousands of features, including features created from combinations of other features, while only a subset of the features and / or combinations may contribute significantly to the output of the machine learning model. Furthermore, training and / or executing a machine learning model with a large number of features and / or a large dataset typically requires more memory, computing resources, and time than a machine learning model with a smaller number of features or datasets.

[0005] The use of complex machine learning models with large data sets and / or multiple features presents additional challenges in online environments where machine learning outputs are generated in real time or near real time (e.g., in response to recent user actions or newly received data). For example, an online system uses a machine learning model to generate scores that are used to select and display products, services, goods, and / or other entities to a user as the user interacts with the online system. When a request for these scores is received, a service in the online system retrieves the features as input to one or more machine learning models, applies the machine learning model to the features to generate the scores, and returns the scores in response to the request. The service is also required to respond to the requests within the latency limits specified in the service level agreement (SLA) for the service. As a result, as the number of features used by the model, the number of requests, and / or the number of scores generated in response to each request increases, the service may have difficulty scaling and / or meeting latency requirements.

[0006] Thus, machine learning and / or analytics may be facilitated through mechanisms for improving resource consumption, latency, and / or scalability associated with executing machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of a system according to the disclosed embodiments is shown.

[0008] Figure 2 A feature removal framework according to the disclosed embodiments is shown.

[0009] Figure 3 A flowchart illustrating a process for simplifying the execution of a machine learning model in accordance with the disclosed embodiments is shown.

[0010] Figure 4 A flow chart illustrating a process of calculating importance scores for features used by a machine learning model in accordance with the disclosed embodiments is shown.

[0011] Figure 5 A computer system according to the disclosed embodiments is shown.

[0012] In the drawings, like reference numerals refer to the same drawing elements. DETAILED DESCRIPTION

[0013] The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown, but should be accorded the widest scope consistent with the principles and features disclosed herein.

[0014] Overview

[0015] The disclosed embodiments provide a method, apparatus, and system for reducing resource overhead in training and / or executing machine learning models. These machine learning models include, but are not limited to, tree-based models, deep learning models, stacked / ensemble models, regression models, and / or other types of machine learning models for generating entity rankings. For example, one or more of these machine learning models are used to generate a score for each user accessing an online system, and these scores can be used to rank content items (e.g., articles, posts, products, services, users, jobs, etc.). The ranked content items are then output to the user in a content feed, search results, recommendations, and / or another format supported by the online system.

[0016] More specifically, the disclosed embodiments provide a feature removal framework that selectively removes features from a machine learning model, thereby reducing latency, processor usage, memory usage, garbage collection, and / or other types of resource overhead during the retrieval and computation of features input to the machine learning model and subsequent execution of the machine learning model using the features.

[0017] In some embodiments, the feature removal framework performs feature removal on a "baseline" version of a machine learning model. For example, the baseline version comprises a version of the machine learning model that is currently deployed in production and / or a real-world environment. In such an environment, the baseline version can be executed in real time or near real time to perform inference, prediction, and / or other types of analysis based on the latest values of features. As features are added to the baseline version (e.g., as changes are made to the baseline version over time), the resource overhead involved in executing the baseline version increases.

[0018] To improve the resource consumption of a machine learning model, the feature removal framework calculates an importance score for each feature (or set of features) input to the baseline version and removes one or more features whose importance score is below a threshold from the baseline version. For example, each importance score represents the combined impact of a feature (or set of features) on the ranking output by the baseline version of the machine learning model over a period of time (e.g., multiple days over a period of weeks). To calculate the importance scores, the "raw" rankings generated by the baseline version from multiple sets of feature values (e.g., records from the training, test, or validation dataset of the machine learning model) are obtained. In each set of features, the feature value (or multiple feature values) of each feature (or set of features) is modified by replacing the feature value with a "default" missing value and / or a feature value from another set of feature values (e.g., another record in the dataset) that minimizes or removes the feature's contribution to the output of the machine learning model, and the baseline version is applied to each set of modified feature values to produce a "modified" ranking. A ranking deviation overlap (or another measure of similarity based on the rankings) is calculated between each original ranking produced by the baseline version from the set of feature values and the corresponding modified ranking output by the baseline version after one or more feature values in the set have been modified. The measured values for the set of features are then aggregated into an importance score for the modified feature values.

[0019] Next, the feature removal framework identifies a subset of features with the lowest importance scores to remove from the machine learning model and trains a "reduced" version of the machine learning model based on the remaining features excluding the identified subset. For example, the feature removal framework sets a numeric and / or quantile threshold on the importance scores of the features input into the baseline version. The feature removal framework then trains the reduced version using features from the baseline version that meet or exceed the threshold. In other words, the reduced version lacks features from the baseline version that have a low threshold impact on the ranking output by the baseline version.

[0020] In some embodiments, a threshold for removing features from a baseline version is selected to reduce the resource overhead of the baseline version to a target resource overhead for the machine learning model. For example, resource overhead latency, processor consumption, memory usage, GC duration, and / or another measure of resource overhead is estimated for a single feature used by the machine learning model by training and executing different versions of the machine learning model using different numbers of features, determining the resource overhead of each version, and calculating a "typical" (e.g., average, median, etc.) per-feature resource overhead by dividing the difference in resource overhead between the two versions of the machine learning model by the difference in the number of features between the versions. The per-feature resource consumption is then used to estimate the number of features to be removed from the baseline version to reduce the resource consumption to the target resource consumption of the machine learning model, and a simplified version is created to exclude the estimated number of features with the lowest importance scores.

[0021] The feature removal framework also includes functionality for managing the retraining and / or execution of the simplified version based on a comparison of the performance of the baseline and the simplified version. For example, the feature removal framework uses online and / or offline experiments to calculate and compare one or more performance metrics between the baseline and the simplified versions. If the experiments determine that the simplified version has a performance that is statistically significantly lower than the performance of the baseline version, the simplified version is retrained using fewer removed features. Conversely, if the simplified version does not have a performance that is statistically significantly lower than the baseline version, the execution of the baseline version in the production environment will be increasingly replaced by the execution of the simplified version (e.g., by increasing production traffic to the simplified version and decreasing production traffic to the baseline version).

[0022] By replacing a baseline version of a machine learning model with a simplified version that uses a smaller number of features, the disclosed embodiments reduce the resource consumption associated with retrieving, computing, and / or otherwise processing features that are input into the machine learning model and subsequently executing the machine learning model using the values of the features. Consequently, the computer system, cluster, and / or other collection of resources on which the machine learning model is executed is able to process requests for scores or rankings from the machine learning model more quickly and / or process a greater number of such requests. At the same time, selectively executing the simplified version, based on a comparison of performance metrics between the baseline version and the simplified version, can ensure that the simplified version does not negatively impact the results and / or objectives associated with using the machine learning model.

[0023] In contrast, conventional techniques typically add features to a machine learning model over time without monitoring the impact of individual features on the output and / or performance of the machine learning model. This retention of features increases the complexity and / or resource consumption of the machine learning model and negatively impacts the scalability and / or latency associated with processing requests for scores or rankings from the machine learning model. Accordingly, the disclosed embodiments improve computer systems, workflows, tools, and / or techniques related to monitoring, training, executing, and updating machine learning models by removing features from the machine learning model in a manner that meets resource consumption goals and / or minimizes the performance impact on the machine learning model.

[0024] A feature removal framework for simplifying machine learning

[0025] Figure 1 Schematic diagram of a system according to the disclosed embodiment is shown. Figure 1 As shown, the system includes an online network 118 and / or other user communities. For example, the online network 118 includes an online professional network that is used by a collection of entities (eg, entity 1 104, entity x 106) to interact with each other in a professional and / or business environment.

[0026] Entities include users who use online network 118 to establish and maintain professional connections, list jobs and community experiences, endorse and / or recommend each other, search and apply for jobs, and / or perform other actions. Entities also or alternatively include companies, employers, and / or recruiters who use online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and / or perform other actions.

[0027] The online network 118 includes a profile engine 126 that allows an entity to create and edit a profile containing information related to the entity's professional and / or industry background, experience, summary, job titles, projects, skills, etc. The profile engine 126 also allows an entity to view the profiles of other entities in the online network 118.

[0028] The profile engine 126 may also or alternatively include mechanisms for assisting entities in completing their profiles. For example, the profile engine 126 may suggest industries, skills, companies, schools, publications, patents, certifications, and / or other types of attributes to an entity as potential additions to its profile. Suggestions may be based on predictions of missing fields, such as predicting an entity's industry based on other information in the entity's profile. Suggestions may also be used to correct existing fields, such as correcting the spelling of a company name in a profile. Suggestions may also be used to clarify existing attributes, such as changing an entity's title of "Manager" to "Engineering Manager" based on the entity's work experience.

[0029] The online network 118 also includes a search engine 128 that allows an entity to search the online network 118 for people, companies, jobs, and / or other work- or business-related information. For example, an entity may enter one or more keywords into a search field to find profiles, job postings, job candidates, articles, and / or other information that includes and / or otherwise matches the keywords. An entity may also use an "advanced search" feature in the online network 118 to search for profiles, jobs, and / or information by category (e.g., first name, last name, title, company, school, location, interests, relationships, skills, industry, group, salary, experience level, etc.).

[0030] The online network 118 also includes an interaction engine 130 that allows entities to interact with each other on the online network 118. For example, the interaction engine 130 may allow a user, company, school, or other entity to add other entities as connections, follow other entities, send and receive emails or messages with other entities, join groups, and / or interact with (e.g., create, share, reshare, like, and / or comment on) posts from other entities.

[0031] Those skilled in the art will appreciate that the online network 118 may include other components and / or features. For example, the online network 118 may include a homepage, a login page, and / or a content feed that provides the entity with the latest posts, articles, and / or updates from the entity's connections and / or groups. Similarly, the online network 118 may include features or mechanisms for recommending connections, job postings, articles, and / or groups to the entity.

[0032] In one or more embodiments, data related to an entity's profile and activity on online network 118 (e.g., data 1 122, data x 124) is aggregated into data repository 134 for subsequent retrieval and use. For example, each profile update, profile view, connection, follow, post, comment, like, share, search, click, message sent, group interaction, address book interaction, response to a recommendation, purchase, and / or other action performed by an entity in online network 118 is recorded and stored in a database, data warehouse, cloud storage, and / or other data storage mechanism that provides data repository 134.

[0033] The data in the data repository 134 may then be used to generate recommendations and / or other insights relevant to users, companies, schools, jobs, groups, and / or other entities in the online network 118. In some embodiments, one or more components of the online network 118 record searches, clicks, views, text inputs, conversions, and / or other feedback during an entity's interaction with a job search tool in the online network 118. This feedback may be stored in the data repository 134 and used as training data for one or more machine learning models 110, and the output of the machine learning models may be used to display and / or otherwise recommend jobs, ads, posts, articles, connections, products, companies, groups, and / or other types of content, entities, or actions to members of the online network 118.

[0034] In some embodiments, the data in the data repository 134 and the one or more machine learning models 110 are used to generate rankings of candidates for jobs or opportunities listed within or outside of the online network 118. Candidates include users who have viewed, searched for, or applied for jobs, positions, roles, and / or opportunities within or outside of the online network 118. Candidates also or alternatively include users and / or members of the online network 118 who have skills, work experience, and / or other attributes or qualifications that match corresponding jobs, positions, roles, and / or opportunities.

[0035] After identifying a candidate, the candidate's profile and / or activity data, along with characteristics and / or features of the corresponding opportunity (e.g., required or desired skills, education, experience, industry, title, etc.), are input into a machine learning model. The machine learning model outputs a score representing the candidate's strength relative to the opportunity and / or qualifications associated with the opportunity (e.g., skills, current position, previous position, overall qualifications, etc.). That is, the machine learning model generates a score based on the similarity between the candidate's profile data and the description of the opportunity using the online network 118. The model optionally adjusts the score based on social and / or other validation of the candidate's profile data (e.g., recognition of skills, recommendations, achievements, awards, etc.). The machine learning model also generates or alternatively generates a score representing the candidate's level of interest in the opportunity.

[0036] The score-based rankings and / or associated insights are then used to improve the quality of candidates and / or opportunities recommended to candidates, increase user activity utilizing the online network 118, and / or guide the decisions of candidates and / or referees (e.g., hiring managers, recruiters, human resources professionals, etc.) involved in screening or placing opportunities. For example, one or more components of the online network 118 may display and / or otherwise output a member's position in a ranking of candidates for jobs (e.g., top 10%, top 20 out of 138, etc.) to encourage the member to apply for jobs for which the member ranked highly. In a second example, the component is responsible for incorporating the candidate's relative interest and / or strength relative to a set of jobs during a job search into search results and / or job suggestions subsequently displayed to the candidate. In a third example, the component recommends candidates who ranked highly for a job to recruiters and / or other referees as potential applicants and / or interview candidates for the job. In a fourth example, the component recommends jobs to candidates based on the job's predicted relevance or attractiveness to the candidate and / or the candidate's likelihood of applying for the job.

[0037] In one or more embodiments, the online network 118 includes functionality for executing the machine learning model 110 and generating scores and / or rankings associated with candidates, opportunities, and / or other entities in real time, near real time, or on demand. In these embodiments, the scores and / or rankings are generated in response to recent activities or actions by or involving the entity.

[0038] For example, the online network 118 includes a service that uses a machine learning model to generate a set of relevance scores that represent the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates for jobs). The service receives a score request in response to a user's job search (or a recruiter's search for candidates matching a job), a user accessing a job recommendation component in the online network 118, and / or a user otherwise interacting with job-related functionality in the online network 118. To improve the accuracy of the scores, the service inputs features reflecting recent activity performed by the user (or recruiter) into the machine learning model, such as features representing the user's searches, clicks, likes, dislikes, and / or other actions performed in the same user session as the user's interaction with the job-related functionality. In response to the input features, the machine learning model calculates a relevance score between the user and the job (or the set of candidates and the job) in real time or near real time (e.g., with latency within limits specified in a service level agreement (SLA) for the service). The service returns relevance scores in response to the rating request, and the online network 118 outputs a ranking of jobs (or candidates) to the user by descending order of relevance scores as search results, recommendations, and / or other representations of jobs (or candidates).

[0039] Those skilled in the art will appreciate that the online network 118 can perform on-demand scoring and / or ranking related to other types or combinations of entities. For example, the online network 118 can receive one or more scoring requests for relevance scores between a user and content items created and / or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and / or videos. The scoring request can be generated in response to a user accessing a homepage, a landing page, and / or a content feed in another portion of the online network 118. In response to the scoring request, the first set of machine learning models 110 is executed to generate recommendations for specific types of content (e.g., web updates, jobs, articles, courses, advertisements, posts, products, connections, products, etc.) for the user based on attributes of the user, recent activity of the user and / or similar users, and / or attributes of the content. The first set of machine learning models 110 outputs a score representing a predicted level of interest of the user in the content, and a subset of content items from the first set of machine learning models 110 with high scores is selected. Features associated with the user and the selected content item are input into one or more additional machine learning models 110, and the additional machine learning models generate an additional set of scores that represent the predicted likelihood of the user performing certain actions (e.g., clicking, liking, commenting, sharing, etc.) on the selected content item. The content items are sorted in descending order of score, and the rankings are updated using one or more rules, filters, and / or exceptions (e.g., based on business needs, member preferences, impression discounts, content diversification, and / or other goals, strategies, or priorities). Finally, the updated rankings are displayed in a content feed for the user. Because the content items are scored and ranked in real time or near real time, the content feed can reflect the latest content posted to the online network 118 and / or user interactions with content in the online network 118.

[0040] In another example, in response to a user accessing a “People You May Know” feature in an online network 118, the online network 118 may generate a scoring request for relevance scores between the user and other registered users of the online network 118. The service processes the scoring request by executing one or more machine learning models 110, which generate relevance scores between the user and other registered users based on features including the user's most recent connection set and / or connection requests in the online network 118. The other registered users are then ranked by descending relevance score, and the ranking is output to the user as connection suggestions within the “People You May Know” feature.

[0041] In one or more embodiments, a feature removal framework 108 provided by and / or executed in conjunction with the online network 118 includes functionality to reduce latency and / or other types of resource overhead when executing the machine learning model 110. As described in detail below, the feature removal framework 108 identifies, for a given machine learning model, a subset of features that have less than a threshold impact on a ranking output by the machine learning model. The feature removal framework 108 then trains a "simplified version" of the machine learning model without the identified features, thereby removing the features from the machine learning model in a manner that minimizes the impact on the performance of the machine learning model. The feature removal framework 108 then deploys and executes the simplified version in place of the original machine learning model, thereby reducing latency, processor usage, memory usage, garbage collection duration, and / or other resource overhead associated with retrieving and / or computing features that were input into the machine learning model and executing the machine learning model using those features.

[0042] Figure 2 The feature removal framework (e.g., Figure 1 feature removal framework 108). Figure 2 As shown, the feature removal framework includes a simplification means 202, an analysis means 204, a management means 206, and a training means 210. Each of these components is described in further detail below.

[0043] The analysis device 204 calculates an importance score 230 that represents the impact of each feature or a set of features on the ranking output by the version of the machine learning model. In one or more embodiments, the analysis device 204 calculates the importance score 230 for a baseline version 208 of the machine learning model, which is deployed in a production environment that is exposed to end users. Within the production environment, the baseline version 208 can be executed in real time or near real time to perform reasoning, prediction and / or other types of analysis based on the latest values of the features from the feature repository 234. When features are added to the baseline version 208 (e.g., as features become available and / or during iterative updates to the baseline version 208), the baseline resource overhead 216 associated with executing the baseline version 208 to process requests for scores generated by the baseline version increases.

[0044] In some embodiments, the machine learning model includes a "ranking model" that produces rankings that are output to an online system (e.g., Figure 1For example, a machine learning model generates a score by which recommendations, search results, and / or other types of entities can be ranked for a given user. As a result, an importance score 230 is calculated to reflect the impact of the corresponding feature on the ranking of entities in a given ranking, rather than calculating a numerical score by which the entities in the ranking can be ranked.

[0045] More specifically, the analysis device 204 obtains a set of raw feature values 220 from the feature repository 234 based on records in a training, testing, validation, and / or another dataset used for training and / or evaluating a machine learning model. The analysis device 204 also obtains a raw ranking 250 generated by the baseline version 208 from the set of raw feature values 220. For example, the analysis device 204 applies the baseline version 208 to the raw feature values 220 to generate the raw ranking 250 and / or retrieves the raw ranking 250 generated during a previous training and / or evaluation of a version from the feature repository 234 and / or another data store.

[0046] Next, the analysis device 204 generates modified feature values 222-224 based on the original feature values 220. In one or more embodiments, each set of original feature values 220 from the data set is used to generate multiple sets of modified feature values 222-224, wherein each set of modified feature values includes one or more original feature values 220 that have been replaced by other values. For example, each set of modified feature values 222-224 includes a feature value that has been replaced by a default "missing" value. The "missing" value may include, but is not limited to: a zero value, the mean or median of the corresponding feature, negative infinity, a "missing" or "other" categorical value defined separately from the categorical value in the categorical feature (e.g., a finite set of values for skills, industry, position, seniority, etc.), a value randomly selected from a distribution of values for the feature, and / or another value that effectively minimizes or removes the contribution of the feature from the output of the version of the machine learning model. The missing value can be set by a user responsible for developing or maintaining the machine learning model and / or an individual feature used by the machine learning model. Missing values are also or alternatively automatically selected by the analysis device 204 based on the type of feature (e.g., numerical, categorical, binary, etc.), the type of machine learning model, and / or the distribution of feature values in the feature.

[0047] In another example, each set of modified feature values 222-224 includes feature values that have been randomly swapped with different feature values for the same feature from a different set of original feature values 220. This swapping of original feature values 220 across records in the dataset allows sets of modified feature values 222-224 to be generated without introducing "artificial" or anomalous feature values into the sets of modified feature values 222. In a third example, one or more sets of modified feature values 222-224 include a plurality of features selected by a user and / or the analysis device 204 (e.g., groups of features that are related to each other, groups of features that are semantically related, groups of features that are randomly selected, etc.), for which the original feature values 220 have been replaced with default values or other values.

[0048] The analysis device 204 applies the baseline version 208 to each set of modified feature values 222-224 to generate a corresponding set of modified rankings (e.g., modified rankings 252-254). For example, the analysis device 204 inputs each set of modified feature values 222 into the baseline version 208 and obtains the modified ranking corresponding to the set of modified feature values 222 as the output of the version.

[0049] The analysis device 204 then calculates a measure of ranking similarity between each modified ranking generated from the set of modified feature values (e.g., modified feature values 222-224) and the original ranking generated from the corresponding set of original feature values 220. In one or more embodiments, the measure of ranking similarity includes a ranking deviation overlap (e.g., ranking deviation overlap 228) calculated using the following formula:

[0050] I d =S 1:d ∩T 1:d

[0051] X d =|I d |

[0052] A d =X d / d

[0053]

[0054] In the above formula, S and T are two rankings, for example, an original ranking produced by a version of the machine learning model based on the original set of feature values 220 and a modified ranking produced by the same version of the machine learning model based on the corresponding set of modified feature values. d represents the intersection of two rankings up to depth d, X drepresents the size of the intersection up to depth d, and A d represents the "agreement" of S and T at depth d, which is calculated as the proportion of the overlap of S and T at depth d. Then, by scaling the agreement at each depth by the weight p d-1 (where 0 < p < 1), summing the weighted agreements up to the maximum depth n, and multiplying the result by (1 - p), the ranked-biased overlap RBO(S, T, p) is calculated. The value of the ranked-biased overlap falls within the range [0, 1], where 0 indicates that S and T are disjoint, and 1 indicates that S and T are identical.

[0055] The parameter p determines the steepness of the weight decay as the depth increases: a smaller p results in a greater top-weighting degree of the ranked-biased overlap. When p = 0, only the top-ranked items in the ranking are considered, and the ranked-biased overlap is 0 or 1. Conversely, when p approaches 1, the weights become arbitrarily flat, and the evaluation of the ranked-biased overlap becomes arbitrarily deep.

[0056] After calculating the ranked-biased overlap between the original ranking set and the modified ranking set corresponding to a given modified feature (or set of features), the analysis device 204 aggregates the calculated ranked-biased overlap into an importance score for the feature (e.g., importance score 230). For example, the analysis device 204 calculates the "similarity score" of a feature as the average ranked-biased overlap between the modified ranking set generated from the modified feature values 222 for the feature and the corresponding set of the original rankings 250. Then, the analysis device 204 uses the similarity score as the importance score for the feature, where a higher similarity score indicates a smaller impact of the feature on the ranking output by the machine learning model, and a lower similarity score indicates a greater impact of the feature on the ranking. Optionally, the analysis device 204 subtracts the similarity score from 1 to obtain an alternative version of the importance score that is directly proportional to the impact of the feature on the ranking (i.e., a higher importance score indicates a greater impact of the feature on the ranking, and a lower importance score indicates a lower impact on the ranking).

[0057] The simplifying device 202 uses the importance scores 230 and the feature removal threshold 232 to identify a set of high-importance features 238 used by the baseline version 208. In some embodiments, the high-importance features 238 include a subset of features used by the baseline version that have importance scores 230 that meet or exceed the feature removal threshold 232, which represents the minimum impact on the ranking output by the baseline version 208. For example, the simplifying device 202 identifies the high-importance features 238 as a predetermined number of features having importance scores 230 that indicate the greatest impact on the ranking produced by the baseline version 208 (e.g., 50 features having the highest importance scores 230, 20% of the features having the highest importance scores 238, etc.), and / or a variable number of features having importance scores 238 that exceed a numerical threshold (e.g., all features having importance scores exceeding 0.01).

[0058] In some embodiments, the simplification device 202 selects and / or adjusts the feature removal threshold 232 based on a comparison of a baseline resource cost 216 for the baseline version 208 and a target resource cost 218 for the machine learning model. In some embodiments, the baseline resource cost 216 includes latency, processor usage, memory footprint, garbage collection duration, and / or another measure of resource or runtime cost for the baseline version 208, and the target resource cost 218 includes a target value for the measure for the machine learning model. For example, the target resource cost 218 includes a performance limit for the machine learning model, which is specified in an SLA for the machine learning model and / or a service that utilizes the machine learning model.

[0059] In some embodiments, the simplification device 202 sets the feature removal threshold 232 to the estimated number of features to be removed from the baseline version 208 to reduce the baseline resource overhead 216 to the target resource overhead 218. For example, the simplification device 202 retrieves historical resource overheads of different versions of the machine learning model (e.g., different baseline versions that have been updated over time) from the model repository 236 and / or another data store. The simplification device 202 uses a model or formula to estimate a “typical” (e.g., average, median, etc.) per-feature resource overhead for the machine learning model (e.g., by dividing the difference in resource overhead between two versions of the machine learning model by the difference in the number of features between the versions). The simplification device 202 then calculates the feature removal threshold 232 by dividing the difference between the baseline resource overhead 216 and the target resource overhead 218 by the typical per-feature resource overhead and rounding the result to the next highest integer.

[0060] The training device 210 trains the simplified version 214 of the machine learning model using only the high-importance features 238 identified by the simplification device 202. For example, the training device 210 removes inputs, layers, processing nodes, and / or other components from the baseline version 208 that are not related to the high-importance features 238. The training device 210 then trains the remainder of the baseline version 208 using the values of the high-importance features 238 (and derived features created from the high-importance features 238) and corresponding labels from the feature store 234.

[0061] In some embodiments, the simplification device 202, the analysis device 204, and / or another component of the system calculates importance scores 230 and / or identifies high-importance features 238 by analyzing the main features in the feature repository 234. These main features are extracted directly from fields or records in the profile data, user activity data, sensor data, and / or other data available as input to the machine learning model. After identifying the high-importance features 238 from the corresponding importance scores 230, the component identifies a subset of the main features that are excluded from the high-importance features 238 and uses the correlation graph and / or feature transformation metadata of the main features to identify a set of derived features that are calculated based on the excluded main features (e.g., by aggregating, scaling, combining, bucketing, or otherwise transforming the excluded features). The component then adds the derived features to the excluded features, thereby removing all features calculated based on the excluded features from the workflow or pipeline for training and executing the simplified version 214.

[0062] In one or more embodiments, the training device 210 further reduces the overhead and / or complexity associated with the machine learning model by generating a compressed baseline version 212 of the machine learning model based on the baseline version 208 and creating a simplified version 214 based on the compressed baseline version 212. For example, the training device 210 creates the compressed baseline version 212 by applying weighted quantization, knowledge distillation, weight pruning, tensor decomposition, and / or another compression technique to one or more deep learning layers in the baseline version 212. In another example, the training device 210 creates the compressed baseline version 212 by applying a significance test to nodes in one or more tree-based portions of the baseline version 212 and pruning a subset of the nodes that are determined to lack statistical significance in prediction. The training device 210 then creates the simplified version 214 by retraining the compressed baseline version 212 using only high-importance features 238 (e.g., from a training dataset for the machine learning model). Alternatively, training device 210 first trains simplified version 214 based on uncompressed baseline version 208 and applies compression to simplified version 214 after simplified version 214 is created.

[0063] After creating the compressed baseline version 212, the simplified version 214, and / or another version of the machine learning model, the training device 210 stores the parameters of each version in the model repository 236. For example, the training device 210 replaces the old values of the parameters in the model repository 236 with the updated parameters, or the training device 210 stores the updated parameters separately from the old values (e.g., by storing each parameter set with a different version number for the corresponding version). The analysis device 204, the simplification device 202, the management device 206, and / or another component of the system can then retrieve each version of the machine learning model from the model repository 236 and / or the training device 210.

[0064] After creating the simplified version 214, the management device 206 verifies that the resource cost of the simplified version 214 meets the target resource cost 218. For example, the management device 206 executes the simplified version 214 in the same production environment as the baseline version 208 and / or in a replica of the production environment, and measures the resource cost of the simplified version 214 in the environment (e.g., the latency, processor consumption, memory usage, etc. of the simplified version 214 during processing one or more scoring requests). If the resource cost of the simplified version 214 exceeds the target resource cost 218, the management device 206 retrains the simplified version 214 with additional features removed from the high-importance features 238 until the resource cost of the simplified version 214 reaches or falls below the target resource cost.

[0065] Once it is determined that the resource cost of simplified version 214 is at or below target resource cost 218, management device 206 replaces at least a portion of the execution of baseline version 208 in the production environment with the execution of simplified version 214 to generate a new ranking with lower resource cost. For example, management device 206 gradually guides production traffic from baseline version 208 to simplified version 214 in a series of "increments" of simplified version 214.

[0066] During (or before) the addition of simplified version 214, management device 206 performs one or more comparisons 226 of baseline performance 240 of baseline version 208 and simplified performance 242 of simplified version 214, and uses the results of the comparisons to adjust baseline execution 260 of baseline version 208 and / or simplified execution 262 of simplified version 214 in the production environment to generate new rankings, which may include baseline ranking 256 from baseline version 208 and / or simplified ranking 258 from simplified version 214. In some embodiments, management device 206 uses one or more online and / or offline experiments to collect and compare performance metrics indicative of baseline performance 240 and simplified performance 242.

[0067] During an online experiment, both baseline version 208 and simplified version 214 were deployed in a production environment, a first set of users in the online system were exposed to a baseline ranking 256 generated by baseline version 208, and a second set of users in the online system were exposed to a simplified ranking 258 generated by simplified version 214. Baseline results 244 were collected between the first set of users and the items in the corresponding baseline ranking 256, and simplified results 246 were collected between the second set of users and the items in the corresponding simplified ranking 258. Baseline performance 240 was determined by calculating a performance metric for baseline version 208 as the ratio of positive results for users exposed to baseline ranking 256, while simplified performance 242 was determined by calculating a performance metric for simplified version 214 as the ratio of positive results for users exposed to simplified ranking 258. For example, the performance metric was calculated for each version as the unique number of users who contributed likes, shares, comments, or other types of interactions with the ranking of content items in the content feed after the ranking was generated for the version.

[0068] Then, based on a comparison of the performance metrics between baseline version 208 and simplified version 214 (e.g., comparison 226), an A / B test is used to determine whether simplified version 214 has statistically significantly lower performance than baseline version 208 (e.g., based on a predetermined significance level of 0.05, 0.01, and / or another value selected by the user conducting the experiment or managing the machine learning model). If it is determined that the performance of simplified version 214 is not statistically significantly lower than that of baseline version 208, the proportion of users exposed to the ranking generated by simplified version 214 is increased, and baseline execution 260 of baseline version 208 is gradually replaced with simplified execution 262 of simplified version 214 in the production environment to reduce the resource overhead of the machine learning model when generating new rankings. If it is determined that simplified version 214 has statistically significantly lower performance than baseline version 208, the exposure of users in the online system to the ranking generated by simplified version 214 is reduced and / or terminated, and the baseline execution 260 of baseline version 208 is maintained or increased to generate a new baseline ranking 260 that is output to users of the online system.

[0069] During the offline experiment, baseline version 208 is executed in the production environment, a first set of users in the online system are exposed to a baseline ranking 256 generated by baseline version 208, and a second set of users are exposed to a randomly generated ranking. A performance metric for baseline version 208 is calculated as a ratio 244 of positive baseline results between the first set of users and the corresponding baseline ranking 256 from baseline version 208, while a performance metric for simplified version 214 is calculated as a ratio of positive results between the second set of users and the randomly generated ranking that matches the simplified ranking 258 generated by simplified version 214. As with the online experiment, an A / B test is then used to perform a comparison 226, which determines the statistical significance of the difference between baseline performance 240 and simplified performance 242. If simplified version 214 is determined to have statistically significantly lower performance than baseline version 208, baseline execution 260 of baseline version 208 is continued in the production environment to generate a new ranking 244, which is output to the users of the online system. If it is determined that the simplified version 214 does not have statistically lower performance than the baseline version 208 , then ramping up of the simplified execution 262 of the simplified version 214 in the production environment may begin.

[0070] Management device 206 also or alternatively provides comparison result 226 to simplification device 202, and simplification device 202 adjusts feature removal threshold 232 based on the result. For example, when simplified version 214 is determined to have statistically significantly lower performance than baseline version 208 in one or more performance metrics, simplification device 202 reduces the number of features to be removed from baseline version 208. The reduction in the number of features to be removed can be performed based on a predetermined "step size" (e.g., feature removal threshold 232 is gradually lowered so that five fewer features are removed from baseline version 208 each time simplified version 214 has significantly lower performance than baseline version 208). The reduction in the number of features to be removed can also or alternatively be determined based on the magnitude of the difference between baseline performance 240 and simplified performance 242 (e.g., feature removal threshold 232 is lowered so that fewer features are removed from baseline version 208 for each "unit" reduction in baseline performance 240 found in simplified performance 242).

[0071] After adjusting the feature removal threshold 232 (e.g., reducing the number of features removed from the baseline version 208), the training device 206 retrains the simplified version 214 with the high-importance features 238 that exclude the reduced number of features with the lowest importance scores 230 from the baseline version 208. The management device 206 then performs one or more additional roundings of comparisons 226 of the baseline performance 240 to the simplified performance 242 of the retrained version 214. Once it is determined that a given simplified version 214 does not have a performance that is statistically significantly lower than that of the baseline version 208, the management device 206 initiates or augments a simplified execution 262 of the simplified version 214 in place of the baseline execution 260 of the baseline version 208 to generate a new ranking in the online system.

[0072] In one or more embodiments, management device 206 uses a plurality of performance metrics to perform comparison 226 of baseline performance 240 to simplified performance 242 and determines a ratio by which to execute baseline version 208 and simplified version 214 to generate new baseline ranking 256 and / or simplified ranking 258. Simplification device 202 also or alternatively uses one or more combinations of performance metrics to select feature removal threshold 232.

[0073] For example, the training device 210 periodically (e.g., weekly, biweekly, monthly, etc.) retrains a new baseline version 208 and / or simplified version 214 of the machine learning model (e.g., using the latest feature values from the feature repository 234 and / or the latest importance scores 230 calculated based on the feature values). Next, the management device 206 calculates a "primary" performance metric for each of the baseline version 208 and the simplified version 214, as the number of unique users who contributed likes, shares, comments, or other types of interactions with the ranking of the content item in the content feed after the ranking was generated by the corresponding version. The management device 206 also calculates a number of "secondary" performance metrics for both the baseline version 208 and the simplified version 214. These secondary performance metrics include, but are not limited to: frequency of use of the online system (e.g., number of sessions, session length, etc.) for users exposed to content reflecting the rankings output by each version (e.g., search results, recommendations, content feeds, etc.), recency of content in the rankings output by each version (e.g., the average number of minutes, hours, or days since content in a given ranking was created), the rate of content creation in the online system (e.g., the average number of new posts by users exposed to the rankings output by each version), and / or usage of features in the online system (e.g., the average number of sessions and / or average session length in the online system having the "company" or "group" feature for users exposed to the rankings output by each version).

[0074] Continuing with the above example, management device 206 uses one or more online and / or offline experiments to perform a comparison 226 of both primary and secondary performance metrics of baseline version 208 and simplified version 214. When the experiment determines that simplified version 214 has a statistically significantly lower primary performance metric than baseline version 208, management device 206 stops using simplified version 214 in generating a new simplified ranking 258 that is output to the user of the online system, and simplification device 202 adjusts feature removal threshold 232 so that more features are included in high-importance features 238. Training device 210 then retrains simplified version 214 to have a greater number of high-importance features 238 and fewer missing features from baseline version 208, and management device 206 performs additional experiments to generate a comparison 226 of simplified performance 242 of the retrained simplified version 214 with baseline performance 240 of baseline version 208. When the experiment determines that the simplified version 214 has a statistically significantly lower secondary performance metric than the baseline version 208, the management device 206 maintains the proportion of business processed by the simplified version 214 and reduces the number of features to be removed from the baseline version 208 in the next scheduled retraining of the simplified version 214.

[0075] in other words, Figure 2 The system manages the creation and / or execution of simplified versions 214 from a reduced set of features in baseline version 208 in a manner that balances the tradeoff between resource overhead and performance in the machine learning model. If the simplified version 214 meets the target resource overhead 218 and has a performance metric that is not significantly worse than the performance metric of the baseline version 208, the system increases the use of the simplified version 214 when generating new rankings in the production environment. If the simplified version 214 does not meet the target resource overhead 218 and does not have significantly lower performance than the baseline version 208, the system retrains the simplified version 214 with fewer high-performance features 238 from the baseline version 208 and / or uses a greater amount of compression. If the simplified version 214 meets the target resource overhead 218 and has significantly worse performance than the baseline version 208, the system retrains the simplified version 214 with a greater number of high-performance features 238 from the baseline version 208 and / or uses a lower amount of compression. Finally, if the simplified version 214 does not meet the target resource overhead 218 and has significantly lower performance than the baseline version 208, the system uses one or more predefined rules and / or heuristics to manage subsequent training and / or execution of the baseline version 208 and / or simplified version 214; adjusts the target resource overhead 218 and / or allows the simplified performance 242 to be within a certain interval below the baseline performance 240; and / or uses human input (e.g., from developers and / or administrators of the machine learning model or the service that executes the machine learning model) to prioritize the resource overhead or performance of the simplified version 214.

[0076] Because simplified version 214 includes fewer features, components, and / or operations than baseline version 208, simplified version 214 involves significantly less complexity and / or resource overhead than baseline version 208. At the same time, selective execution of the two versions based on comparison 226 of performance metrics between baseline version 208 and simplified version 214 ensures that simplified version 214 does not negatively impact results and / or objectives associated with using the machine learning model.

[0077] For example, an experiment was conducted to compare a baseline version 208 of a machine learning model that uses 120 features to rank content items in a content feed with a corresponding simplified version 214 that lacks 10 features with the lowest importance scores 230 from baseline version 208. The experiment determined that simplified version 214 improved queries per second (QPS) by 0.26, reduced the 99th percentile latency of baseline version 208 by 2.44%, reduced the 99th percentile latency of baseline version 208 on mobile devices by 1.07%, reduced the 90th percentile garbage collection duration from 131.3ms to 103.2ms, and reduced processor utilization from 15.72% to 11.15%. The experiment also determined that, despite using fewer features, simplified version 214 had better performance metrics for user-engaged content feeds than baseline version 208. Thus, simplified version 214 improved the runtime performance and prediction performance of the machine learning model when compared to baseline version 208.

[0078] Those skilled in the art will recognize that Figure 2 The system can be implemented in various ways. First, the analysis device 204, training device 210, management device 206, feature repository 234, and / or model repository 236 can be provided by a single physical machine, multiple computer systems, one or more virtual machines, a grid, one or more databases, one or more file systems, and / or a cloud computing system. The analysis device 204, training device 210, and management device 206 can be implemented together and / or separately by one or more hardware and / or software components and / or layers.

[0079] Second, a number of metrics and / or techniques can be used to calculate the importance score 230, resource cost, performance metric, and / or other parameters that influence the creation of the simplified version 214 from the baseline version 208. For example, the importance score 230 can be calculated by aggregating Kendall's rank correlation coefficient, Spearman's rank correlation coefficient, and / or other ranking similarity measures calculated between the original ranking 250 and the modified rankings 252-254. In another example, the baseline resource cost 216, the target resource cost 218, and / or the resource cost of the simplified version 214 can be specified based on latency, QPS, error rate, processor utilization, memory utilization, garbage collection duration, heap utilization, timeout count, and / or other types of system performance metrics. In a third example, the baseline performance 240 and the simplified performance 242 can include, but are not limited to, session count, session length, click-through rate (CTR), conversion rate, message count, search, and / or other measures of user engagement or activity with the online system.

[0080] Figure 3 A flow chart illustrating a process for simplifying the execution of a machine learning model according to the disclosed embodiments is shown. In one or more embodiments, one or more of the steps may be omitted, repeated, and / or performed in a different order. Thus, Figure 3 The specific arrangement of steps shown should not be construed as limiting the scope of the embodiments.

[0081] Initially, a resource overhead for a baseline version of a machine learning model that uses a set of features to generate entity rankings is determined (operation 302). For example, the resource overhead includes latency, processor usage, memory footprint, garbage collection duration, and / or another measure of computational resource consumption for the baseline version.

[0082] Next, a determination is made as to the number of features to be removed from the baseline version to reduce the resource overhead to a target resource overhead (operation 304). For example, the target resource overhead comprises an upper limit on the resource overhead of the machine learning model, as specified in an SLA for the machine learning model. The target resource overhead may also or instead be calculated as a ratio or percentage of the resource overhead of the baseline version. The “per feature” resource overhead of the baseline version is estimated based on how the number of features and resource overhead of the baseline version differ over time (e.g., as features are added to or removed from the baseline version); and the number of features to be removed from the baseline version is calculated as the difference between the resource overhead of the baseline version and the target resource overhead divided by the per feature resource overhead, rounded to the nearest integer.

[0083] A set of importance scores is calculated for the set of features (operation 306). Figure 4In more detail, each importance score represents the influence of the corresponding feature on the entity ranking generated by the baseline version. A first subset of features to be removed is also identified as the number of features with the lowest importance scores (operation 308). For example, the first subset of features includes the number of features with the least influence on the entity ranking generated by the baseline version.

[0084] A simplified version of the machine learning model is then trained using the second subset of features excluding the first subset of features (operation 310). For example, the simplified version is trained using the architecture of the baseline version and the remaining features used by the baseline version excluding the plurality of features having the lowest importance scores identified in operation 308. Because fewer features are computed, retrieved, and / or processed using the simplified version, the resource overhead of the simplified version is lower than that of the baseline version. To further reduce complexity and / or overhead in the simplified version, a model compression technique may optionally be applied to the baseline version before training the simplified version and / or after training the simplified version.

[0085] After the simplified version is created, the resource cost of the simplified version is compared to the target resource cost to determine whether the simplified version meets the target resource cost (operation 312). For example, the simplified version is executed in the same environment as the baseline version, or in a copy of the environment, and the resource cost of the simplified version in the environment is compared to the target resource cost. If the simplified version meets the target resource cost, no additional processing related to the resource cost of the simplified version is required.

[0086] If the simplified version does not meet the target resource cost, the number of features in the first subset is increased (operation 314), and operations 308-310 are repeated to retrain the simplified version using a smaller number of remaining features from the baseline version. Operations 312-314 are also repeated until the resource cost of the simplified version meets the target resource cost of the machine learning model.

[0087] After determining that the simplified version meets the target resource cost, a first performance metric is calculated for the baseline version, and a second performance metric is calculated for the simplified version (operation 316). For example, an offline experiment is used to determine a first set of results from a first set of users in an online system being exposed to a first set of rankings output by the baseline version, and a second set of results from a second set of users in the online system being exposed to randomly generated rankings that match the second set of rankings output by the simplified version. In another example, an online experiment is used to output a first set of rankings generated by the baseline version to the first set of users, and a second set of rankings generated by the simplified version to the second set of users, and the first and second sets of results are monitored while the first and second sets of users use the online system. In both examples, the first performance metric is calculated as the ratio of positive results (e.g., clicks, saves, likes, comments, shares, and / or other actions indicating user engagement) between the first set of users and the first set of rankings, and the second performance metric is calculated as the ratio of positive results between the second set of users and the second set of rankings.

[0088] The first and second performance metrics are compared to determine whether the performance of the simplified version is significantly lower than that of the baseline version (operation 318). For example, an A / B test or another type of statistical significance test is used to determine the statistical significance of the difference between the first and second performance metrics. If the results of the test include a second performance metric that is lower than the first performance metric and a p-value that is lower than or equal to a predetermined significance level, then the simplified version is determined to have statistically significantly lower performance than the baseline version. If the second performance metric is not lower than the first performance metric or the p-value is greater than the significance level, then the simplified version is determined not to have statistically significantly lower performance than the baseline version.

[0089] When the simplified version has statistically significantly lower performance than the baseline version, the number of features in the first subset is reduced (operation 320), and operations 308-310 are performed to retrain the simplified version using a larger number of features from the baseline version. Operations 312-318 are then repeated to verify that the retrained simplified version meets the target resource cost while not having statistically significantly lower performance than the baseline version. When the simplified version has significantly lower performance than the baseline version and fails to meet the target resource cost, one or more rules and / or manual input may be used to select a number of features to be removed from the baseline version, a compression level in the simplified version, and / or other parameters associated with creating the simplified version from the baseline version.

[0090] When the simplified version does not have statistically significantly lower performance than the baseline version, the execution of the baseline version is replaced with the execution of the simplified version to generate a new entity ranking (operation 322). For example, the execution of the simplified version in the production environment to generate the new entity ranking output to the online system user is gradually increased. As the execution of the simplified version gradually increases, the execution of the baseline version in the production environment is correspondingly reduced.

[0091] In one or more embodiments, multiple performance metrics are monitored and compared between a baseline version and a simplified version, and different types of actions are taken based on the results of the comparison. For example, the performance metrics for a given version (e.g., baseline, simplified, etc.) of a machine learning model that generates rankings of content for display in a content feed include a "primary" performance metric that measures the unique number of users who contribute likes, shares, comments, or other types of interactions with the rankings generated by the version. The performance metrics also include multiple "secondary" performance metrics, including but not limited to: frequency of use of the online system for users exposed to the rankings output by the version (e.g., number of sessions, session length, etc.), recency of content in the rankings (e.g., the average number of minutes, hours, or days since content in a given ranking was created), the rate of content creation in the online system (e.g., the average number of new posts by users exposed to the rankings), and / or usage of features in the online system (e.g., the average number and / or average session length of sessions in the online system with the "company" or "group" feature for users exposed to the rankings).

[0092] Continuing with the above example, if the simplified version has a statistically significantly lower primary performance metric than the baseline version, execution of the simplified version to generate a new ranking for output to users of the online system is stopped, and the simplified version is retrained to have fewer missing features from the baseline version. If the simplified version does not have a statistically significantly lower primary performance metric than the baseline version but has a statistically significantly lower secondary performance metric than the baseline version, the proportion of business handled by the simplified version will be maintained (or selectively reduced), and the number of features to be removed from the baseline version will be reduced during the planned retraining of the next simplified version (e.g., weekly, biweekly, or monthly). If the simplified version does not have a significantly lower primary or secondary performance metric than the baseline version, then the simplified version continues to be added to the production environment according to a predetermined increase or decrease plan and / or increase amount (e.g., an additional 5% for each additional business, which occurs daily or weekly).

[0093] The machine learning model can continue to be simplified (operation 324). During this simplification, performance metrics for the baseline version and the simplified version continue to be calculated and compared (operations 316-318) to verify that the simplified version does not have significantly lower performance than the baseline version. When the simplified version has significantly lower performance than the baseline version, the number of features removed from the baseline version is reduced (operation 320), and the simplified version is retrained with a feature subset that excludes the reduced number of features (operations 308-310). The resource cost of the retrained simplified version is also compared with the target resource cost (operation 312). If the simplified version does not meet the target resource cost, the number of features in the first subset is increased (operation 314), and the simplified version is retrained using a second subset of smaller features (operations 308-310). Thus, such monitoring of the resource cost and performance of the simplified version can continue, adjusting the number of features from the baseline version to be excluded from the simplified version, and / or executing the simplified version until the simplified version is no longer used to simplify the execution of the machine learning model (e.g., when the execution of the machine learning model is terminated and / or the simplified version cannot improve the resource cost of the machine learning model without significantly reducing the performance of the machine learning model).

[0094] Figure 4 A flow chart illustrating a process for calculating importance scores for features used by a machine learning model according to the disclosed embodiments is shown. In one or more embodiments, one or more of the steps may be omitted, repeated, and / or performed in a different order. Thus, Figure 4 The specific arrangement of steps shown should not be construed as limiting the scope of the embodiments.

[0095] First, multiple sets of feature values for a feature and raw rankings output by a machine learning model from the feature value sets are obtained (operation 402). For example, the sets of feature values can be obtained from records in a training, testing, validation, and / or other dataset used to create or evaluate a machine learning model. Each set of feature values can be input into the machine learning model, and a corresponding raw ranking can be obtained as output from the machine learning model.

[0096] Next, one or more feature values in the set of feature values are modified to generate a modified set of feature values (operation 404). For example, the modified set of feature values is created by replacing one or more feature values in the set with default missing values that minimize the impact of the corresponding feature on the output of the machine learning model. In another example, each feature value to be modified is swapped with another feature value of the same feature from a different set of feature values (e.g., a different record in the same dataset). Thus, the modified set of feature values contains the same feature values as the original set of feature values, except for the feature values that have been replaced with default missing values or swapped with other feature values from the different set of feature values.

[0097] The modified set of feature values is input into the machine learning model to generate a modified ranking (operation 406), and a ranking deviation overlap between the modified ranking and the original ranking is calculated (operation 408). As a result, the ranking deviation overlap indicates the extent to which the modified feature values affect the ranking output by the machine learning model.

[0098] Operations 404-408 are repeated while continuing to modify the feature values (operation 410). For example, each feature value (or set of feature values) in a given set of feature values is modified one or more times (operation 404) such that each set of feature values results in multiple modified sets of feature values, and each modified set of feature values generated from the given set of feature values contains a different subset of the modified feature values. Operations 406-408 are then performed for each modified set of feature values to measure the impact of the modified features on the ranking output by the machine learning model. Such modifications to the feature values can be repeated to generate modified sets of feature values based on a certain number of sets of feature values, to generate a certain number of modified sets of feature values based on each set of feature values, and / or to modify a certain number of features or a set of features in each set of feature values.

[0099] Finally, the overlap of the ranking deviations between the modified ranking associated with each feature and the corresponding original ranking is aggregated into an importance score for the feature (operation 412). For example, a similarity score is calculated for each feature by aggregating the overlap of the ranking deviations between the modified ranking generated by the machine learning model after modifying the value of the feature and the corresponding original ranking, and the importance score for the feature is calculated by subtracting the similarity score from 1. As a result, a higher importance score indicates that the feature has a greater impact on the ranking generated by the machine learning model; while a lower importance score indicates that the feature has a smaller impact on the ranking. As described above, the importance score can then be used to selectively remove features from the baseline version of the machine learning model.

[0100] Figure 55. A computer system 500 is shown in accordance with the disclosed embodiments. The computer system 500 includes a processor 502, a memory 504, a storage device 506, and / or other components found in electronic computing devices. The processor 502 can support parallel processing and / or multi-threaded operation with other processors in the computer system 500. The computer system 500 also includes input / output (I / O) devices such as a keyboard 508, a mouse 510, and a display 512.

[0101] Computer system 500 includes the functionality of various components for executing the present embodiment. Specifically, computer system 500 includes an operating system (not shown) that coordinates the use of hardware and software resources on computer system 500, and one or more applications that perform specialized tasks for users. To perform tasks for users, applications obtain access to hardware resources on computer system 500 from the operating system and interact with users through the hardware and / or software framework provided by the operating system.

[0102] In one or more embodiments, the computer system 500 provides a system for simplifying machine learning. The system includes an analysis device, a training device, a simplification device, and a management device, one or more of which may alternatively be referred to as or implemented as a module, mechanism, or other type of system component. The simplification device determines the resource overhead for a baseline version of a machine learning model that uses a set of features to generate entity rankings and the number of features to be removed to reduce the resource overhead to a target resource overhead. Next, the analysis device calculates importance scores for the features, where each importance score represents the impact of the corresponding feature on the entity ranking. The simplification device then identifies a first subset of features to be removed as the number of features with the lowest importance score, and the training device uses a second subset of features excluding the first subset of features to train a simplified version of the machine learning model.

[0103] The management device calculates a first performance metric for the baseline version based on a first set of results associated with the first set of rankings output by the baseline version, and calculates a second performance metric for the simplified version based on a second set of results associated with the second set of rankings output by the simplified version. When a comparison of the first performance metric and the second performance metric indicates that the simplified version does not have a performance that is statistically significantly lower than that of the baseline version, the management device replaces the execution of the baseline version with the execution of the simplified version to generate a new entity ranking.

[0104] Furthermore, one or more components of computer system 500 may be remotely located and connected to other components via a network. Portions of this embodiment (e.g., analysis devices, training devices, simplification devices, management devices, data repositories, model repositories, online networks, etc.) may also be located on different nodes of a distributed system implementing the embodiment. For example, this embodiment may be implemented using a cloud computing system that simplifies the execution of machine learning models on a remote online system.

[0105] The data structures and code described in the detailed description are typically stored on a computer-readable storage medium, which can be any device or medium that can store code and / or data for use by a computer system. Computer-readable storage media include, but are not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact disks), DVDs (digital versatile disks or digital video disks), or other media now known or later developed that can store code and / or data.

[0106] The methods and processes described in the detailed description may be embodied as code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and code and stored in the computer-readable storage medium.

[0107] Furthermore, the methods and processes described herein may be embodied in hardware modules or devices. These modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors (including dedicated or shared processor cores) that execute specific software modules or codes at specific times, and / or other programmable logic devices now known or later developed. When the hardware modules or devices are activated, they will execute the methods and processes embodied therein.

[0108] The foregoing description of various embodiments has been provided for purposes of illustration and description only. They are not intended to be exhaustive or to limit the invention to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. In addition, the above disclosure is not intended to limit the invention.

Claims

1. A method comprising: creating a baseline version of the machine learning model by training the machine learning model using a first set of feature values for the first set of features; deploying the baseline version in an environment, wherein the deployed baseline version outputs a first set of entity rankings; creating a second set of feature values for the first set of features by replacing at least one first value in the first set of feature values with at least one second value different from the at least one first value, wherein the at least one second value is configured to minimize a contribution of the first set of features to an output of the machine learning model; generating a second set of entity rankings by applying the baseline version to the second set of feature values; calculating a similarity score between the first set of entity rankings and the second set of entity rankings; identifying a set of high-importance features based on the similarity scores, wherein the set of high-importance features comprises a subset of the first set of features; creating a simplified version of the machine learning model by training the machine learning model using the set of high-importance features instead of the first set of features; determining a resource cost for the simplified version by executing the simplified version in the environment or in a copy of the environment; and In response to the resource cost satisfying a target resource cost, replacing at least a portion of the deployed baseline version with the simplified version in the environment, wherein the simplified version outputs a second set of entity rankings different from the first set of entity rankings.

2. The method according to claim 1, further comprising: calculating a first performance metric for the baseline version based on a first set of results, the first set of results being associated with a first set of rankings output by the baseline version; calculating a second performance metric for the simplified version based on a second set of results, the second set of results being associated with a second set of rankings output by the simplified version; as well as When a comparison of the first performance metric and the second performance metric indicates that the simplified version has statistically significantly lower performance than the baseline version: removing the feature with the lowest importance score from the set of features; and The simplified version is retrained using a third subset of the features, the third subset of features excluding the feature with the lowest importance score.

3. The method according to claim 2, wherein: Calculating and comparing the first performance metric and the second performance metric includes: outputting the first set of rankings to a first set of users, and outputting the second set of rankings to a second set of users; monitoring the first set of results and the second set of results during use of the online system by the first set of users and the second set of users; and The first performance metric and the second performance metric and a statistical significance of a difference between the first performance metric and the second performance metric are calculated based on the first set of results and the second set of results.

4. The method according to claim 2, wherein: The first performance metric and the second performance metric comprise a ratio of positive results between a set of users and a corresponding set of rankings output by a version of the machine learning model.

5. The method according to claim 2, further comprising: performing additional comparisons of one or more additional performance metrics between the baseline version of the machine learning model and the simplified version of the machine learning model; as well as Prior to retraining the simplified version, the number of features to be removed from the set of features is adjusted based on the additional comparison.

6. The method according to claim 5, wherein: The one or more additional performance metrics include at least one of: Frequency of use of online systems; the recency of content in the entity ranking; the rate of content creation in the online system; and The usage of the feature in the online system.

7. The method of claim 1 , further comprising computing a set of importance scores for the set of features input into the baseline version by: obtaining a first set of feature values for the set of features; Outputting, by the baseline version, an original ranking based on the first set of feature values; modifying one or more feature values in the first set of feature values; inputting the modified one or more feature values into the baseline version; Outputting a modified ranking by the baseline version based on the modified one or more feature values; as well as An importance score for one or more features represented by the modified one or more feature values is calculated using the ranking deviation overlap between the modified ranking and the original ranking.

8. The method according to claim 7, wherein: Modifying the one or more feature values includes at least one of: replacing a first eigenvalue of a feature in the first set of eigenvalues with a second eigenvalue of the feature from a second set of values for the set of features; and A third eigenvalue in the first set of eigenvalues is replaced with a default missing value.

9. The method according to claim 1, wherein Training the simplified version involves: compressing the baseline version; and The compressed baseline version is retrained with the first subset of features removed.

10. The method of claim 1 , further comprising determining a first subset of features having an influence on a ranking of an entity that is less than a threshold influence, the ranking of the entity being indicated by an importance score output by the baseline version, by: for each primary feature in the first subset of features, identifying one or more derived features that depend on the primary feature; and The one or more derived features are added to the first subset of features.

11. The method according to claim 1, wherein The resource overhead includes at least one of the following: Delay; processor utilization; memory usage; and Garbage collection duration.

12. The method of claim 1, wherein: Each of the entity rankings comprises a ranking of content items according to a score output by the machine learning model; and Each of the scores represents a predicted likelihood that a user will perform an action on a content item.

13. A system comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: applying a baseline version of the machine learning model to a first set of feature values for the set of features to produce a first set of entity rankings; creating a baseline version of the machine learning model by training the machine learning model using a first set of feature values for the first set of features; deploying the baseline version in an environment, wherein the deployed baseline version outputs a first set of entity rankings; creating a second set of feature values for the first set of features by replacing at least one first value in the first set of feature values with at least one second value different from the at least one first value, wherein the at least one second value is configured to minimize a contribution of the first set of features to an output of the machine learning model; generating a second set of entity rankings by applying the baseline version to the second set of feature values; calculating a similarity score between the first set of entity rankings and the second set of entity rankings; identifying a set of high-importance features based on the similarity scores, wherein the set of high-importance features comprises a subset of the first set of features; creating a simplified version of the machine learning model by training the machine learning model using the set of high-importance features instead of the first set of features; determining a resource cost for the simplified version by executing the simplified version in the environment or in a copy of the environment; and In response to the resource cost satisfying a target resource cost, replacing at least a portion of the deployed baseline version with the simplified version in the environment, wherein the simplified version outputs a second set of entity rankings different from the first set of entity rankings.

14. The system according to claim 13, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to: determining a resource cost of the baseline version; determining a number of features to remove from the set of features to reduce the resource overhead to a target resource overhead for the machine learning model; calculating a set of importance scores for the set of features, wherein each importance score in the set of importance scores represents an impact of the corresponding feature on the first set of entity rankings; as well as A second set of feature values is identified as features having an importance score indicating an influence on the first set of entity rankings that is less than a threshold influence.

15. The system according to claim 14, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to: calculating a first performance metric for the baseline version based on a first set of results associated with the first set of entity rankings; calculating a second performance metric for the simplified version based on a second set of results associated with the second set of entity rankings; executing the simplified version to generate a third set of entity rankings when a first comparison of the first performance metric and the second performance metric indicates that the simplified version does not have a statistically significantly lower performance than the baseline version; as well as When a second comparison of the first performance metric and the second performance metric indicates that the simplified version has a statistically significantly lower performance than the baseline version, the feature having the lowest importance score is removed from the set of features.

16. The system according to claim 15, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to: performing a third comparison of one or more additional performance metrics between the baseline version and the simplified version; and Based on the third comparison, the number of features to be removed from the set of features is adjusted.

17. The system of claim 14, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to perform operations comprising computing a set of importance scores for a set of features input into the baseline version by: obtaining a first set of feature values for the set of features; Outputting, by the baseline version, an original ranking based on the first set of feature values; modifying one or more feature values in the first set of feature values; inputting the modified one or more feature values into the baseline version; Outputting a modified ranking by the baseline version based on the modified one or more feature values; as well as An importance score for one or more features associated with the modified one or more feature values is calculated using the ranking deviation overlap between the modified ranking and the original ranking.

18. The system of claim 13, wherein: The simplified version is created by: compressing the baseline version; and The compressed baseline version is retrained with the first subset of features removed.

19. The system of claim 13, wherein: The second set of feature values includes: a first set of primary features having an influence on the first set of entity rankings that is less than a threshold influence; and A second set of derived features is dependent upon the first set of primary features.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method comprising: creating a baseline version of the machine learning model by training the machine learning model using a first set of feature values for the first set of features; deploying the baseline version in an environment, wherein the deployed baseline version outputs a first set of entity rankings; creating a second set of feature values for the first set of features by replacing at least one first value in the first set of feature values with at least one second value different from the at least one first value, wherein the at least one second value is configured to minimize a contribution of the first set of features to an output of the machine learning model; generating a second set of entity rankings by applying the baseline version to the second set of feature values; calculating a similarity score between the first set of entity rankings and the second set of entity rankings; identifying a set of high-importance features based on the similarity scores, wherein the set of high-importance features comprises a subset of the first set of features; creating a simplified version of the machine learning model by training the machine learning model using the set of high-importance features instead of the first set of features; determining a resource cost for the simplified version by executing the simplified version in the environment or in a copy of the environment; and In response to the resource cost satisfying a target resource cost, replacing at least a portion of the deployed baseline version with the simplified version in the environment, wherein the simplified version outputs a second set of entity rankings different from the first set of entity rankings.

Images