Data-driven online score caching for machine learning

Abstract

The disclosed embodiments provide a system for processing scoring requests. During operation, the system matches an identifier for an entity in a scoring request with a cache entry in a score cache. Next, the system retrieves from the cache entry a previous value of a score generated by a machine learning model based on previous values of features of the entity, as well as a first encoded representation of previous values of a subset of the features that have an influence on the score greater than a threshold. The system then compares the first encoded representation with a second encoded representation of the latest values of the subset of features of the entity. When the comparison indicates that the latest value matches the previous value, the system outputs the previous value of the score for the entity in a response to the scoring request.

Description

Technical Field

[0001] The disclosed embodiments relate to machine learning and, more particularly, to techniques for performing data-driven online score caching for machine learning. Background Art

[0002] Analytics is typically used to discover trends, patterns, relationships, and / or other properties related to large collections of complex, interconnected, and / or multidimensional data. The discovered information is then used to derive insights and / or guide decisions or actions related to the data.

[0003] To gather such insights, large datasets of features are analyzed using regression models, artificial neural networks, support vector machines, decision trees, naive Bayes classifiers, and / or other types of machine learning models. The discovered information can then be used to guide decisions and / or perform actions related to the data. For example, the output of a machine learning model is used to guide marketing decisions, assess risk, detect fraud, predict behavior, and / or customize or optimize the use of an application or website.

[0004] However, during the creation and training of machine learning models for analytics, a significant amount of time, effort, and overhead is spent on feature selection. For example, a dataset used for a machine learning model may have thousands of features, including features created from combinations of other features, while only a subset of features and / or combinations may significantly contribute to the output of the machine learning model. Furthermore, training and / or executing machine learning models with a large number of features and / or large datasets typically requires more memory, computing resources, and time than those with fewer features or datasets.

[0005] The use of complex machine learning models with large data sets and / or a large number of features presents additional challenges in online environments where machine learning outputs are generated in real time or near real time (e.g., in response to recent user actions or newly received data). For example, an online system uses a machine learning model to generate scores that are used to select and display products, services, goods, and / or other entities to a user as the user interacts with the online system. When a request for these scores is received, a service in the online system retrieves features as input to one or more machine learning models, applies the machine learning models to the features to generate scores, and returns the scores in response to the request. Additionally, the service is required to respond to the request within latency constraints specified in a service level agreement (SLA) for the service. As a result, as the number of requests and / or the number of scores generated in response to each request increases, the service may have difficulty scaling and / or meeting latency requirements.

[0006] Thus, machine learning and / or analytics may be facilitated by mechanisms for improving resource consumption, latency, and / or scalability associated with large-scale and / or online scoring via machine learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A schematic diagram of a system according to the disclosed embodiments is shown.

[0008] Figure 2 A system for processing rating requests according to the disclosed embodiments is shown.

[0009] Figure 3 A flow chart illustrating the processing of a rating request according to the disclosed embodiments is shown.

[0010] Figure 4 A computer system according to the disclosed embodiments is shown.

[0011] In the drawings, like reference numerals refer to the same drawing elements. DETAILED DESCRIPTION

[0012] The following description is presented to enable any person skilled in the art to make and use the embodiments, and is provided in the context of a specific application and its requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the present invention is not limited to the embodiments shown, but should be given the widest scope consistent with the principles and features disclosed herein.

[0013] Overview

[0014] The disclosed embodiments provide a method, apparatus, and system for processing rating requests for scores generated by a machine learning model, such as a relevance score representing a predicted likelihood of a positive outcome between users, jobs, content, goods, services, companies, schools, businesses, and / or other entities.

[0015] In some embodiments, the scoring request is generated online or on-demand, which requires that a response to the scoring request be generated in real time or near real time. When the response is not received in a timely manner, the accuracy of the output generated based on the score and / or the user experience associated with the output is negatively impacted. For example, after a user performs a job search in an online system, a scoring request is received for a relevance score between the user and a set of posted or described jobs. If the scoring request does not receive a response within a specified timeout, the job posting is displayed to the user without a relevance score, which reduces the quality or relevance of the job to the user. Conversely, a longer timeout allows job postings to be output to the user based on the relevance score, but may cause the user to experience a noticeable delay before the job posting is loaded.

[0016] In order to speed up the online processing of scoring requests for scores from a machine learning model, a cached score generated by the machine learning model at an earlier time is selectively returned in response to the scoring request. When a scoring request for a score from a machine learning model is received, one or more entities in the request (e.g., a user and a collection of published jobs) are matched to entries in one or more caches. The entry stores the previous value of the score generated by the machine learning model for the entity (e.g., during processing of a previous scoring request for the same entity), as well as the previous values of the features used by the machine learning model to generate the previous value of the score. If an entry for the entity is not found in the cache(s), the score is calculated by the machine learning model using the latest values of the features and returned in response to the scoring request.

[0017] If a cache entry for the entity is available, one or more comparisons involving data from the cache(s) are performed to determine a scoring method for processing the scoring request. First, the previous value of the most important feature from the cache entry is compared with the latest value of the same feature for the entity. If the previous value differs from the latest value, the latest value of the feature is used to calculate the score, and the score is returned in response to the scoring request.

[0018] If the previous value is the same as the latest value, one or more time-to-live (TTL) values associated with the previous value of the feature and / or score are compared with the current time to determine whether any previous value has expired. If no previous value has expired, the previous value of the score is returned in response to the scoring request rather than recalculating the score using the latest value of the feature. If any previous value has expired, the score is calculated using the latest value of the feature and returned in response to the scoring request.

[0019] By serving cached scores previously generated by a machine model during the processing of a scoring request for scores, the disclosed embodiments reduce computational overhead, resource consumption, and / or latency compared to conventional techniques that execute machine learning models to generate a new set of scores each time a scoring request is received. At the same time, periodic invalidation of cached scores is enforced by comparing the latest feature values associated with the scoring request with the previous feature values used to generate cached scores and / or TTLs for the previous feature and / or score values. As a result, the cached scores returned in response to the scoring request are verified to be relatively recent, which maintains the accuracy and / or relevance of recommendations, insights, and / or other outputs generated based on the scores. Thus, the disclosed embodiments improve computer systems, applications, user experiences, tools, and / or techniques related to executing machine learning models and / or processing requests.

[0020] Online score caching for machine learning

[0021] Figure 1 Schematic diagram of a system according to the disclosed embodiment is shown. Figure 1 As shown in , the system includes an online network 118 and / or other user communities. For example, the online network 118 includes an online professional network that is used by a collection of entities (e.g., entity 1 104, entity x 106) to interact with each other in a professional and / or business environment.

[0022] The entities include users who use the online network 118 to establish and maintain professional connections, list work and community experiences, endorse and / or recommend each other, search and apply for jobs, and / or perform other actions. The entities also or alternatively include companies, employers, and / or recruiters who use the online network 118 to list jobs, search for potential candidates, provide business-related updates to users, advertise, and / or perform other actions.

[0023] The online network 118 includes a profile component 126 that allows the entity to create and edit a profile containing information about the entity's professional and / or industry background, experience, summary, job titles, projects, skills, etc. The profile component 126 also allows the entity to view the profiles of other entities in the online network 118.

[0024] The profile component 126 may also or alternatively include mechanisms for assisting the entity in completing its profile. For example, the profile component 126 may suggest industries, skills, companies, schools, publications, patents, certifications, and / or other types of attributes to the entity as potential additions to the entity's profile. The suggestions may be based on predictions of missing fields, such as predicting the entity's industry based on other information in the entity's profile. The suggestions may also be used to correct existing fields, such as correcting the spelling of a company name in a profile. The suggestions may also be used to clarify existing attributes, such as changing an entity's job title of "Manager" to "Engineering Manager" based on the entity's work experience.

[0025] The online network 118 also includes a search component 128 that allows the entity to search the online network 118 for people, companies, jobs, and / or other work- or business-related information. For example, the entity can enter one or more keywords into a search field to find profiles, job postings, job candidates, articles, and / or other information that includes and / or otherwise matches the keyword(s). The entity can additionally use an "advanced search" feature in the online network 118 to search for profiles, jobs, and / or information by category (such as first name, last name, title, company, school, position, interests, relationships, skills, industry, group, salary, experience level, etc.).

[0026] The online network 118 also includes an interaction component 130 that allows the entities to interact with each other on the online network 118. For example, the interaction component 130 can allow a user, company, school, or other entity to add other entities as connections, follow other entities, send and receive emails or messages with other entities, join groups, and / or interact with (e.g., create, share, reshare, like, and / or comment on) posts from other entities.

[0027] Those skilled in the art will appreciate that the online network 118 may include other components and / or features. For example, the online network 118 may include a homepage, a landing page, and / or a content feed that provides the entity with the latest posts, articles, and / or updates from the entity's connections and / or groups. Similarly, the online network 118 may include features or mechanisms for recommending connections, job postings, articles, and / or groups to the entity.

[0028] In one or more embodiments, data related to an entity's profile and activity on online network 118 (e.g., data 1 122, data x 124) is aggregated into data repository 134 for subsequent retrieval and use. For example, each profile update, profile view, connection, follow, post, comment, like, share, search, click, message, interaction with a group, address book interaction, response to a recommendation, purchase, and / or other action performed by an entity in online network 118 is recorded and stored in a database, data warehouse, cloud storage, and / or other data storage mechanism that provides data repository 134.

[0029] The data in the data repository 134 is then used to generate recommendations and / or other insights related to listings of jobs or opportunities within the online network 118. For example, one or more components of the online network 118 may record searches, clicks, views, text inputs, conversions, and / or other feedback during an entity's interaction with a job search tool in the online network 118. The feedback may be stored in the data repository 134 and used as training data for one or more machine learning models, and the output of the machine learning model(s) may be used to display and / or otherwise recommend jobs, advertisements, postings, articles, connections, products, companies, groups, and / or other types of content, entities, or actions to registered users of the online network 118.

[0030] More specifically, the data in the data repository 134 and one or more machine learning models are used to generate rankings of candidates associated with jobs or opportunities listed within or outside of the online network 118. Figure 1 , the identification mechanism 108 identifies candidates 116 associated with an opportunity. For example, the identification mechanism 108 identifies the candidates 116 as users within or outside of the online network 118 who have viewed, searched for, and / or applied for jobs, positions, roles, and / or opportunities. The identification mechanism 108 also or alternatively identifies the candidates 116 as registered users of the online network 118 whose skills, work experience, and / or other attributes or qualifications match the corresponding jobs, positions, roles, and / or opportunities.

[0031] After identifying the candidates 116, the profile and / or activity data of the candidates 116 and the characteristics and / or features of the corresponding opportunity (e.g., required or desired skills, education, experience, industry, title, etc.) are input into (one or more) machine learning models. The (one or more) machine learning models then output a score that identifies the strengths of the candidate 116 relative to the opportunity and / or qualifications associated with the opportunity (e.g., skills, current position, previous position, overall qualifications, etc.). For example, the (one or more) machine learning models generate the score based on the similarity between the candidate's profile data from the online network 118 and the description of the opportunity. The (one or more) models further adjust the score based on social and / or other verification of the candidate's profile data (e.g., recognition of skills, recommendations, achievements, awards, patents, publications, reputation scores, etc.). A ranking is then generated by sorting the candidates 116 in descending order of score.

[0032] In turn, ranking based on the scores and / or associated insights improves the quality of candidates 116, the recommendations of opportunities for candidates 116, and / or the recommendations of candidates 116 for opportunities. Such rankings may also or alternatively increase user activity on the online network 118 and / or guide decisions by candidates 116 and / or moderators (e.g., hiring managers, recruiters, human resources professionals, etc.) regarding screening or placement of opportunities. For example, one or more components of the online network 118 may display and / or otherwise output a user's position in a ranking of candidates for jobs (e.g., top 10%, top 20%, etc. of 138) to encourage the user to apply for jobs for which the user ranks highly. In a second example, during the sorting of jobs as search results in response to a user's job search, the component(s) may consider the user's relative position in the ranking of a set of jobs. In a third example, the component(s) may output the ranking of candidates 116 for a given set of job qualifications as search results to the recruiter after the recruiter performs a search using the job qualifications included as parameters of the search. In a fourth example, the component(s) may output jobs to the user based on the predicted relevance or attractiveness of the job to the user and / or the likelihood that the user will apply for the job.

[0033] In one or more embodiments, the online network 118 includes functionality to generate scores and / or rankings of recommendations related to candidates 116, opportunities, and / or other entities in real time, near real time, or on demand. In these embodiments, the scores and / or rankings are generated in response to recent activities or actions performed by or involving the entity.

[0034] For example, the online network 118 includes a service that uses a machine learning model to generate a set of relevance scores that represent the compatibility of a user with a set of jobs (or the compatibility of a set of users as candidates 116 for jobs). The service receives a score request in response to a user's job search (or a recruiter's search for candidates 116 that match jobs), a user's access to a job recommendation component in the online network 118, and / or a user's interaction with job-related functionality in the online network 118 in other ways. To improve the accuracy of the score, the service inputs features reflecting the user's (or recruiter's) recent activity into the machine learning model, such as features representing the user's searches, clicks, likes, dislikes, and / or other actions performed in the same user session in which the user interacted with the job-related functionality. In response to the input features, the machine learning model calculates the relevance score between the user and the job (or the set of candidates 116 and the job) in real time or near real time (e.g., with latency within limits specified in a service level agreement (SLA) for the service). The service returns relevance scores in response to rating requests, and the online network 118 outputs a ranking of jobs (or candidates) to the user by descending relevance scores as search results, recommendations, and / or other representations of jobs (or candidates).

[0035] Those skilled in the art will appreciate that the online network 118 can perform on-demand scoring and / or ranking related to other types or combinations of entities. For example, the online network 118 can receive one or more scoring requests for relevance scores between a user and content items created and / or shared by other users of the online network 118. These content items include, but are not limited to, posts, articles, comments, updates, and / or videos. The scoring request(s) can be generated in response to a user accessing a homepage, a landing page, and / or a content feed in another portion of the online network 118. After the relevance scores are generated by one or more machine learning models, one or more rankings of the content items by relevance scores are generated and / or mixed into the content feed for the user. Because the content items are scored and ranked in real time or near real time, the content feed can reflect the latest content posted to the online network 118 and / or the user's interaction with content in the online network 118.

[0036] In another example, in response to a user accessing a “People You May Know” feature in online network 118, online network 118 may generate a scoring request for relevance scores between the user and other registered users of online network 118. The service processes the scoring request by generating relevance scores between the user and other registered users based on features including the user's most recent connection set and / or connection requests in online network 118. The other registered users are then ranked in descending order of relevance scores, and the rankings are output to the user as connection recommendations within the “People You May Know” feature.

[0037] To reduce the latency and / or resource overhead associated with generating these types of scores for a large number of requests and / or entities, the online network 118 processes some or all requests for scores using a cache of scores. Figure 2 As shown in FIG, system 202 for processing scoring requests includes an analyzing device 204 and a processing device 206, both of which are coupled to a feature store 234, a feature cache 236, and a score cache 238. Each of these components is described in more detail below.

[0038] The analysis device 204 responds to the online system (e.g., Figure 1 The rating request 202 is received based on recent activity in the online network 118 of the system. For example, the rating request 202 is generated in response to an interaction between a user of the online system and a component 250 (e.g., search, job search, job recommendations, recruitment, connection recommendations, etc.) that displays results tailored to the user's preferences, interests, experience, or background (e.g., users, jobs, companies, schools, content, advertisements, products, services, etc.). The rating request 202 can be generated by the component 250 and / or a downstream service called by the component 250.

[0039] Scoring request 202 includes parameters 208 for generating results related to recent activity. For example, scoring request 202 may be generated in response to a user accessing or interacting with a job search or job recommendation component in an online system. Parameters 208 of scoring request 202 include, but are not limited to, an identifier of the user, identifiers of one or more jobs potentially relevant to the user, and an optional identifier of a machine learning model used to calculate relevance scores between the user and the jobs. The user's identifier may be obtained from data generated after the user logs into the online system (e.g., by matching the user's authentication credentials with a unique identifier for the user in a registered user data repository). The job identifier may be obtained by matching the title, skills, qualifications, location, and / or other attributes in a job posting or description with corresponding attributes of the user in the user's profile data and / or search terms entered by the user during a job search. The identifier for the machine learning model may be obtained as an identifier of the latest version of the machine learning model used to generate scores between the user and the job, an identifier of a version of the machine learning model personalized for the user, and / or an identifier of a version of the machine learning model to which a user and / or job is assigned in an A / B test.

[0040] The analysis device 204 uses data in the feature store 234, the feature cache 236, and / or the score cache 238 to determine the scoring method 232 for processing the scoring request 202. The feature store 234 includes the latest values of the features used as input into the machine learning model. For example, the feature store 234 includes a service that retrieves the latest feature values mapped to the identifiers in the parameters 208 from a distributed file system and / or other type of data store. The service may also or alternatively generate calls to downstream services that generate the latest feature values on demand.

[0041] The feature cache 236 stores cache entries containing mappings of entity identifiers 222 that can be specified in parameters 208 to previous feature values 226 used to process previous scoring requests for the same parameters 208. For example, the cache entry includes entity identifiers 222 for a user and one or more jobs, which are mapped to previous feature values 226 associated with the user and the job(s). After a previous scoring request containing the identifiers is processed using the previous feature values 226, a cache entry can be created and populated with the previous feature values 226. The cache entries in the feature cache 236 expedite the retrieval of features during the processing of the scoring request 202 and / or other subsequent scoring requests.

[0042] Similarly, the score cache 238 stores cache entries containing a mapping of an entity identifier 220 having the same value as the entity identifier 222 to a previous score 224 generated from a previous feature value 226 during processing of a previous scoring request. The score cache 238 additionally includes an encoding 218 of the previous feature value 226. For example, the encoding 218 includes a hash value, a character string, and a character string. 224 。

[0043] In one or more embodiments, analysis device 204 determines a scoring method 232 based on one or more comparisons 210 of data from feature store 234, feature cache 236, and / or score cache 238. First, analysis device 204 retrieves data by performing a lookup of feature store 234, feature cache 236, and score cache 238 using an identifier in parameter 208. If a cache entry in score cache 238 or feature cache 236 does not include an identifier for parameter 208, analysis device 232 specifies the calculation of a new score value 242 for parameter 208 as scoring method 232. Processing device 206 then applies a machine learning model to the latest values of the features of parameter 208 from feature store 234 to generate score value 242. Processing device 206 then sends a response 240 to scoring request 202 containing score value 242 to component 250 and / or the initiator of scoring request 202.

[0044] If a cache entry exists in the score cache 238 and the feature cache 236 for the identifier in the parameter 208, the analysis device 204 performs a first comparison of the important feature values 228 for the parameter 208 from the feature store 234 with the corresponding previous feature values 226 for the same parameter 208 from the feature cache 236 and / or the score cache 238. In one or more embodiments, the important feature values 228 include the latest values of a set of "most important" features identified as having an impact greater than a threshold on the performance or output of the machine learning model. For example, the most important features include a predetermined number of features that have the greatest impact on the accuracy of the machine learning model and / or a variable number of features that have an impact on the accuracy of the machine learning model that exceeds a threshold.

[0045] In some embodiments, the analysis device 204 and / or another component of the system uses a feature interpretation technique to identify the most important features for the machine learning model. For example, the component determines the importance of each feature of the machine learning model by randomizing the values of the features in the validation data for the machine learning model. Next, the component calculates the difference between the error of the machine learning model with the randomized values in the validation data and the error of the machine learning model with the original feature values in the validation data. The component then ranks the features by decreasing the difference in error, thereby ranking features that have a greater impact on the error before features that have a smaller impact on the error. Finally, the component applies a number, a percentage, and / or another threshold to the difference in error to identify a subset of features as the most important features (e.g., such that the 20% of features that have the greatest impact on the error of the machine learning model are identified as the most important features).

[0046] To compare the significant feature value 228 with the previous feature value 226, the analysis device 204 obtains the encoding 218 of the previous feature value 226 from the score cache 238. The analysis device 204 also applies the same technique (e.g., hash function, embedding model, etc.) used to generate the encoding of the significant feature value 228 to produce a separate encoding 216 of the significant feature value 228. If the encodings 216-218 are different from each other, then one or more significant features of the identifier mapped to the parameter 208 have changed since the previous score 224 was calculated. If the encodings 216-218 are the same, then no significant features have changed since the previous score 224 was calculated. Therefore, the analysis device 204 uses the encodings 216-218 as a compact representation of the significant feature value 226 and the corresponding previous feature value 226. These compact representations can be stored in a fixed amount of memory and / or can be quickly compared to detect changes in the significant feature value 226 since the previous score 224 was calculated.

[0047] If any important features have changed since the previous score 224 was calculated, the analysis device 204 specifies a recalculation of the score as a scoring method 232. The processing device 206 executes the scoring method 232 by inputting the important feature values 228 and the additional feature values 230 (e.g., the latest values of features that are less important to the output of the machine learning model) for the parameters 208 into the machine learning model, obtaining a score value 242 as an output of the machine learning model, and returning the score value 242 in response 240 to the scoring request 202.

[0048] If no significant features have changed since the time the previous score 224 was calculated, the analysis device 204 performs one or more comparisons 210 of the TTL 212 associated with the previous score 224 and / or the previous feature value 226 used to calculate the previous score 224 with the current time 214. In some embodiments, the TTL 212 includes a counter, a timestamp, or other representation of the age of the previous score 224, the previous feature value 226, and / or other data used by the system. If the current time 214 has exceeded the TTL for a particular data segment, the data segment is determined to have expired. Subsequently, if the comparison 210 of the TTL 212 with the current time 214 indicates that the previous score 224 or any previous feature value 226 used to calculate the previous score 224 has expired, the analysis device 204 specifies the calculation of a new score value 242 for the parameter 208 as the scoring method 232.

[0049] If the comparison 210 of the TTL 212 to the current time 214 indicates that the previous score 224 and / or all previous feature values 226 used to calculate the previous score 224 are still valid, the processing device 206 sets the scoring method 232 to reuse the previous score 224. The processing device 206 then retrieves the previous score 224 from the analysis device 204, the score cache 238, and / or another component and returns the previous score 224 as a score value 242 in a response 240.

[0050] In one or more embodiments, the TTL 212 includes a TTL for a previous score 224 and / or one or more TTLs for individual previous feature values 226 and / or groups of previous feature values 226. For example, a previous score 224 includes a TTL that is set to a number of hours, days, and / or other periods after the creation of the previous score 224. Individual features associated with previous feature values 226 have TTLs 212 that vary with the frequency with which the features change (e.g., a feature that measures one hour of user activity may have a shorter TTL than a feature that measures one day or one week of user activity).

[0051] Continuing with the above example, the previous feature values 226 include a group of “least important” features (e.g., features whose impact on the result score generated by the machine learning model is less than a threshold). During the comparison 210 of the TTL 212 and the current time 214, the analysis device 204 sets the TTL of each feature in the “least important” group to the TTL of the previous score 224. As a result, the previous feature values 226 of the least important features expire only when the corresponding previous score 224 expires.

[0052] Continuing with the above example, the previous feature values 226 include another set of "moderately important" features that are between the most important features and the least important features (e.g., features whose impact on the machine learning model falls between two thresholds associated with the most important features and the least important features). Thus, if the most important features include the 20% of features that have the greatest impact on the score generated by the machine learning model, and the least important features include the 20% of features that have the least impact on the score generated by the machine learning model, then the "moderately important" features include the remaining 60% of features that are not included in the most important features or the least important features. During the comparison 210, the analysis device 204 uses the TTL of each feature in the "moderately important" group to determine whether the corresponding previous feature value has expired. If the current time 214 has not exceeded any TTL 212 of the previous feature value 226 in the moderately important group or the TTL of the previous score 224, the previous score 224 is still valid and can be returned as 2 in the response 240 to the scoring request 202. If the current time 214 exceeds one or more TTLs 212 of previous feature values 216 or the TTL of previous scores 224 in the group, the previous scores 224 are invalidated and the processing device 206 generates a new score value 242 for the parameter 208 and returns the new score value 242 in the response 240 .

[0053] After generating the new score value 242, the analysis device 204, the processing device 206, and / or another component of the system replaces the previous score 224 in the cache entry for the parameter 208 in the score cache 238 with the new score value 242. The component also updates the code 218 in the cache entry with the value of the code 216, which was calculated based on the significant feature value 228 used to generate the new score value 242. Similarly, the component replaces the previous feature value 226 in the cache entry for the entity identifier 222 found in the parameter 208 in the feature cache 236 with the significant feature value 228 and / or additional feature value 230 used to generate the new score value 242. The component also updates the TTL 212 to reflect the new previous score 224 and code 218 in the score cache 238 and the new previous feature value 226 in the feature cache 236. As a result, the entry for the entity represented by the parameter 208 in the score cache 238 and the feature cache 236 reflects the latest score value 242 and the feature values used to calculate the score value 242, which allows one or more subsequent scoring requests 202 for the same parameter 208 to be processed using the comparison 210 related to the latest score value 242 and feature values.

[0054] In some embodiments, the TTL 212 for previous scores 224 and / or previous feature values 226, the encoding 218, the size of the score cache 238 and feature cache 236, and / or other factors affecting the functionality of the system are manually tuned and / or set based on rules and / or formulas related to the mode for receiving the scoring request 202, updates to the scores or feature values used in processing the scoring request 202, the size of the score cache 238 and / or feature cache 236, and / or the computing resources available to calculate scores in response to the scoring request. For example, the TTL for the previous score 224 is calculated based on a formula, heuristic, or function that includes as input the size of the score cache 238, the latency or throughput associated with calculating a new score based on available resources, the typical (e.g., average, median, etc.) decrease in model accuracy based on score staleness, and / or the importance of the score to a function or goal related to using the score in the online system (e.g., search result or recommendation accuracy, user engagement, revenue, etc.). In another example, the TTL for a given previous feature value is calculated based on a formula, heuristic, or function that includes as input the impact of the feature on the resulting score, the time period over which the feature changes (e.g., hourly, daily, weekly, etc.), the latency associated with retrieving the most recent value for the feature, and / or the size of the feature cache 236. In a third example, a hash function and / or another method for computing the encodings 216-218 is selected to balance the resource overhead or complexity associated with computing the encoding 218 and the likelihood of collisions between the encodings 216 and 218.

[0055] Typically, these factors are selected or tuned to balance the tradeoff between implementation overhead (e.g., scalability, resource limitations, latency, throughput, etc.) and the accuracy of the scores returned in response to the scoring request 202. A longer list of important feature values 228 results in greater resource and system overhead (e.g., due to more frequent calculation of scores, retrieval of feature values, and / or greater complexity in the calculation of the codes 216-218), but results in "fresher" and more accurate scores. Conversely, a longer list of additional feature values 230 reduces resource and system overhead (e.g., due to less frequent calculation of scores and retrieval of additional feature values 230), but reduces the accuracy or recency of the scores. More complex hash functions or techniques for calculating the codes 216-218 increase resource consumption and / or latency, but reduce the likelihood of collisions between the codes 216 and 218 and the subsequent likelihood of undetected changes to important feature values 228. Longer feature or score TTLs reduce overhead (e.g., by reducing the need to recalculate scores and / or retrieve the latest feature values), but also reduce the accuracy and / or recency of the scores.

[0056] By serving cached scores previously generated by the machine model during the processing of a scoring request for scores, compared to conventional techniques that execute the machine learning model to produce a new set of scores each time a scoring request is received, Figure 2 The system reduces computational overhead, resource consumption, and / or latency. For example, approximately 62% of user-job pairs in the second and subsequent scoring requests resulted in a cache hit in the score cache 238 within an hour of the first scoring request for the same user-job pair. With a maximum latency of 5 ms for reading from the score cache 238 and a 95% latency of 50 ms for calculating a new score, the system incurs an average latency of 40%*(50+5)ms+60%*5ms, or 25 ms, which is approximately half the 95% scoring latency.

[0057] At the same time, periodic invalidation of cached scores is enforced by comparing the latest feature values associated with the scoring request with the previous feature values used to generate the cached scores and / or TTL for the previous feature and / or score values. As a result, the cached scores returned in response to the scoring request are verified to be relatively recent (e.g., within a window starting from the current time or calculated based on the latest important feature values 228), which maintains the accuracy and / or relevance of recommendations, insights, and / or other outputs generated based on the scores. In contrast, techniques that do not enforce TTL for features and / or scores return stale (e.g., older or outdated) scores that do not reflect the latest user activity and / or preferences. These stale scores may cause the same results (e.g., search results, recommendations, etc.) to be repeatedly returned in response to scoring requests for the same user, which reduces the user's ability to discover new results and / or effectively utilize functionality associated with the scoring request and / or results (e.g., search, recommendations, etc.). Therefore, the disclosed embodiments improve computer systems, applications, user experiences, tools, and / or techniques related to executing machine learning models and / or processing requests.

[0058] Those skilled in the art will appreciate that this can be accomplished in various ways. Figure 2 First, the analysis device 204, the processing device 206, the feature repository 234, the feature cache 236, and / or the score cache 238 can be provided by a single physical machine, multiple computer systems, one or more virtual machines, a grid, one or more clusters, one or more databases, one or more file systems, and / or a cloud computing system. The analysis device 204 and the processing device 206 can also be implemented together and / or separately by one or more hardware and / or software components and / or layers.

[0059] Second, the data in feature cache 236 and score cache 238 can be stored and / or retrieved in a variety of ways. For example, feature cache 236 and score cache 238 can be combined into a single cache, or the data in feature cache 236 and score cache 238 can be divided among multiple caches. In another example, feature cache 236 can be omitted if feature repository 234 has sufficient bandwidth or speed to provide the latest feature values for all scoring requests received by the system. In this case, TTL 212 for previous feature values 226 can be omitted, and only the TTL for previous scores 224 can be compared with current time 214 to determine whether the previous scores 224 have expired. In a third example, feature cache 236 and / or score cache 238 can comprise a local cache on the host on which the instance of analysis device 204 and / or processing device 206 is executed, or a distributed cache accessed by all instances of analysis device 204 and / or processing device 206 within the same data center, cluster, and / or configuration center. When the feature cache 236 and / or the score cache 238 include a local cache, a “sticky routing” technique is used to ensure that a series of scoring requests including one or more identical parameters 208 are sent to the same host, allowing the host to process the scoring requests using cached scores, feature values, and / or encodings of feature values associated with those parameters 208.

[0060] Third, the importance of features used by the machine learning model can be determined using various techniques. For example, the impact of features on the output or performance of the machine learning model can be determined via recursive feature elimination, measuring the correlation between individual features and the output, computing feature importance using node impurities of tree-based models, using least absolute shrinkage and selection operators (lasso) to identify important features with regression models, and / or training an interpretable model to approximate the predictions of the machine learning model.

[0061] Figure 3 A flow chart illustrating the processing of a rating request according to the disclosed embodiments is shown. In one or more embodiments, one or more steps may be omitted, repeated, and / or performed in a different order. Figure 3 The specific arrangement of steps shown in should not be construed as limiting the scope of the technology.

[0062] Initially, a scoring request related to an entity and a machine learning model is received (operation 302). For example, the scoring request is generated in response to recent activity by an entity (e.g., a user) in an online system. The scoring request includes one or more identifiers for the entity, one or more additional entities (e.g., jobs, products, services, content, etc.) associated with the recent activity, and / or the machine learning model.

[0063] The scoring request is processed based on the availability of cache entries for the entity and the machine learning model (operation 304). For example, the availability of the cache entry is determined by performing a lookup in a local or distributed cache using an identifier for the entity and / or the machine learning model. If a score for the entity was previously calculated by the machine learning model in response to a previous scoring request involving the same entity and machine learning model, then a cache entry exists for the entity and model. Conversely, if there was no previous scoring request for the entity and model and / or if the previous cache entry for the entity and model has been invalidated, then no cache entry exists for the entity and model.

[0064] If a cache entry for the entity and model is not available, the machine learning model is applied to the latest features for the entity to produce an updated value for the score for the entity (operation 318), and the updated value for the score is output in response to the scoring request (operation 320). A mapping of identifiers for the entity and model to the updated value of the score and an encoding of the updated values of the first subset of features is also stored in the cache entry (operation 322), which allows the cache entry to be retrieved in response to a lookup that includes an identifier for the entity and / or machine learning model.

[0065] If a cache entry for the entity and the model is available (e.g., if the identifier of the entity can be matched to the cache entry), an encoded representation of the previous value of the score for the entity and the previous values of the first subset of features used to generate the previous value is retrieved from the cache entry (operation 306). The previous value of the score can be generated by the machine learning model based on the previous values of the first subset of features and additional features related to the entity before the current time. The encoded representation includes another type of encoding such as a hash value, an embedding, a checksum, and / or a concatenation of the previous values of the first subset of features. The first subset of features includes features that are identified as having an impact on the performance of the machine learning model that is greater than a threshold (e.g., based on a permutation of the first subset of features in validation data for the machine learning model). For example, the first subset of features includes a predetermined number and / or percentage of features that have the greatest impact on the output or error of the machine learning model.

[0066] Next, a comparison is performed between the previous value of the first subset of features for the entity and the latest value of the first subset of features (operation 308) to determine whether the latest value matches the previous value (operation 310). For example, the latest value of the first subset of features for the first entity at the current time is retrieved from the feature repository and a hash of the latest value is calculated. The hash is then compared to the hash of the previous value of the same feature from the cache entry to ensure equality. If the hashes are equal, then the latest value is determined to match the previous value. If the hashes are not equal, then the latest value does not match the previous value.

[0067] If the comparison indicates that the latest value for the first subset of features of the entity is different from the previous value for the first subset of features of the entity, a new score is calculated for the entity, output in response to the scoring request, and stored in a cache entry for the entity along with an encoding of the latest value for the first subset of features (operations 316-320). If the comparison indicates that the latest value is not different from the previous value, a comparison of the current time with one or more TTLs associated with the previous value of the score is performed (operation 312) to determine whether the previous value of the score has expired (operation 314). For example, the TTL includes the TTL for the previous value of the score and / or one or more TTLs for the previous values of the second subset of features used to calculate the score (e.g., features that have less impact on the performance of the machine learning model than the first subset of features).

[0068] If the current time exceeds any TTL, the previous value of the score has expired. As a result, the latest value of the score is generated by the machine learning model (operation 316), output in response to the scoring request (operation 318), and stored in the cache entry along with the encoding of the latest value of the first subset of features (operation 320). If the current time does not exceed any TTL, the previous value of the score has not expired, and the previous value of the score is output in response to the scoring request (operation 322).

[0069] Operations 302-322 may be repeated during processing of requests for scores related to various entities and / or machine learning models (operation 324). For example, operations 302-322 may be used to process online scoring requests for scores generated by machine learning models for various types and / or combinations of entities.

[0070] Figure 4 A computer system 400 is shown in accordance with the disclosed embodiments. The computer system 400 includes a processor 402, a memory 404, a storage device 406, and / or other components found in electronic computing devices. The processor 402 may support parallel processing and / or multi-threaded operations with other processors in the computer system 400. The memory 404 may include a fractional cache (e.g., Figure 2 score cache 238) and feature cache (e.g., Figure 2 Entries in the score cache include a mapping of entity identifiers to previous scores calculated for corresponding entities (e.g., users, jobs, etc.), and entries in the feature cache include a mapping of entity identifiers to previous feature values used to calculate previous scores for corresponding entities. Computer system 400 may also include input / output (I / O) devices such as a keyboard 408, a mouse 410, and a display 412.

[0071] Computer system 400 includes various components for implementing the functionality of the present embodiment. Specifically, computer system 400 may include an operating system (not shown) that coordinates the use of hardware and software resources on computer system 400, as well as one or more applications that perform specialized tasks for users. To perform tasks for users, applications obtain access to hardware resources on computer system 400 from the operating system and interact with users through the hardware and / or software framework provided by the operating system.

[0072] In one or more embodiments, computer system 400 provides a system for processing scoring requests. The system includes an analysis device and a processing device, one or more of which may alternatively be referred to as or implemented as a module, mechanism, or other type of system component. The analysis device matches an identifier for an entity in a scoring request with a cache entry in a score cache. Next, the analysis device retrieves from the cache entry a previous value of a score generated by a machine learning model based on previous values of a set of features for the entity before a current time, and a first encoded representation of previous values of a subset of features that had an influence greater than a threshold on the machine learning model generating the score. The analysis device also retrieves the latest value of a first subset of features for a first entity from a feature repository at the current time, and compares the first encoded representation of the latest value of the subset of features for the entity with a second encoded representation. When the comparison indicates that the latest value of the subset of features matches the previous value of the subset of features and the previous value of the score is valid, the processing device outputs the previous value of the score in response to the scoring request associated with the entity and the machine learning model.

[0073] When the comparison indicates that the latest and previous values of the subset of features do not match and / or the previous value of the score has expired, the processing device applies the machine learning model to the latest values of the set of features for the entity to produce an updated value of the score for the entity. The processing device then outputs the updated value of the score in the response.

[0074] In addition, one or more components of the computer system 400 can be remotely located and connected to other components via a network. Various components of this embodiment (e.g., analysis means, processing means, feature storage, feature cache, score cache, data storage, online network, etc.) can also be located on different nodes of a distributed system implementing the embodiment. For example, this embodiment can be implemented using a cloud computing system that processes score requests for remote users or sets of entities.

[0075] The data structures and code described in the detailed description are typically stored on a computer-readable storage medium, which can be any device or medium that stores code and / or data that can be used by a computer system. The computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact disks), DVDs (digital versatile disks or digital video disks), or other media capable of storing code and / or data, now known or later developed.

[0076] The methods and processes described in the detailed description section may be embodied as code and / or data, which may be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied as data structures and code and stored in the computer-readable storage medium.

[0077] In addition, the methods and processes described herein can be included in hardware modules or devices. These modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors (including dedicated or shared processor cores) that execute specific software modules or code segments at specific times, and / or other programmable logic devices now known or later developed. When the hardware module or device is activated, it executes the methods and processes contained therein.

[0078] The foregoing description of various embodiments has been presented for purposes of illustration and description only. It is not intended to be exhaustive or to limit the invention to the disclosed forms. Therefore, many modifications and variations will be apparent to those skilled in the art. In addition, the above disclosure is not intended to limit the invention.

Claims

1. A method for processing a scoring request for a score generated by a machine learning model, comprising: matching an identifier for the first entity in the first scoring request with a cache entry in the score cache; Retrieve from the cache entry: a previous value of a score generated by the machine learning model before the current time based on previous values of the set of features for the first entity; and a first encoded representation of previous values of a first subset of the features that had an influence greater than a threshold on the machine learning model generating the score; Retrieving, from a feature repository, the most recent values of the first subset of features for the first entity at the current time; performing a first comparison of the first encoded representation with a second encoded representation of the most recent values of the first subset of the features of the first entity; as well as When the first comparison indicates that the most recent value for the first subset of features for the first entity matches the previous value for the first subset of features for the first entity, outputting the previous value of the score for the first entity in a first response to the first scoring request.

2. The method according to claim 1, further comprising: Verifying that the previous value of the score for the first entity has not expired before including the previous value of the score for the first entity in the first response based on a second comparison of the current time and one or more times to live (TTLs) associated with the previous value of the score for the first entity.

3. The method according to claim 2, wherein: The one or more TTLs include at least one of the following: a first TTL for the previous value of the score; and A second TTL of one or more previous values of a second subset of previous values of the features of the first entity that were input into the machine learning model to produce the previous value of the score.

4. The method according to claim 2, wherein: The one or more TTLs include TTLs for one or more of previous values of the second subset of features for the first entity that were input into the machine learning model to generate the previous value of the score.

5. The method according to claim 1, further comprising: performing a second comparison of the previous value of the first subset of features for a second entity with the most recent value of the first subset of features for the second entity; when the second comparison indicates that the most recent value of the first subset of features for the second entity does not match the previous value of the first subset of features for the second entity, applying the machine learning model to the most recent value of the set of features for the second entity to produce an updated value for the score for the second entity; as well as The latest value of the score for the second entity is output in a second response to a second request related to the second entity and the machine learning model.

6. The method according to claim 1, further comprising: performing a second comparison of the previous value of the first subset of features for a second entity with the most recent value of the first subset of features for the second entity; when the second comparison indicates that the most recent value of the first subset of features for the second entity matches the previous value of the first subset of features for the second entity, performing a third comparison of a current time with one or more times to live (TTLs) associated with previous values of the score for the second entity; When the third comparison indicates that the previous value of the score for the second entity has expired, applying the machine learning model to the most recent values of the set of features for the second entity to produce an updated value for the score for the second entity; as well as The latest value of the score for the second entity is output in a second response to a second request related to the second entity and the machine learning model.

7. The method according to claim 6, further comprising: A mapping of an identifier for the second entity to the most recent value of the score for the second entity and an encoding of the most recent value of the first subset of features for the second entity is stored in another cache entry for the second entity.

8. The method according to claim 1, further comprising: The first subset of features is identified as having a large impact on performance of the machine learning model based on the ranking of the first subset of features.

9. The method according to claim 1, wherein Matching the identifier for the first entity in the first scoring request with the cache entry in the score cache includes: A lookup of the score cache is performed based on the identifier for the first entity and an additional identifier for the machine learning model.

10. The method according to claim 1, wherein The score cache includes at least one of the following: local cache; and Distributed cache.

11. The method according to claim 1, wherein Performing the first comparison of the first encoded representation with the second encoded representation of the most recent values of the first subset of features of the first entity comprises: obtaining a first hash of the previous values for the first subset of features of the first entity from the encoded representation; computing a second hash of the most recent values of the first subset of features of the first entity as a representation of the second encoding; and The first hash is compared to the second hash.

12. The method according to claim 1, wherein The first entity includes at least one of the following: Users; and Work.

13. A system for processing a scoring request for a score generated by a machine learning model, comprising: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to: matching an identifier for the first entity in the first scoring request with a cache entry in the score cache; Retrieve from the cache entry: a previous value of a score generated by the machine learning model before the current time based on previous values of the set of features for the first entity; and a first encoded representation of the previous values of a first subset of the features that had an influence greater than a threshold on the machine learning model generating the score; performing a first comparison of the first encoded representation with a second encoded representation of most recent values of the first subset of the features of the first entity; as well as When the first comparison indicates that the latest value of the first subset of features for the first entity matches the previous value of the first subset of features for the first entity, outputting the previous value of the score for the first entity in a first response to the first scoring request.

14. The system according to claim 13, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to: Verifying that the previous value of the score for the first entity has not expired before including the previous value of the score for the first entity in the first response based on a second comparison of the current time with one or more times to live (TTLs) associated with the previous value of the score for the first entity.

15. The system according to claim 14, wherein: The one or more TTLs include at least one of the following: a first TTL for said previous value of said score; as well as A second TTL of one or more of the previous values for a second subset of features of the first entity that were input into the machine learning model to produce the previous value of the score.

16. The system of claim 13, wherein: The memory further stores instructions that, when executed by the one or more processors, cause the system to: performing a second comparison of the previous value of the first subset of features for a second entity with the most recent value of the first subset of features for the second entity; When the second comparison indicates that the most recent value of the first subset of features for the second entity does not match the previous value of the first subset of features for the second entity, applying the machine learning model to the most recent value of the set of features for the second entity to produce an updated value of the score for the second entity; as well as The latest value of the score for the second entity is output in a second response to a second request related to the second entity and the machine learning model.

17. The system according to claim 13, wherein: Retrieving a cached representation of the previous values for the first subset of features of the first entity includes: matching the identifier for the first entity and the machine learning model from the first scoring request with an entry in a cache; and The cached representation is read from the entry.

18. The system of claim 13, wherein: Performing the first comparison of the previous value of the first subset of features for the first entity with the latest value of the first subset of features for the first entity includes: obtaining a first hash of the previous values for the first subset of features of the first entity from a cached representation; computing a second hash based on the most recent values of the first subset of features for the first entity; and The first hash is compared to the second hash.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform a method comprising: matching an identifier for the first entity in the first scoring request with a cache entry in the score cache; Retrieve from the cache entry: a previous value of a score generated by the machine learning model before the current time based on previous values of the set of features for the first entity; and a first encoded representation of the previous values of a first subset of the features that had an influence greater than a threshold on the machine learning model generating the score; performing a first comparison of the first encoded representation with a second encoded representation of most recent values of the first subset of the features of the first entity; as well as When the first comparison indicates that the most recent value for the first subset of features for the first entity matches the previous value for the first subset of features for the first entity, outputting the previous value of the score for the first entity in a first response to the first scoring request.

20. The non-transitory computer-readable storage medium of claim 19, the method further comprising: performing a second comparison of the previous value of the first subset of features for a second entity with the most recent value of the first subset of features for the second entity; When the second comparison indicates that the most recent value of the first subset of features for the second entity does not match the previous value of the first subset of features for the second entity, applying the machine learning model to the most recent value of the first subset of features for the second entity to produce a most recent value of the score for the second entity; as well as The latest value of the score for the second entity is output in a second response to a second request related to the second entity and the machine learning model.

Images