A system and method for determining the skill factor of a client application.

Abstract

A method, system, and apparatus are provided that include a computer program encoded on a computer storage medium for determining the skill factor of a client application. The method may include: performing a statistical analysis of multiple users who interacted with the client application to generate a first set of users and a second set of users; determining the results of several random matches between users selected from the first set of users and users selected from the second set of users to generate a score indicating how often a user selected from the first set of users wins against a user selected from the second set of users; modifying the client application if the generated score exceeds or is equal to the skill score; and providing the modified client application to multiple users on each client device.

Description

Background Art

[0001] (Cross - reference to Related Applications) This application claims the benefit and priority of a U.S. provisional patent application having U.S. Provisional Patent Application No. 63 / 508,332, entitled "SYSTEM AND METHOD FOR DETERMINING A SKILL FACTOR OF A CLIENT APPLICATION", filed on June 15, 2023. The content of the U.S. provisional patent application having U.S. Provisional Patent Application No. 63 / 508,332 is hereby incorporated by reference in its entirety.

[0002] (Background of the Invention) Many digital games require both skill and luck to play well. What does it mean for chance to be dominant over skill, or vice versa? The extreme cases in the areas of skill and chance are relatively easy to identify. For example, a video game that simulates rolling a die is a pure chance game. Conversely, a video chess game is a pure skill game. Further examples of skill games can include checkers, billiards, and bowling. In contrast, raffles and roulette are pure chance games. Many digital games are composed of a mix of skill and chance. In those cases, the terms skill factor or dominant factor relate to an investigation of the relative proportions of skill and chance. If chance is an essential part that affects performance, then chance is dominant in the digital game. In other words, for skill to be dominant, skill must control the final outcome, and the final outcome must be within the scope of the player's control to some extent. However, such determinations are often subjective, and thus, there can be different reasonable views regarding whether the result of a digital game is dominated by skill or chance.

Summary of the Invention

Means for Solving the Problems

[0003] (Summary of the invention) The present invention relates to a system and method for determining the skill factor of a client application. According to the present invention, a determination can be made regarding which client applications are skill-based and which are chance-based by statistically quantifying the importance of chance when determining the outcome of a client application. In embodiments, the present invention can collect appropriate performance data of a client application and then randomly divide the collected data into two data subsets of equal (or nearly equal) size. The present invention can use a first data subset to create or otherwise identify definitions of skilled and unskilled users by performing a statistical analysis of users who interact with the client application at least a predetermined number of times or otherwise participated. The present invention can use a second data subset to simulate matches between identified high-skilled users and identified low-skilled users of the client application and determine a score indicating, for example, how often the skilled user wins against the unskilled user. The present invention can determine whether a client application is skill-based or chance-based by comparing the determined score with a predetermined score indicating a skill-based client application. If the determined score exceeds or is equal to the predetermined score, the client application can be identified as skill-based. [Brief explanation of the drawing]

[0004] The embodiments described above will be better understood from the following detailed description in conjunction with the attached drawings. The drawings are not intended to be drawn to scale. For clarity, not all components are labeled in all drawings.

[0005] [Figure 1]Figure 1 is a block diagram illustrating an exemplary system for determining the skill factor of a client application.

[0006] [Figure 2] Figure 2 is a flowchart illustrating an exemplary method for determining the skill factor of a client application.

[0007] [Figure 3] Figure 3 is a flowchart illustrating an exemplary method for determining the skill factor of a client application.

[0008] [Figure 4] Figure 4 is a block diagram of an exemplary computing device according to this embodiment that can perform one or more of the operations described herein. [Modes for carrying out the invention]

[0009] (Description of the invention) Herein, in order to provide an overall understanding of the structure, function, manufacturing and use principles of the devices and methods disclosed herein, certain exemplary embodiments are described. One or more examples of these embodiments are shown in the accompanying drawings. Those skilled in the art will understand that the devices and methods specifically described herein and shown in the accompanying drawings are non-limiting and exemplary embodiments, and that the scope of the invention is defined solely by the claims. Features illustrated or described in relation to one exemplary embodiment may be combined with features of other embodiments. Such modifications and variations are intended to fall within the scope of the invention. Furthermore, in this disclosure, components with similar names in embodiments generally have similar features, and therefore, within a particular embodiment, each feature of each similarly named component is not necessarily fully described.

[0010] The present invention relates to a system and method for determining the skill factor of a client application. According to the present invention, it is possible to determine which client applications are skill-based and which are chance-based by statistically quantifying the importance of chance when determining the outcome of a user's interaction or engagement with the client application. In some implementations of the present invention, relevant performance data for the client application can be collected, and the collected data can then be randomly divided into two data subsets of equal (or nearly equal) size. In one embodiment, the present invention can use a first data subset to create or otherwise identify definitions of skilled and unskilled users by performing a statistical analysis of users who have interacted with or otherwise engaged with the client application at least a predetermined number of times. In some implementations of the present invention, a second data subset can be used to simulate matches between identified skilled and unskilled users of the client application and determine a score indicating how often the skilled user wins against the unskilled user. In one embodiment, the present invention can determine whether a client application is skill-based or chance-based by comparing the determined score with a predetermined score indicating a skill-based client application. If the determined score exceeds or is equal to a predetermined score, the client application can be identified as skill-based. Otherwise, the client application can be identified as not skill-based. Such a methodology can be used alone or in combination with one or more further techniques to determine the skill factor of a client application.

[0011] For illustrative purposes only and not limiting, this disclosure refers to digital games as exemplary client applications to illustrate various aspects of the present invention. However, the present invention can be used in and with any suitable type of client application in which a user interacts or otherwise engages with the client application and other users through skills, chances, or a combination thereof.

[0012] Figure 1 is a block diagram illustrating an exemplary system 100 for determining the skill factor of a client application. The server system 114 can provide functionality for collecting data related to the interaction and engagement between the user and the client application (e.g., a player's gameplay in a digital game). The server system 114 may include software components and databases that can be deployed, for example, in one or more data centers 112 located in one or more geographical locations. The software components of the server system 114 may include a skill factor determination engine 116, a first client application skill factor module 118, a second client application skill factor module 120, ..., and an Nth client application skill factor module 122, where N can be any suitable natural number. The software components may include sub-components that can run on the same or different individual data processing devices. The database of the server system 114 may include, for example, a user data database 124 and a client application data database 126, but other databases are also possible. The database may reside in one or more physical storage systems or may be cloud-based. The software components and data are described further below.

[0013] As illustrated in Figure 1, the skill factor determination engine 116 can communicate with the first client application skill factor module 118, the second client application skill factor module 120, ..., the Nth client application skill factor module 122, the user data database 124, and the client application data database 126. In some implementations of the present invention, the skill factor determination engine 116 may include one or more of the first client application skill factor module 118, the second client application skill factor module 120, ..., and the Nth client application skill factor module 122. The user data database 124 may include, for example, user interaction history (from the perspective of digital games, such as which digital games were played, the number of games won in each digital game, the number of games lost in each digital game, the number of games played for each digital game, the score in each digital game, the time spent playing each digital game, etc.), user identification information (e.g., username), history of user connections to system 100, user purchases, user achievements, user tasks, user interactions with other users (e.g., chats), user deposits / withdrawals, user acquisition or use of virtual items, other circumstances in the client application, and any appropriate information regarding one or more users of the client application and their interactions with the client application. The client application data database 126 may include, for example, information about client applications implemented using system 100.The client application data database 126 may include information about each client application, such as the virtual environment of each client application, images, videos, text, and / or audio data of each client application, event data corresponding to past, present, or future events, and client application state data that defines the current state of each client application.

[0014] For example, software applications such as digital games or other web-based or suitable client applications may be provided as end-user client applications to enable users to interact with the server system 114. Software applications can relate to and / or provide a wide variety of functions and information, including, for example, entertainment (e.g., games, music, video), business (e.g., word processing, accounting, spreadsheets), news, weather, finance, sports, etc. In some cases, software applications can provide digital games. Digital games may be, or include, sports games, adventure games, virtual playing card games, virtual board games, puzzle games, racing games, or any other suitable type of digital game. In one embodiment, a digital game may be an asynchronous competitive skill-based game in which players can compete with each other in the digital game, but do not need to play the digital game simultaneously. In an alternative embodiment, a digital game may be a synchronous competitive skill-based game in which players can play the digital game simultaneously and compete with each other in the digital game in real time. Other suitable software applications are also possible.

[0015] The software application or its components can be accessed via the network 110 (e.g., the Internet) by users of client devices such as client device A102, client device B104, client device C106, ..., client device M108, where M can be any suitable natural number. Each client device can be any suitable type of electronic device capable of running the software application and communicating with the server system 114 via the network 110, such as a smartphone, tablet computer, laptop computer, desktop or personal computer. Other client devices are also possible (e.g., portable or desktop game consoles, smart TVs, smartwatches, and other similar computing devices). In alternative embodiments, the user data database 124, the client application data database 126, or any part thereof can be stored on one or more client devices. Furthermore, or alternatively, software components for system 100 (e.g., skill factor determination engine 116, first client application skill factor module 118, second client application skill factor module 120, ..., and / or Nth client application skill factor module 122) or any part thereof can reside on one or more client devices or be used to operate on them.

[0016] Several implementations of the present invention are led to answer the following question: "How often does a skilled user beat an unskilled user?" According to several implementations of the present invention, matches between the "best" user and the "worst" user of a client application can be simulated, and then it is possible to analyze how often the best user defeats or otherwise wins against the worst user. Such information can be used to determine whether the client application is skill-based or chance-based. Figure 2 is a flowchart illustrating an exemplary method 200 for determining the skill factor of a client application using, for example, a skill factor determination engine 116 together with a first client application skill factor module 118, according to embodiments of the present disclosure. In block 205, the skill factor determination engine 116 can select a client application (e.g., from a group of client applications) for skill factor determination (e.g., by retrieving data from a client application data database 126). In block 210, the skill factor determination engine 116 can collect a predetermined number of performance observations of the client application (e.g., digital game performance observations for a digital game) to enable comparisons between users of the client application. In one embodiment, performance observations may be the result of interaction or engagement between the user and the client application. For illustrative purposes only and not limiting, in the context of digital games, performance observations (i.e., digital game performance observations) may be the results of a digital game and may include data such as wins and losses that occurred in the game, the resulting scores, the time played in each game, the players associated with those wins and losses, and / or other similar data.Based on the nature of the client application, the number of performance observations available for analysis, and other similar factors, any appropriate number of performance observations can be collected for the client application, such as 500, 1000, 2500, 5000, 10000, or more. However, in some implementations of the present invention, the skill factor determination performed by the skill factor determination engine 116 using the first client application skill factor module 118 is generally not initiated until a predetermined number of performance observations have been collected for the client application, but the skill factor determination engine 116 using the first client application skill factor module 118 can initiate the analysis at any appropriate time with any appropriate amount of collected data. Performance observations can be stored in, for example, a user data database 124 and collected from there.

[0017] In block 215, the skill factor determination engine 116 can use the first client application skill factor module 118 to randomly separate the collected performance observations or any portion thereof into two data subsets. For example, the first client application skill factor module 118 can separate the collected performance observations into a first data subset having a first size and a second data subset having a second size, although the first client application skill factor module 118 can create any appropriate number of subsets. In one embodiment, the first data subset and the second data subset may be equal in size or approximately equal in size, although each subset may be of any appropriate size. For illustrative purposes only and not limited to the context of digital games, if a player plays 100 games, the skill factor determination engine 116 may use the first client application skill factor module 118 to randomly select 50 scores that should be in the first data subset and 50 scores that should be in the second data subset. Alternatively, the skill factor determination engine 116 may use the first client application skill factor module 118 to randomly select 25 scores that should be in the first data subset and 75 scores that should be in the second data subset, or 75 scores that should be in the first data subset and 25 scores that should be in the second data subset, or it may make any other appropriate allocation of size between the data subsets. The first and second data subsets can be stored in, for example, the user data database 124 and read from there.In one embodiment, a first data subset of collected performance observations (e.g., a first set of interaction performance data corresponding to a first set of game results or gameplay results involving one or more users interacting with the client application) can be used to classify users as skilled or unskilled, and a second data subset of collected performance observations (e.g., a second set of interaction performance data corresponding to further data from a second set of game results or gameplay results involving one or more users interacting with the client application) can be used to analyze the abilities of skilled users (e.g., users in the first set) and unskilled users (e.g., users in the second set) when combined. In other words, the first data subset can be used for training, and the second data subset can be used for validation. According to some implementations of the present invention, the creation of two separate datasets can be used to establish mathematical independence. For example, if the same dataset is used to determine which users are skilled, and then to validate whether skilled users beat or otherwise win against unskilled users, then even a purely chance-based client application may appear to be largely skill-based for some datasets. In contrast, by keeping the data used to characterize users separate from the data used to verify regularities in which skilled users outperform unskilled users, the bias from one step to the next in the exemplary method 200 illustrated in Figure 2 can be minimized.

[0018] In block 220, the skill factor determination engine 116 may use the first client application skill factor module 118 to generate, or otherwise identify, definitions of skilled and unskilled users using a first data subset. In some implementations of the present invention, the first client application skill factor module 118 may use the first data subset to determine which users exhibit the highest skill level by performing appropriate statistical calculations of the performance of users who have interacted with the client application at least a predetermined number of times, or otherwise been involved at least a predetermined number of times, such as the scores of players who have played at least a predetermined number of games in a digital game, in the example of a digital game. In one embodiment, the statistical calculation may be a median calculation. Other appropriate statistical calculations are also possible and may be performed by the first client application skill factor module 118, such as an average calculation, the best or maximum performance or value, or any other appropriate statistical calculation. However, a median calculation can provide a consistent and optimal indicator of what a user is likely to achieve. For example, in an asynchronous single-player digital game where the best player is the one whose score is consistently higher than their opponent's, the margin of victory may not be significant, and outliers that could significantly impact the average score of players should not be of further importance in deriving player skill classifications. Therefore, the use of median calculation may have advantages over other types of statistical calculations and is used solely for the purposes of this disclosure to illustrate various aspects of the present invention.

[0019] According to one embodiment of the present invention, any appropriate number, such as 10, 20, 50, or 100, can be used for a predetermined number of user interactions or engagements. The appropriate value for a predetermined number of user interactions or engagements can be based on a variety of factors. For illustrative purposes only and not limiting, in most skill-based digital games, skills are acquired at least partially through practice. Therefore, since a skilled player is likely to be a player with a minimum amount of experience, the appropriate value for a predetermined number of games can be based on the number of games a player generally needs to play to gain more experience or skill in a particular digital game. Furthermore, according to the present invention, sufficient user data can be collected to accurately determine which users are skilled and which are not. For example, if the first client application skill factor module 118 analyzes a user who interacted, or otherwise engaged, only, for example, twice, the median calculation for that user will be based on a single record (for example, since the dataset can be split in half in block 215). Such performance metrics are equivalent to evaluating a golfer based on a single hole played, or a baseball player based on a single batting stance, and are therefore unlikely to be a reliable means of classifying users. Thus, a predetermined number of user interactions or engagements can be set to an appropriate value that provides the best balance between having a healthy set of users to analyze and ensuring there is enough data to accurately classify users.

[0020] The skill factor determination engine 116 can use the first client application skill factor module 118 to rank the performance of median calculations across all users (e.g., from highest to lowest). A predetermined upper percentage of all users can be defined as "skilled" or otherwise identified by the first client application skill factor module 118. Any suitable percentage, such as 5%, 10%, 15%, 20%, etc., can be used as the predetermined upper percentage. The first client application skill factor module 118 can interact with the client application a predetermined number of times or otherwise evaluate the ranked median performance of the users involved to determine or otherwise identify which users are "unskilled" or otherwise "skill - less" users. In an embodiment, the predetermined number of user interactions or involvements for skilled and unskilled users can be the same value or different values. Using a predetermined number of user interactions or involvements for unskilled users can ensure that unskilled users are not confused with "no - skill - at - all" users because they do not understand how to interact or be involved with the client application. In one embodiment, a predetermined lower percentage of all users can be defined as "skill - less" by the first client application skill factor module 118. Any suitable percentage, such as 5%, 10%, 15%, 20%, etc., can be used as the predetermined lower percentage. In some embodiments, the predetermined upper percentage and the predetermined lower percentage can be the same or different percentages.

[0021] Users who have interacted with or engaged with a client application at least a predetermined number of times may not be truly "skillless," as they are likely to have acquired at least some degree of skill through practice. Since users labeled "skillless" undergo practice at the same or substantially the same level as users labeled "skilled," this may introduce a bias in the analysis of the client application's skill composition performed by the first client application skill factor module 118, towards classifying more client applications as chance-based. Thus, such a potential bias may raise the threshold for classifying client applications as skill-based, which may be acceptable for a large selection of client applications. Alternatively, different percentile breakpoints can be used to define "skilled" and "skillless" users for a given upper and lower percentage, respectively. Choosing broader breakpoints to define skilled and unskilled users may introduce bias into the analysis performed by the first client application skill factor module 118, leading to a classification of more client applications on a chance-based basis. Such potential bias may result in a higher threshold for classifying client applications on a skill-based basis, which may be acceptable for a large selection of client applications.

[0022] In block 225, the skill factor determination engine 116 can use the first client application skill factor module 118 to compare the performance of random matches between skilled and unskilled users using a second data subset. To determine how often a skilled user defeats or otherwise wins against an unskilled user, the first client application skill factor module 118 can use the second data subset to generate a predetermined number of random matches between skilled and unskilled users. For example, any appropriate number such as 1000, 5000, 10000, or 25000 can be used for the predetermined number of random matches. Such matches may be random combinations of one skilled user and one unskilled user. For each match, the first client application skill factor module 118 can select random client application performance for each user in the match. Client application performance can be obtained from the second data subset that was not used to determine whether a user is skilled or unskilled. In one embodiment, the first client application skill factor module 118 can create a match by first selecting a user and then selecting achievements from that user for the client application. In the example of a digital game, the first client application skill factor module 118 can select one random skilled player and one random unskilled player, then select one random score from a second data subset for the skilled player and one random score from the second data subset for the unskilled player.By selecting battles in this way, it is possible to ensure that the probability of obtaining battles from all points on the "skill continuum" (e.g., the 90th to 100th percentiles) and all points on the "non - skill continuum" (e.g., the 0th to 10th percentiles) is equal, and this can be used to avoid biases arising from the frequency of interaction or engagement of users with various levels of strength. To account for such biases, the first client application skill factor module 118 can choose to select a random achievement from all achievements recorded by skilled users and compare it with a random achievement from all achievements recorded by unskilled users. However, the distribution of skill levels in battles is proportional to the number of user interactions or engagements at each skill level, thereby creating an undesirable bias. In the example of a digital game, the most excellent players (99th percentile) may play more than 10 times as many games as players in the 90th percentile, which means that the probability of obtaining a skilled player in each battle from the 99th percentile rather than the 90th percentile is at least 10 times higher.

[0023] For example, by comparing the performance, time, or other win evaluation criteria of a skilled user versus an unskilled user, the skill factor determination engine 116 can use the first client application skill factor module 118 to select a winner of a random match (e.g., the player with the higher score). By reasonably aggregating the performance of a predetermined number of random matches (e.g., the number of wins by the skilled player divided by the total number of matches between the skilled and unskilled players), in block 230, the skill factor determination engine 116 can use the first client application skill factor module 118 to determine a score representing how often a skilled user defeats or otherwise wins against an unskilled user. The score can be expressed as an alphanumeric value (e.g., a numerical score or a letter score), a percentage, or any appropriate value that can be within any reasonable range. In block 235, the skill factor determination engine 116 can use the first client application skill factor module 118 to compare the determined score with a predetermined skill score and determine whether the client application is skill-based or chance-based. If the determined score exceeds or is equal to the predetermined skill score, in block 240, the skill factor determination engine 116 can use the first client application skill factor module 118 to identify or classify the client application as skill-based. However, if the determined score is less than the predetermined skill score, in block 245, the skill factor determination engine 116 can use the first client application skill factor module 118 to identify or classify the client application as random or chance-based (for example, one or more results, such as game results or gameplay results for one or more users utilizing the client application, may be random or chance-based).

[0024] The predetermined skill score can be any appropriate score, such as 600, 700, or 800, out of the maximum possible score (e.g., 1000, 2000, etc.). Alternatively, the predetermined skill score can be any appropriate percentage, such as 60%, 70%, or 80%, or any other reasonable predetermined value. Such a predetermined score depends on various factors, including, for example, which verifications the first client application skill factor module 118 is trying to satisfy. For example, advantage verification and critical factor verification can be used alone or in combination to determine the optimal predetermined skill score for measuring whether a client application is skill-based or chance-based. Advantage verification is the dominant verification when evaluating the presence of a chance element. Even if an activity requires some skill, the chance element is satisfied if chance outweighs skill. Under such verification, there can be a continuum where one side has pure skill and the other side has pure chance. In such a continuum, client applications such as chess are almost on the pure skill side, while conventional slot machines are on the pure chance side. Between these sides of the domain, there are many activities that involve elements of both skills and chance. Critical element verification provides that a particular client application is considered chance-based if it includes chance as a critical element that influences the outcome of user interaction or engagement in and with the client application. Such verification recognizes that while skills may primarily influence the outcome, a client application can be considered chance-based if chance has more than just a contingent effect on the outcome of user interaction or engagement. Specifically, the term “importance” modifies the mathematical accuracy of “dominant element” verification, replacing it with a subjective verification that recognizes a client application as chance-based if, while skills may primarily influence the outcome, the final outcome is substantially dependent on chance.Therefore, client application games can be considered skill (or skill advantage-based) games by selecting a reasonable predetermined skill score that indicates a non-critical level of chance in a client application and in user interaction or engagement with it, satisfying both advantage verification and critical element verification.

[0025] In an alternative embodiment, any or all of the predetermined values ​​described above can be determined dynamically. According to the alternative embodiment, the skill factor determination engine 116 and / or the first client application skill factor module 118 can use appropriate machine learning / artificial intelligence techniques to dynamically select or otherwise choose any or all reasonable values ​​such as, for example, the number of performance observations for the client application, the number of user interactions or engagements in and with the client application, the top percentage of all users, the bottom percentage of all users, the number of random matches, and the skill score. For example, one or more machine learning models can be trained based on data from either or both of the user data database 124 and the client application data database 126. One or more machine learning models can then be used to dynamically select reasonable amounts of each or any of the aforementioned values ​​based on, for example, the characteristics of the client application, user characteristics such as the outcome history of user interactions or engagements in the client application (e.g., in the context of digital games, wins, losses, scores, number of times, etc.), skill factor determinations for similar client applications, and other similar characteristics or data. One or more machine learning models can be updated or otherwise adapted as the characteristics, performance, and other similar data related to the client application and users evolve over time.

[0026] According to one embodiment, if the first client application skill factor module 118 determines that a client application is skill-based, the skill factor determination engine 116 and / or the first client application skill factor module 118 (or other appropriate processing logic of the server system 114) may appropriately modify one or more features or functions of the client application, such as changing or modifying the manner of the graphical user interface of the client application, enabling / disabling features and functions, etc. For illustrative purposes only and not limiting in the context of digital games, the skill factor determination engine 116 and / or the first client application skill factor module 118 may enable paid participation competitions in asynchronous or synchronous competitive digital games that have been determined to be skill-based. Conversely, the skill factor determination engine 116 and / or the first client application skill factor module 118 may disable paid participation competitions in asynchronous or synchronous competitive digital games that have been determined not to be skill-based. However, any appropriate visual or audio element, or other feature or function of the client application, can be modified, enabled, disabled, or otherwise updated or customized by the skill factor determination engine 116 and / or the first client application skill factor module 118, depending on whether the client application is skill-based. For example, changes, modifications, or updates can be made to any or all aspects of the graphical display of the client application (e.g., one or more graphical elements of a digital game, such as any aspect of the "look and feel" of the graphical interface displayed by the digital game), information displayed within the client application, features and functions of the client application, etc.For illustrative purposes only and not limited to the perspective of digital games, the graphical display of a digital game may be updated to display, for example, player incentives, special offers (e.g., limited-time offers or LTOs), advertisements, etc., within or to players within a digital game that has been determined to be skill-based. Alternatively, if a digital game is determined to be skill-based, different player incentives, special offers, advertisements, etc., may be displayed to players within the digital game. Further and / or alternative prizes, rewards, and / or gifts may be displayed or offered to players in the relevant gift store for the digital game if the digital game is determined to be skill-based. For example, players of a skill-based digital game may be offered prizes, rewards, and gifts that differ from those offered to other players of digital games that have not been determined to be skill-based or have not yet been determined to be skill-based. In this way, the menu or list of prizes, rewards, and / or gifts displayed to players in the relevant gift store may be tailored to players of a digital game that has been determined to be skill-based. Furthermore, or alternatively, if a client application is determined to be skill-based, graphic information displayed to the user outside the client application, such as advertisements or suggestions presented to the user on a client device outside the client application, can be updated, modified, or otherwise customized. Other customizations and modifications are also possible for client applications determined to be skill-based.

[0027] According to embodiments of the present invention, the skill factor determination engine 116 can use the first client application skill factor module 118 to analyze simulated matches between skilled and unskilled users in a client application and determine whether the client application is skill-based or chance-based. Once the determination is made, the skill factor determination engine 116 can use the first client application skill factor module 118 to continue analyzing live user interactions or engagements within and with the client application over time (either in real-time, batch, or offline processing, or any combination of both) to update the determination (e.g., analysis of one or more real-time interactions or one or more near real-time interactions). In some embodiments, updating the determination may include updating the client application's designation from skill-based to chance-based, or vice versa. For illustrative purposes only and not limited to the perspective of digital games, the skill factor determination engine 116 can use the first client application skill factor module 118 to automatically analyze the results of actual live matches in a digital game to update the determination of whether the digital game is skill-based or chance-based. For example, in the case of a digital game previously determined to be skill-based, if the results of live game analysis differ from the analysis of simulated matches in a way that suggests the existence of significant further opportunities, then, for example, the skill factor determination engine 116 and / or the first client application skill factor module 118 (or other appropriate processing logic of the server system 114) may restrict or eliminate certain features or aspects of the digital game (e.g., paid competitions), or the entire digital game may be deleted or disabled.

[0028] Embodiments of the present invention can be used with many different types of client applications, such as digital games. Because Method 200 is based on the simulation comparing observed user performance, it can minimize false positives (i.e., indicating that a digital game is skill-based when it is not actually skill-based), although those observed performances may have been obtained under very different conditions. However, in some situations, the present invention may produce false negatives (i.e., failing to indicate that a digital game is skill-based when it is actually skill-based). In some implementations of the present invention, to address such potential problems, Method 200 can be used with one or more further client application skill factor modules (e.g., second client application skill factor module 120, ..., nth client application skill factor module 122) that use, include, or otherwise incorporate alternative skill determination techniques. Figure 3 is a flowchart illustrating an exemplary Method 300 for determining the skill factor of a client application, such as a digital game. For illustrative purposes only and not limiting, Method 300 may utilize the first client application skill factor module 118 and the second client application skill factor module 120 by the skill factor determination engine 116. However, some implementations of the present invention may use any reasonable number of client application skill factor modules in any appropriate order, each supporting similar or different techniques for determining the skill factor of a client application. In Figure 3, the skill factor determination engine 116 may select a client application to determine whether it is skill-based or chance-based (step 305).The skill factor determination engine 116 can communicate with the client application data database 126 to select and retrieve relevant information about the client application. The skill factor determination engine 116 can collect a predetermined number of data, such as performance observations, for the client application to enable a comparison between skilled and unskilled users of the client application (step 310), as described above for method 200 shown in Figure 2. The skill factor determination engine 116 can communicate with the user data database 124 to collect or otherwise retrieve performance observation data. The skill factor determination engine 116 can use the first client application skill factor module 118 to perform method 200 shown in Figure 2 (step 315) by randomly dividing the collected data into two data subsets, using the first data subset to identify skilled and unskilled users, and using the second data subset to compare the performance of random matches between skilled and unskilled users to determine a score indicating how often skilled users win against unskilled users (for example, as illustrated and described with respect to blocks 215 to 230 in Figure 2). The skill factor determination engine 116 can use the first client application skill factor module 118 to determine whether the client application passes method 200 (for example, whether the determined score exceeds or is equal to a predetermined skill score, as illustrated and described with respect to block 235 in Figure 2) (step 320). If the client application passes the (first) method 200, the skill factor determination engine 116 may use the first client application skill factor module 118 to determine that the client application is skill-based (step 325).In one embodiment, a client application determined to be skill-based can be presented or otherwise provided to the user on each client device with limited or no restrictions on user interaction or engagement within the client application (e.g., paid competitions possible in digital games).

[0029] However, if the skill factor determination engine 116 using the first client application skill factor module 118 determines that the client application has not passed the (first) method 200 (for example, the determined score is less than a predetermined skill score, as illustrated and described with respect to block 235 in Figure 2) (step 320), the skill factor determination engine 116 may collect further data of the client application for use by the second client application skill factor module 120 (step 330). The skill factor determination engine 116 may communicate with the user data database 124 to collect or otherwise read out any reasonable further data. In some implementations of the present invention, the skill factor determination engine 116 may communicate with the client application data database 126 to collect or otherwise read out any reasonable further data, either thereon. The type and amount of further data collected by the skill factor determination engine 116 depends on the technique or methodology used by the second client application skill factor module 120. The skill factor determination engine 116 can use the second client application skill factor module 120 to analyze the further data collected from the client application (step 335) and to determine whether the client application is skill-based or opportunity-based.

[0030] In some implementations of the present invention, the second client application skill factor module 120 may use any suitable further or alternative techniques or methodologies for determining the game skill factor of a client application. For example, the second client application skill factor module 120 may use the methodology disclosed in U.S. Patent No. 8,882,576 ('576 Patent), entitled “Determination of Game Skill Factor,” the entirety of which is incorporated herein by reference. The method disclosed in the '576 Patent is based on score variance and can compare user-level score variance with game-level score variance. If user-level score variance accounts for a large proportion of the game-level score variance, the digital game is skill-based. Alternatively, modifications of Method 200 in Figure 2 may be used. For example, instead of separating the data into two subsets or parts, a training portion and a validation portion (as illustrated and described with respect to block 215 in Figure 2, e.g.), the data may be held as a single dataset. A single dataset can be used to compare the scores of a reasonable top percentage of users (e.g., by median score) with the scores of a reasonable bottom percentage of users (e.g., by median score). If the top percentage of users most frequently beat or otherwise beat the bottom percentage of users, the client application can be identified as skill-based, or otherwise classified. Alternatively, or as another variation of Method 200 in Figure 2, instead of initially separating the data into two subsets or parts, the first data subset can be generated by randomly separating the first part from the collected actual observations. The first data subset can be used to create definitions of skilled and unskilled users (e.g., as illustrated and described with respect to block 220 in Figure 2). Once the definitions are created, the data from the first data subset can be returned to the pool of collected actual observations.Next, a second data subset can be generated by randomly separating a second portion from the pool of collected performance observations. The second data subset can be used to compare the performance of random matches between skilled and unskilled users (for example, as illustrated and described with respect to block 225 in Figure 2). Alternatively, as another variation of method 200 in Figure 2, instead of collecting a predetermined number of performance observations from a client application and then randomly separating the data into two data subsets (for example, as illustrated and described with respect to blocks 210 and 215 in Figure 2), a first predetermined number of performance observations can be randomly collected for the first data subset, and a second predetermined number of performance observations can be randomly collected for the second data subset. The first and second predetermined numbers of performance observations can each be any appropriate predetermined number, and can be the same or different predetermined numbers.

[0031] Alternatively, the second client application skill factor module 120 (or another client application skill factor module) may use the Exceptional Player Frequency Likelihood (EPFL) methodology. The EPFL methodology can assess whether a client application is predominantly skill-based by determining whether the observed user results in the client application are better explained by a high-skill client application or a purely luck-based client application. The methodology can be based on the win rate of individual users. For each user, the EPFL methodology can first calculate both the likelihood of observing the user's win rate under the assumption of a high-skill client application and the likelihood of observing the user's win rate under the assumption of a purely luck-based client application. The EPFL methodology can combine the likelihoods (assuming a high-skill client application) across all users and combine the likelihoods (assuming a purely luck-based client application) across all users. If the combined likelihood (assuming a high-skill client application) is greater than the combined likelihood (assuming a purely luck-based client application), the client application being evaluated can be determined to be primarily skill-based.

[0032] The EPFL methodology uses two different thresholds: p max And odds ratios can be used. max The threshold can be used to determine what qualifies a user as an exceptional case. max A threshold can be used to determine how large a sample should be when collected. For example, p maxThe larger the odds ratio threshold, the fewer exceptional users there will be, and therefore more users will need to be reviewed to identify exceptional users, potentially resulting in a larger data sample. The odds ratio threshold can determine whether the element of chance is deemed insignificant for the client application, and as a result, client applications with sufficiently large odds may be considered to have insignificant chances. In some implementations of the present invention, p max The minimum threshold for the odds ratio can be determined, for example, by a second client application skill factor module 120 that performs a simulation of a pure chance client application. Any threshold determined should not, with reasonable confidence, allow a pure chance client application to pass. max And in order to determine the minimum threshold of the odds ratio, the second client application skill factor module 120 can set the number of users and the number of interactions or involvements per user for a pure chance client application match to be simulated. For each user and each client application, the second client application skill factor module 120 can randomly determine whether the user won or lost. The second client application skill factor module 120 sets an appropriate range of p max Using the values, an exceptional player algorithm can be applied and the odds ratio can be recorded. The second client application skill factor module 120 can repeat the random determination and recording steps a predetermined number of times (e.g., 500, 1000, 5000, 10000, etc.). After repeating the two steps a predetermined number of times, the second client application skill factor module 120 can calculate a predetermined upper limit of confidence intervals (e.g., 75%, 85%, 95%, etc.) from the simulation. Based on the results, the second client application skill factor module 120 can calculate p maxAnd the minimum threshold of the odds ratio can be determined. Other techniques and methodologies are possible for use by the second client application skill factor module 120 (or other client application skill factor modules).

[0033] As shown in Figure 3, the skill factor determination engine 116 can use the second client application skill factor module 120 to determine (step 340) whether the client application passes the second technique or methodology implemented for the second client application skill factor module 120. If the client application passes the second technique, the skill factor determination engine 116 can use the second client application skill factor module 120 to determine (step 325) that the client application is skill-based. However, if the skill factor determination engine 116 uses the second client application skill factor module 120 to determine that the digital game did not pass the second technique or methodology implemented for the second client application skill factor module 120 (step 340), further data on the client application can be collected by the skill factor determination engine 116 for use by the second client application skill factor module 120 (step 330). In one embodiment, further data may be collected (step 330), and the client application may be analyzed and validated (steps 335 and 340) until the client application passes a second technique or methodology implemented for the second client application skill factor module 120. Alternatively, the second client application skill factor module 120 may execute the second technique or methodology implemented for the second client application skill factor module 120 for a predetermined number of times (or until a certain amount of time has elapsed) or until the client application passes the second technique or methodology, whichever comes first.As a result, a client application can be considered skill-based if it passes either or both of the first and second techniques implemented by the first client application skill factor module 118 and the second client application skill factor module 120, respectively. Otherwise, if the client application fails both techniques or methodologies, it can be identified or otherwise classified as chance-based rather than skill-based. If a client application is determined to be chance-based, certain aspects of the client application can be restricted or excluded (e.g., disabling paid entry competitions in the context of digital games), or the client application can be disabled or deleted entirely.

[0034] According to an alternative embodiment of the present invention, if a client application fails to pass the first skill factor determination technique (step 320) implemented by the first client application skill factor module 118 or the second skill factor determination technique (step 340) implemented by the second client application skill factor module 120, a plurality of further client application skill factor modules can be used to determine whether the client application is skill-based. Any appropriate number of further client application skill factor modules can be used. For example, each of the plurality of further client application skill factor modules can implement one of the further or alternative techniques or methodologies described above or even in Method 200 of Figure 2. If a client application fails to pass the second technique or methodology implemented for the first client application skill factor module among the plurality of further client application skill factor modules, the client application can be analyzed and verified by another further client application skill factor module among the plurality of further client application skill factor modules (e.g., after appropriate data collection by the skill factor determination engine 116). Such a process can continue through the remaining client application skill factor modules among several further client application skill factor modules, such as until the client application passes the techniques or methodologies implemented for the client application skill factor modules among the further client application skill factor modules, until a predetermined number of trials are performed, or until a predetermined period of time has elapsed.In one embodiment, if a client application fails to pass any of the technologies, further data can be collected for the client application, which can then be re-tested through a series of further client application skill factor modules, such as until a predetermined number of attempts are made, or until a specific predetermined time period has elapsed, until the client application passes any or any combination of technologies or methodologies. Otherwise, the client application can be identified or otherwise classified based on chance rather than skill. If a client application is determined to be chance-based, certain aspects of the client application can be restricted or excluded (e.g., disabling paid entry competitions in the context of digital games), or the client application can be disabled or deleted entirely.

[0035] In some implementations of the present invention, the second client application skill factor module 120 can dynamically select specific techniques or methodologies for analyzing and validating a client application based on, for example, the type of client application being analyzed (e.g., in terms of digital games, synchronous vs. asynchronous, board-based vs. non-board-based, etc.). For illustrative purposes only and not limitation, a board-based determination can be made by (i) calculating the player-level score variance and then averaging the variances between players, (ii) calculating the game-level score variance, and (iii) calculating the ratio of (i) / (ii). If the ratio exceeds or is equal to a predetermined board game ratio (e.g., a ratio close to 1), the digital game can be considered board-based. In contrast, if the ratio is small (e.g., a ratio close to 0), the digital game is not considered board-based. The second client application skill factor module 120 (or any or all of a plurality of further client application skill factor modules) can dynamically select, or otherwise choose, appropriate techniques or methodologies to be used for the client application using appropriate machine learning / artificial intelligence techniques. For example, a machine learning model can be trained on data from either or both of the user data database 124 and the client application data database 126. The machine learning model can then be used to dynamically select appropriate techniques or methodologies for validating the client application based on the client application's characteristics, user interaction or engagement history (e.g., wins, losses, scores, time, etc., in the context of digital games), and other similar characteristics or data. The machine learning model can be updated or otherwise adapted as the characteristics, performance, etc., related to the client application evolve over time.

[0036] In some implementations of the present invention, either or both of the first client application skill factor module 118 and the second client application skill factor module 120 (and any further client application skill factor modules) can continuously analyze the client application over time to update the determination. In the example of a digital game, either or both of the first client application skill factor module 118 and the second client application skill factor module 120 can automatically analyze the results of actual live matches in the digital game to update the determination of whether the digital game is based on skill or chance. In alternative embodiments, the present invention can use a combination of real-time and batch (offline) processing to update such determinations. For example, the first client application skill factor module 118 can be used for batch processing to make a first determination using data stored in and read from a user data database 124. After the first determination is made, the second client application skill factor module 120 (or each or any of several further client application skill factor modules) can be used for real-time processing when the client application is exposed (i.e., “live”) to provide real-time updated analysis as the user is within and interacting with or engaging with the client application. Other configurations of the first client application skill factor module 118 and the second client application skill factor module 120 (and each or any of several further client application skill factor modules) for determining the skill factor of the client application are also possible.

[0037] Figure 4 is a block diagram of an exemplary computing device 400 according to this embodiment, capable of performing one or more of the operations described herein. The computing device 400 may be connected to other computing devices in a LAN, intranet, extranet, and / or the Internet. The computing device 400 may operate as a server machine in a client-server network environment or as a client in a peer-to-peer network environment. The computing device 400 may be provided by a personal computer (PC), a set-top box (STB), a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by that machine. Furthermore, although only a single computing device 400 is shown, the term “computing device” should also be interpreted to include any set of computing devices that execute a set of instructions (or multiple sets) individually or together to perform the methods described herein.

[0038] An exemplary computing device 400 may include computer processing devices 402 (e.g., general-purpose processors, ASICs, etc.), main memory 404, static memory 406 (e.g., flash memory, etc.), and data storage devices 408, which can communicate with each other via a bus 430. The computer processing devices 402 may be provided by one or more general-purpose processing devices, such as microprocessors, central processing units, etc. In exemplary examples, the computer processing device 402 may comprise a composite instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing a processor or combination of instruction sets. The computer processing device 402 may also comprise one or more dedicated processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and network processors. The computer processing device 402 may be configured to perform the operations described herein in accordance with one or more aspects of this disclosure in order to perform the operations and steps described herein.

[0039] The computing device 400 may further include a network interface device 412 that can communicate with the network 414. The data storage device 408 may include a machine-readable storage medium 428 (e.g., a non-temporary computer-readable medium for storing instructions for execution) that can store one or more sets of instructions, such as instructions for performing operations described herein, according to one or more aspects of the present disclosure. The instructions 418 that implement the core logic instructions 426 may also reside entirely or at least partially in the main memory 404 and / or the computer processing device 402 during their execution by the computing device 400, the main memory 404, and the computer processing device 402, which also constitute the computer-readable medium. The instructions may be further transmitted or received over the network 414 via the network interface device 412.

[0040] Although the machine-readable storage medium 428 is shown as a single medium in the exemplary examples, the term “computer-readable storage medium” should be interpreted to include a single or multiple mediums that store one or more sets of instructions (e.g., a centralized or distributed database and / or associated caches and servers). The term “computer-readable storage medium” should also be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for machine execution, causing a machine to perform the methods described herein. Thus, the term “computer-readable storage medium” should be interpreted to include, but not be limited to, solid-state memory, optical media, magnetic media, and the like.

[0041] The subject matter described herein offers numerous technical advantages. For example, the server system 114 can be extended to support simultaneous skill factor determination for a large number of client applications, such as hundreds, thousands, or more client applications, thereby substantially improving the allocation of computer resources. Accordingly, some implementations of the present invention can improve the efficiency and processing power of computer hardware resources (e.g., computer processing and memory) to determine the skill factor of a client application and provide substantially faster skill factor determination time, especially for a large number of client applications. By improving the speed and efficiency of skill factor determination for a large number of client applications, computer hardware resources can be freed more quickly and used for other tasks and processes, resulting in a significant improvement in the utilization of computer resources.

[0042] The subject matter and embodiments of operation described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, or one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein can be implemented as one or more modules of computer programs, i.e., computer program instructions, encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively, the program instructions can be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage board, a random-access or serial-access memory array or device, or one or more combinations thereof. Furthermore, although the computer storage medium is not a propagating signal, the computer storage medium can be the source or destination of computer program instructions encoded within an artificially generated propagating signal. Computer storage media can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices), or can be contained within them.

[0043] The operations described in this disclosure can be implemented as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0044] The term “data processing device” encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computer processing devices, computers, systems on a chip, or a combination of the above. Computer processing devices may include one or more processors, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), central processing units (CPUs), or multi-core processors that may contain dedicated logic circuits. In addition to hardware, a device may also include code that creates an execution environment for the computer program in question, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code that constitutes one or more of these. Devices and execution environments can realize a variety of different computing model foundations, such as the foundations for web services, distributed computing, and grid computing.

[0045] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative languages, procedural languages, or functional languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not have to, correspond to files in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language resource), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on one computer or located in one site, or distributed across multiple sites and interconnected by a communication network.

[0046] The processes and logic flows described herein can be executed by one or more programmable processors that run one or more computer programs to perform actions by acting on input data and producing outputs. The one or more programmable processors can be part of at least one computing system. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the devices can also be implemented as dedicated logic circuits.

[0047] Processors suitable for executing computer programs include, for example, both general-purpose and dedicated microprocessors, as well as any one or more processors in any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, optical disks, solid-state drives, etc., or is operablely coupled to them to receive data from them, transfer data to them, or both. However, a computer does not need to have such devices. Moreover, a computer can be incorporated into another device, for example, a smartphone, a mobile audio or media user, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into dedicated logic circuits.

[0048] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or light-emitting diode (LED) monitor, and a keyboard and pointing device, such as a mouse, trackball, touchpad, or stylus, thereby allowing the user to provide input to the computer. Other types of devices can also provide user interaction; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Other possible input devices include touchscreens or other touch-sensor devices such as single-point or multi-point resistive or capacitive trackpads, speech recognition hardware and software, optical scanners, optical pointers, digital image capture devices, and associated interpretation software. In addition, a computer can interact with a user by sending resources to and receiving resources from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0049] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser that a user can interact with in an implementation of the subject matter described herein, or in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communication in any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), peer-to-peer networks (e.g., ad-hoc peer-to-peer networks), and the like.

[0050] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network. The client-server relationship arises thanks to computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from the user). Data generated on the client device (e.g., user interaction history) can be received from the client device by the server.

[0051] One or more computer systems can be configured to perform a specific operation or action thanks to software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform an action while it is running. One or more computer programs can be configured to perform a specific operation or action thanks to containing instructions that cause the data processing device to perform an action when executed by the device.

[0052] Throughout this disclosure, any reference to “one embodiment” or “embodiment” means a specific feature, structure, or characteristic described in relation to an embodiment included in at least one embodiment. Therefore, occurrences of the phrase “in one embodiment” or “in one embodiment” in various places throughout this disclosure do not necessarily all refer to the same embodiment. In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.”

[0053] While this disclosure includes many details of specific implementations, these should not be interpreted as limitations on the scope of any invention or claimable, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this disclosure in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, features may be described above as being implemented in a particular combination, and may be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0054] Similarly, while the operations and / or logical flows are depicted in the drawings and / or described herein in a specific order, this should not be understood as requiring that such operations and / or logical flows be executed in a specific or sequential order shown, or that all shown operations be executed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0055] Thus, specific embodiments of the subject matter are described. Other embodiments are within the scope of the following claims. In some cases, the actions enumerated in the claims may be performed in a different order, and the desired results may still be achieved. In addition, the processes depicted in the accompanying figures do not necessarily require the specific order or sequence shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

[0056] The terms “example” or “exemplary” are used herein to mean that they serve as examples, cases, or illustrations. Any aspect or design described herein as “example” or “exemplary” should not necessarily be construed as being preferable or advantageous to other aspects or designs. Rather, the use of the terms “example” or “exemplary” is intended to specifically present the concepts. The term “or” as used in this application is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or it is evident from the context, “X includes A or B” is intended to mean any of the natural inclusive substitutions. That is, if X includes A, X includes B, or X includes both A and B, “X includes A or B” is satisfied under any of the aforementioned instances. In addition, the articles “a” and “an” as used in this application and the attached claims should generally be construed as meaning “one or more” unless otherwise specified or it is evident from the context that they refer to a singular form. Furthermore, throughout this text, the use of the terms “embodiment,” “one embodiment,” “implementation,” or “implementation” is not intended to mean the same embodiment or implementation unless otherwise stated. In addition, terms such as “first,” “second,” “third,” and “fourth” as used herein are labels to distinguish different elements and do not necessarily have to mean an order that follows their numerical designation.

[0057] In the above description and claims, phrases such as "at least one of..." or "one or more of..." may appear, followed by a conjunctive list of elements or features. The term "and / or" may also appear in lists of two or more elements or features. Such phrases are intended to mean any of the enumerated elements or features individually, or any of the enumerated elements or features in combination with any of the other enumerated elements or features, unless implicitly or explicitly contradicted by the context in which they are used. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or A and B together," respectively. The same interpretation is intended for lists containing three or more items. For example, the phrases “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, and / or C” are intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together,” respectively. In addition, the use of the term “based on” in the foregoing and in the claims is intended to mean “at least partially based on” so as to allow for features or elements that are not enumerated.

[0058] The above description of exemplary implementations of the present invention is not intended to be exhaustive or to limit the invention to the exact embodiments disclosed. Specific implementations and examples of the present invention are described herein for illustrative purposes, but various equivalent modifications are possible within the scope of the invention, as will be recognized by those skilled in the art. The subject matter described herein can be put into practice in systems, apparatus, methods, and / or articles, depending on the desired configuration. The implementations described herein do not represent all implementations that correspond to the subject matter described herein. Rather, they are merely some examples that correspond to embodiments relating to the subject matter described herein. While several variations are described in detail above, other modifications or additions are also possible. In particular, further features and / or variations can be provided in addition to those described herein. For example, the above-described implementations may cover various combinations and partial combinations of the disclosed features, and / or combinations and partial combinations of some of the further features described above. Other implementations may fall within the following claims.

Claims

1. It is a method, The method involves performing a statistical analysis of multiple users who interacted with a client application to generate a first set of users and a second set of users, using a first set of interaction records, wherein the first set of interaction records represents data of a first set of game results related to the multiple users who interacted with the client application. The at least one data processor determines the results of several random matches between one or more users selected from the first set of users and one or more further users selected from the second set of users in order to generate a score indicating how often one or more users selected from the first set of users win against one or more users selected from the second set of users, wherein the second set of interaction performance represents further data of the second set of game results related to the multiple users who interacted with the client application. If the score generated by the at least one data processor exceeds or is equal to the skill score, modify the client application. The at least one data processor provides the modified client application to the multiple users on each client device. Methods that include...

2. The method according to claim 1, further comprising using at least one data processor to select a client application from a plurality of client applications.

3. The method according to claim 1, further comprising using at least one data processor to extract a first set of interaction records representing the data of the first set of game results relating to the plurality of users of the client application, wherein the first set of interaction records is available for classifying the respective skill levels of the plurality of users.

4. The method according to claim 3, further comprising using at least one data processor to extract a second set of interaction records representing the further data of the second set of game results relating to the plurality of users of the client application, wherein the second set of interaction records can be used to analyze the ability of at least one user from the first set of users to at least one user from the second set of users.

5. The method according to claim 1, wherein the client application runs on each of the user's client devices.

6. The method according to claim 1, further comprising determining that if the generated score is less than the skill score, one or more results related to the client application are random, by the at least one data processor.

7. The method according to claim 1, further comprising analyzing one or more real-time interactions between one or more of the multiple users and the client application.

8. The method of claim 7, further comprising updating the designation of the client application based on the analysis of one or more real-time interactions between the client application and one or more of the multiple users.

9. It is a system, At least one data processor, Memory for storing instructions and Equipped with, When the instruction is executed by the at least one data processor, it causes the at least one data processor to perform an operation. The aforementioned operation is, The at least one data processor performs a statistical analysis of multiple users who interacted with a client application to generate a first set of users and a second set of users, wherein the first set of interaction data represents a first set of game result data related to the multiple users who interacted with the client application. The at least one data processor determines the results of several random matches between one or more users selected from the first set of users and one or more users selected from the second set of users in order to generate a score indicating how often one or more users selected from the first set of users win against one or more users selected from the second set of users, wherein the second set of interaction results represents further data of the second set of game results related to the multiple users who interacted with the client application. If the score generated by the at least one data processor exceeds or is equal to the skill score, modify the client application. The at least one data processor provides the modified client application to the user on each client device. A system that includes this.

10. The system according to claim 9, wherein the operation further includes selecting a client application from a plurality of client applications by the at least one data processor.

11. The system according to claim 9, wherein the operation further comprises, by at least one data processor, extracting a first set of interaction records representing the data of the first set of game results relating to the plurality of users of the client application, the first set of interaction records being available for classifying the respective skill levels of the plurality of users.

12. The system according to claim 11, wherein the operation further comprises, by the at least one data processor, extracting a second set of interaction records representing the data of a first set of game results relating to the plurality of users of the client application, the second set of interaction records being available for analyzing the ability of at least one user from the first set of users to at least one user from the second set of users.

13. The system according to claim 9, wherein the client application runs on each of the user's client devices.

14. The system according to claim 9, further comprising the operation by the at least one data processor determining that if the generated score is less than the skill score, one or more results related to the client application are random.

15. The system according to claim 9, wherein the operation further comprises, after determining that the client application is skill-based, analyzing one or more real-time interactions between one or more of the multiple users and the client application.

16. The system according to claim 15, wherein the operation further includes updating the designation of the client application based on the analysis of one or more real-time interactions between the one or more users of the client application and the client application.

17. A non-temporary computer-readable medium for storing instructions, When the instruction is executed by at least one programmable processor, it causes the at least one programmable processor to perform an action. The aforementioned operation is, The method involves performing a statistical analysis of multiple users who interacted with a client application to generate a first set of users and a second set of users, using at least one programmable processor, with the first set of interaction records representing data of a first set of game results related to the multiple users who interacted with the client application. At least one data processor determines the results of several random matches between one or more users selected from the first set of users and one or more users selected from the second set of users in order to generate a score indicating how often one or more users selected from the first set of users win against one or more users selected from the second set of users, wherein the second set of interaction performance represents further data of the second set of game results related to the multiple users who interacted with the client application. If the score generated by the at least one data processor exceeds or is equal to the skill score, modify the client application. The at least one data processor provides the modified client application to the user on each client device. Non-temporary computer-readable media, including [specific media].

18. The non-temporary computer-readable medium according to claim 17, wherein the operation further comprises the selection of a client application from a plurality of client applications by the at least one data processor.

19. The non-temporary computer-readable medium according to claim 17, further comprising analyzing one or more real-time interactions of one or more users among the plurality of users relating to the client application.

20. The non-temporary computer-readable medium according to claim 19, further comprising updating the designation of the client application based on the analysis of one or more real-time interactions of one or more users among the plurality of users relating to the client application.

Images