System and method for scoring user reactions to a software program
By recording user actions and metrics in real time and using machine learning algorithms to score user interactions with software programs, the problem of not being able to assess user responses in real time in existing technologies has been solved, enabling accurate prediction of user satisfaction and product optimization.
Patent Information
- Application Number
- CN201980021610.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-03-23
- Filing Date
- 2019-03-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-03-11
AI Technical Summary
Existing user interaction rating systems can only provide ratings after the session ends, and cannot provide real-time assessments of the positive or negative aspects of user responses.
By processing devices to record user actions, determining corresponding metrics in real time, and using machine learning and predictive analytics algorithms to score individual interaction sessions, the system can conduct real-time evaluations in conjunction with user feedback data.
It enables real-time scoring of user interactions with software programs, accurately predicts future user responses, optimizes software product design, and improves user satisfaction.
Smart Images

Figure CN112334882B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Patent Application No. 15 / 934,445, filed on March 23, 2018, entitled “SYSTEMS AND METHODS FOR SCORING USE REACTIONS TO A SOFTWARE PROGRAM,” the contents of which are incorporated herein by reference in their entirety. Background Art Technical Field
[0003] The present description relates generally to monitoring user interaction with a software program, and more particularly, to monitoring a user and determining in real time whether the user's reaction to the software program is positive or negative. Technical Background
[0005] Currently, systems and methods for scoring a user's interaction with a software program in order to determine the user's reaction to the software program only provide a delayed indication of the user's interaction. That is, such systems and methods can only score the user's interaction (i.e., whether it is positive or negative) after the user's session on the software program ends. Summary of the invention
[0006] In one embodiment, a method for scoring an individual interaction session between a user and a software program includes: recording, by a processing device, one or more actions taken by the user with the software program to obtain recorded session data; determining, by the processing device, one or more metrics corresponding to at least one of the one or more actions taken by the user with the software program in real time; measuring, by the processing device, the one or more actions taken by the user based on the one or more metrics to obtain metric data; and providing, by the processing device, the recorded session data and the metric data as input to one or more of a machine learning algorithm and a predictive analytics algorithm. The one or more of the machine learning algorithm and the predictive analytics algorithm scores the individual interaction session.
[0007] In another embodiment, a system for scoring individual interaction sessions between a user and a software program includes a processing device and a non-transitory processor-readable storage medium having one or more programming instructions stored thereon. The one or more programming instructions, when executed, cause the processing device to record one or more actions taken by the user with the software program to obtain recorded session data, determine one or more metrics corresponding to at least one of the one or more actions taken by the user with the software program, measure the one or more actions taken by the user based on the one or more metrics to obtain metric data, and provide the recorded session data and the metric data as input to one or more of a machine learning algorithm and a predictive analytics algorithm. The one or more of the machine learning algorithm and the predictive analytics algorithm score the individual interaction session.
[0008] In yet another embodiment, a non-transitory computer-readable storage medium is operable by a computer to score an individual interaction session between a user and a software program. The non-transitory computer-readable storage medium includes one or more programming instructions stored thereon, the one or more programming instructions for causing a processing device to record one or more actions taken by the user with the software program to obtain recorded session data, determine one or more metrics corresponding to at least one of the one or more actions taken by the user with the software program, measure the one or more actions taken by the user based on the one or more metrics to obtain metric data, and provide the recorded session data and the metric data as input to one or more of a machine learning algorithm and a predictive analytics algorithm. The one or more of the machine learning algorithm and the predictive analytics algorithm scores the individual interaction session.
[0009] These and other features provided by the embodiments described herein will be more fully understood in view of the following detailed description, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The embodiments illustrated in the drawings are illustrative and exemplary in nature and are not intended to limit the subject matter defined by the claims. The following detailed description of the illustrative embodiments may be understood when read in conjunction with the following drawings, in which like structures are represented by like reference numerals, and in which:
[0011] Figure 1 schematically depicts an illustrative computing network of a system for scoring a user's interaction with a software program in real time in accordance with one or more embodiments shown and described herein;
[0012] Figure 2 Schematically depicts the Figure 1A server computing device further illustrating hardware and software that may be used in scoring a user's interaction with a software program in real time according to one or more embodiments shown and described herein;
[0013] Figure 3 depicts a flow chart of an illustrative method of acquiring recorded session data, determining metrics, and measuring the metrics in accordance with one or more embodiments shown and described herein;
[0014] Figure 4 depicts a flow for an illustrative general determination of a product success score in accordance with one or more embodiments shown and described herein;
[0015] Figure 5 schematically depicts a flow chart of an illustrative method of constructing a decision tree in accordance with one or more embodiments shown and described herein;
[0016] Figure 6 schematically depicts an illustrative decision tree in accordance with one or more embodiments shown and described herein;
[0017] Figure 7 depicts an illustrative method of traversing a decision tree to determine a product success score in accordance with one or more embodiments shown and described herein;
[0018] Figure 8 depicts another illustrative decision tree in accordance with one or more embodiments shown and described herein;
[0019] Fig. 9 An illustrative screenshot depicting a product success score report according to one or more embodiments shown and described herein; and
[0020] Fig.10 Depicted is a flow chart of an illustrative method of monitoring in real time and determining predictive indicators of user interaction with a software program in accordance with one or more embodiments shown and described herein. DETAILED DESCRIPTION
[0021] There is a need for systems and methods that can provide predictive indicators of user interaction with software programs in real time. Therefore, generally with reference to the accompanying drawings, the embodiments described herein are directed to systems, methods, and computer-readable media for measuring the interaction of users with software products in real time to determine the user's satisfaction with the software product. The systems, methods, and computer-readable media described herein measure the user's interaction by recording user interaction sessions and determining and observing one or more metrics used as indicators of user satisfaction. The observed metrics are then combined with the user's post-session feedback (if any) to obtain a score. This score can then be aggregated with other scores, and the aggregate can be used to determine how the user responds to a particular feature of the software product, and can be used to predict how the user may respond in the future (to the same feature or a proposed new feature).
[0022] Some testing methods for evaluating user interaction with a software product may utilize A / B testing, which may also be referred to as split testing. A / B testing involves comparing two versions of a software program (e.g., web pages or websites) to see which version performs better. More specifically, two variants of the software program (e.g., variant A and variant B) are compared by providing the two variants to similar users at substantially the same time. The variant that exhibits the most positive user interaction is generally considered the best variant and may be selected as the user's only software program going forward.
[0023] A / B testing has become increasingly popular as a method for optimizing software products and achieving improved customer satisfaction (e.g., which can lead to enhanced revenue and conversion rates). However, the implementation of A / B testing is fraught with challenges that can lead to erroneous conclusions and suboptimal product optimization.
[0024] The A / B testing process initially defines a feature / metric that needs improvement as measured by an attribute such as enhanced usage. The next step is to deliver different designs of that feature to a non-overlapping sample of customers using the live product, and measure the target feature usage. During this process, the experiment can be tightly controlled to ensure that gains / losses are attributed to a single product change. At the end of the experiment, a design is recommended that results in an improvement in the target metric.
[0025] Although changes adopted in a product can lead to improvements in the usage of a target feature, what is often observed in practice is that this improvement is primarily at the expense of other features. Essentially, high usage of a target feature cannibalizes the usage of other features, which leads to local improvements at the expense of overall holistic improvements, which focuses on suboptimal design optimizations.
[0026] One way to overcome the impact of dialing is to precisely define an overall customer success metric that can be measured along with other features / metrics during the A / B testing process. Essentially, this enables experiments to be conducted for local improvements while also ensuring that the local improvements do not come at the expense of overall improvements. That is, if a change that results in a local feature improvement through dialing lowers the overall customer satisfaction metric, then that change can be ignored. Such an overall improvement can also be referred to as the maximum improvement in user satisfaction.
[0027] Some solutions to this problem have involved defining an overall overall customer success metric. However, such metric definitions have been hand-crafted, relying on the background knowledge of experts to select a prioritized list of metrics and develop some ratio of such metrics to each other. For example, time to first process a document (TTFED) is an overall metric that measures how quickly a user can process a document after performing a search. The processing of a document is weighted differently for different actions or use of features (such as download, send email, print action, etc.). The exact weights used for different features are hand-crafted and rely on the judgment of experts. Similarly, the time to success (TTS) metric has also been utilized as an overall metric.
[0028] The present disclosure uses machine learning to define a primary and overall score for user success, rather than multiple metrics derived from multiple different user behaviors. As will be described in more detail herein, all measured metrics are aggregated, and a machine learning algorithm picks up and selects a combination of metrics that define success and failure. The precise definition of success and failure is determined by a live, real-life in-product customer survey, which is used as an input to a predictive machine learning algorithm. This overall approach may be referred to herein as Sigma Methodology, which results in a Product Success Score (PSS), also referred to as a Sigma Score.
[0029] The Sigma methodology and Product Success Score described here have multiple uses and advantages in web analytics (including AB testing) and in the application of business intelligence and analytics.
[0030] As used herein, the term "real time" generally refers to tasks accomplished by the various systems, methods, and computer-readable media described herein while a user is interacting with a software program. That is, the systems, methods, and computer-readable media described herein are capable of obtaining information about a user's interaction with a software program while the user is actively using the software program, particularly information related to whether the user's interaction is positive, negative, neutral, etc., which information can be determined (in real time or subsequently) using one or more metrics as described in more detail herein.
[0031] Now referring to the accompanying drawings, Figure 1 An illustrative computing network of components of a system for scoring a user's interaction with a software program according to the embodiments shown and described herein is depicted. Figure 1 As shown, computer network 10 may include a wide area network (WAN) (such as the Internet), a local area network (LAN), a mobile communication network, a public service telephone network (PSTN), a personal area network (PAN), a metropolitan area network (MAN), a virtual private network (VPN), and / or another network. Computer network 10 may generally be configured to electronically connect one or more computing devices and / or components thereof. Illustrative computing devices may include, but are not limited to, user computing device 12a, server computing device 12b, and administrator computing device 12c.
[0032] The user computing device 12a may generally function as an interface between a user and other components connected to the computer network 10. Thus, the user computing device 12a may be used to perform one or more user-oriented functions, such as receiving one or more inputs from a user or providing information to the user, as described in more detail herein. Thus, the user computing device 12a may include at least a display and / or input hardware, as described in more detail herein. In some embodiments, the user computing device 12a may include software being evaluated, as described herein. Additionally, Figure 1 Included in the server computing device 12b is an administrator computing device 12c. In the event that the server computing device 12b requires supervision, updates, or corrections, the administrator computing device 12c may be configured to provide the required supervision, updates, and / or corrections. The administrator computing device 12c may also be used to input additional data into the data corpus stored on the server computing device 12b.
[0033] The server computing device 12b may receive data from one or more sources, generate data, store data, index data, search data, and / or provide data to the user computing device 12a in the form of a software program, a questionnaire, or the like.
[0034] It should be understood that while user computing device 12a and administrator computing device 12c are depicted as personal computers and server computing device 12b is depicted as a server, these are non-limiting examples. More specifically, in some embodiments, any type of computing device (e.g., mobile computing device, personal computer, server, etc.) may be used for any of these components. Additionally, while each of these computing devices may be used in Figure 1 1 and 12. In the example, the user computing device 12a, the server computing device 12b, and the administrator computing device 12c are shown as a single piece of hardware, but this is by way of example only. More specifically, each of the user computing device 12a, the server computing device 12b, and the administrator computing device 12c may represent multiple computers, servers, databases, components, etc.
[0035] Figure 2 Depicted from Figure 1 The server computing device 12b further illustrates a system for scoring the user's interaction with the software program in real time. Figure 2 The components depicted in FIG. 1 are described with respect to server computing device 12 b, but it should be understood that similar components may also be used with user computing device 12 a and / or administrator computing device 12 c without departing from the scope of the present disclosure. Figure 1 ).
[0036] According to the embodiments shown and described herein, the server computing device 12b may include a non-transitory computer-readable medium for searching and providing data implemented as hardware, software, and / or firmware. Although in some embodiments, the server computing device 12b may be configured as a general-purpose computer with the necessary hardware, software, and / or firmware, in other embodiments, the server computing device 12b may also be configured as a special-purpose computer specifically designed to perform the functions described herein. In embodiments where the server computing device 12b is a general-purpose computer, the methods described herein generally provide methods for improving things that reside entirely within the realm of computers and the Internet (i.e., improving the functionality of software).
[0037] Likewise Figure 2 As shown, the server computing device 12b may include a processor 30, input / output hardware 32, network interface hardware 34, a data storage component 36 (which may store session data 38a, user activity data 38b, signal score data 38c, product success score (PSS) data 38d, and other data 38e), and a non-transitory memory component 40. The memory component 40 may be configured as a volatile and / or non-volatile computer-readable medium, and therefore, may include random access memory (including SRAM, DRAM, and / or other types of random access memory), flash memory, registers, compact disks (CDs), digital versatile disks (DVDs), and / or other types of storage components. Additionally, the memory component 40 may be configured to store various processing logics, such as, for example, operation logic 41, session logic 42, monitoring logic 43, investigation logic 44, prediction logic 45, and / or reporting logic 46 (each of which may be implemented as a computer program, firmware, or hardware, as examples). The local interface 50 is also included in Figure 2 and may be implemented as a bus or other interface to facilitate communication between components of the server computing device 12b.
[0038] Processor 30 may include any processing component configured to receive and execute instructions (such as from data storage component 36 and / or memory component 40). Input / output hardware 32 may include a display, keyboard, mouse, printer, camera, microphone, speaker, touch screen and / or other devices for receiving, sending and / or presenting data (e.g., devices that allow a user to interact directly or indirectly with server computing device 12b). Network interface hardware 34 may include any wired or wireless networking hardware, such as a modem, LAN port, wireless fidelity (Wi-Fi) card, WiMax card, mobile communication hardware and / or other hardware for communicating with other networks and / or devices.
[0039] It should be understood that the data storage component 36 may reside locally and / or remotely from the server computing device 12b and may be configured to store one or more pieces of data and selectively provide access to the one or more pieces of data. Figure 2 As shown, data storage component 36 may store session data 38a, user activity data 38b, signal score data 38c, PSS data 38d, and / or other data 38e, as described in greater detail herein.
[0040] Included in the memory component 40 are operating logic 41, session logic 42, monitoring logic 43, survey logic 44, prediction logic 45, and / or reporting logic 46. The operating logic 41 may include an operating system and / or other software for managing components of the server computing device 12b. The session logic 42 may provide a software product to a user in the form of an interactive session (such as a research session, etc.), as described in more detail herein. The monitoring logic 43 may monitor the user's interaction with the software product during the interactive session, and determine one or more metrics for performance determination and / or prediction from the user's interaction, as described in more detail herein. After the user has interacted with the software program, the survey logic 44 may provide the user with a post-software experience survey. The prediction logic 45 may predict the user's response to the software based on historical data. The reporting logic 46 may provide data to one or more users, wherein the data relates to an evaluation of the interaction between the user and the software program.
[0041] It should be understood that Figure 2 The components shown in are illustrative only and are not intended to limit the scope of the present disclosure. More specifically, although Figure 2 The components in are illustrated as residing within the server computing device 12b, but this is a non-limiting example. In some embodiments, one or more of these components may reside outside the server computing device 12b. Similarly, as previously described herein, although Figure 2With respect to server computing device 12b, however other components, such as user computing device 12a and administrator computing device 12c, may include similar hardware, software, and / or firmware.
[0042] As mentioned above, regarding Figure 2 The various components described may be used to perform one or more processes and / or provide functionality for evaluating a user's interaction with a software program. Figure 3 Illustrative examples of various processes are described below.
[0043] At step 305, a software program to be evaluated may be provided. The software program may generally be provided in any manner and is not otherwise limited by the present disclosure. For example, the software program may be provided via the Internet, such as, for example, a web-based document search engine software product provided to users via the Internet. As another example, the software program may be a legal research software program that provides users with the ability to search and / or find legal documents (including cases, papers, trial transcripts, journal articles, etc.).
[0044] In some embodiments, the software program may require the user to authenticate himself. Identity authentication may be used for a variety of reasons, such as, for example, to verify that the user is authorized to use the software, to ensure that the correct version of the software is provided to the user, to determine one or more demographics for classifying the user, to determine any other groups of which the user may be a member, etc. The user may be authenticated via any authentication method now known or later developed. For simplicity, the present disclosure relates to password-based authentication, but it should be understood that other authentication methods may be used without departing from the scope of the present disclosure. At step 310, password authentication may generally include receiving a login request from the user. For example, the user may navigate to a specific website hosting the software program and click a button, enter a username and / or password, provide a digital certificate, key, etc., or otherwise indicate that he / she wishes to log in to the software program. Therefore, at step 315, it may be determined whether the user's login is authorized. That is, the system may determine whether the user has been fully identified and / or authorized to use the software program. Otherwise, the login may be denied at step 320, and when another login request is received, the process may return to step 310. If the user login is authorized, then the user session can begin at step 325. Starting a user session can typically include granting access rights to the software program to the user. A timestamp or the like can be recorded when the user session is started, thereby accurately recording the amount of time the user interacts with the software program.
[0045] Once a user session is initiated, the user's activities may be recorded at step 330. That is, one or more actions taken by the user using the software program may be recorded. In some embodiments, all of the user's interactions with the software program may be recorded. In other embodiments, only a portion of the user's interactions with the software program may be recorded. Illustrative examples of interactions or other activities that the system may record include, but are not limited to, mouse cursor position / movement, scroll positioning, scroll speed, button clicks, link clicks, typing, search terms entered, documents viewed, portions of text viewed, items downloaded, etc. As part of step 330, time-related data may also be recorded. Illustrative examples of time that may be recorded include, but are not limited to, the time a user logs in and out of the software program, the amount of time to complete a particular task, the amount of time to complete multiple consecutive tasks, etc. As part of step 330, various user-related data may also be recorded. For example, when a user registers login credentials for accessing a software program, they may be required to provide demographic information such as, for example, name, age, gender, job title, company size, the industry in which the user works, etc.
[0046] At step 335, the system may determine in real time one or more metrics corresponding to the user's activities within the software program (i.e., one or more actions taken by the user with respect to the software program). Determining the one or more metrics typically includes observing the user's interaction with the software program in real time and determining which metric in a list of metrics corresponds to the user's interaction. Such determination may be accomplished, for example, by utilizing a reference file or the like that references specific metrics against specific user actions in the software program.
[0047] The one or more metrics may generally correspond to one or more actions taken by the user with the software program. That is, the metrics may indicate certain characteristics of a specific action completed by the user and / or a specific aspect of the interaction between the user and the software product when interacting with the software. Illustrative examples of metrics include, but are not limited to, the number of searches completed by the user, the number of document views completed by the user, the number of reference reports requested by the user, the number of document transfers requested by the user, the number of screeners added and / or removed by the user, the number of searches conducted in a specific content browsing, the duration of the session, the total number of processed documents, the time measured from the beginning of the session until the first document is processed, the discounted cumulative gain (DCG) of the session, the amount of time passed between different searches, the average search page load time, the average document page load time, etc. In some embodiments, the metrics may be grouped into feature categories. Illustrative feature categories may include, but are not limited to, information features, search and discovery features, session navigation features, document use features, research features, time features, errors, and response time features. Illustrative examples of information features may include, but are not limited to, session ID, date and time, and subclasses of clients. Illustrative examples of search and discovery features may include, but are not limited to, the median document position viewed in a session, the average discounted cumulative gain in a session, and the average time to first process a document in a session. Illustrative examples of session navigation features may include, but are not limited to, session length in minutes, usage of features in specific menu items per session, and total number of clicks on any feature per session. Illustrative examples of document usage features may include, but are not limited to, documents accessed via search pages per session, documents accessed via non-search features, and documents downloaded per session. Illustrative examples of research features may include, but are not limited to, citations copied per session and legal topic briefs and legal issue tracking reports generated per session. Illustrative examples of time features may include, but are not limited to, time between events (e.g., maximum value in a session, average value in a session, median value in a session), time between searches, and time between document views. Illustrative examples of errors may include, but are not limited to, number of failed deliveries. Illustrative examples of response time features may include, but are not limited to, search results page load time (including maximum, average, and median times), document page load time (including maximum, average, and median times), and other page load times.
[0048] The determined metrics may be predefined via any method identified for defining metrics. For example, in some embodiments, metrics may be defined via background research on Net Promoter Score (NPS) feedback, internal surveys, and workshops with various teams to identify multiple metrics and / or actions within the software product that may be key factors in the success and failure of user sessions. Illustrative metrics may be derived and selected from expert recommendations and / or team agreements regarding specific metrics. Any metric may be defined provided that the metric is considered to reflect some meaningful combination of one or more user actions in any session using the software product, and may include a record of the software product's response to user actions, provided that user actions can be observed and / or recorded in real time.
[0049] At step 340, user activity may be measured based on the determined metrics. That is, the system may measure one or more interactions performed by the user based on the one or more metrics to obtain metric data. Measuring the one or more interactions performed by the user may include observing user actions within the software program and qualifying and / or quantifying the actions based on the corresponding metrics. Such qualification and / or quantification may allow the system to determine whether the user action is positive, negative, or neutral, which may later be used to determine a product success score, as described in more detail herein.
[0050] Certain user tasks may be assigned greater weights than other user tasks. For example, a user task that is more indicative of measurement success may be weighted greater than a task that is less indicative of measurement success. In another example, a user task involving downloading a document may be weighted greater than another user task involving scrolling a document. This may be due, for example, to the task of downloading a document being more indicative of active user behavior within the software program than the task of scrolling a document. Therefore, to ensure that certain user tasks are given appropriate weights relative to other user tasks, at step 340, the system may weight one or more user tasks as part of measuring the user's activity.
[0051] Measuring the user activity according to step 340 may also include determining user satisfaction with completing one or more tasks. User satisfaction may be related to a specific interaction with the software program (e.g., whether the user is satisfied with the ease of performing a search, the ease of viewing content, etc.), to multiple specific interactions with the software program, and / or to the entire software program. Determining user satisfaction may include analyzing one or more tasks and determining an estimated satisfaction based on the manner in which the user completes the task within the software program, the amount of time the user takes to complete the task, whether the user successfully or unsuccessfully completes the specific task, etc. For example, determining user satisfaction may be based on the amount of time that passes between a user performing a search and the user downloading a document as a result of the search, whether any additional searches are required, the number of times the user must click a "next" button in order to view a set of search results, etc. In some embodiments, determining user satisfaction may be accomplished using input, data, etc. received from a machine learning algorithm that is specifically trained to determine whether a user is satisfied with a specific task. In some embodiments, the user satisfaction may be determined based at least in part on feedback received from the user, as described in more detail herein.
[0052] In various embodiments, the measurements according to step 340 may include differentiating the work-to-utility for each task completed by a user within the software program. For example, a developer of a software program may wish to minimize the amount of work required by a user to complete a task while also maximizing the usefulness of a particular task to a user (e.g., the amount of work required to enter a search term and find relevant documents). That is, if a user is searching for documents within a software program, the developer of the software program may wish to minimize the number of searches that the user must run before finding a particular document, because an increased number of searches run to find a particular document may result in the user being dissatisfied with the interaction with the software program.
[0053] In various embodiments, the measurement according to step 340 further includes measuring the one or more actions taken by the user based on the determined scoring category. The scoring category may be related to demographic information associated with the user. That is, a user with a specific demographic background may be more likely to perform a specific task within the software program, may be more likely to perform a specific task in a specific manner, may be more likely to utilize certain functions of the software program, may be more likely to complete the task faster, etc. than other users with different specific demographic backgrounds. In this way, the system can be specifically configured to identify these demographic factors and measure user activities accordingly. Illustrative examples of demographic data include, but are not limited to, name, age, gender, job title, company size, industry in which the user works, etc. For example, if the user is a lawyer, the demographic information may include the legal field in which the user practices, how long the user has been a lawyer, whether the user is engaged in a single-person practice, a small law firm, a large law firm, a regional law firm, a national law firm, an international law firm, a company's internal legal department, etc., whether the user is a senior lawyer, a partner, an employee lawyer, a general counsel, etc., the specific legal field in which the user practices, etc. The system can determine the user category that the user falls into based on the user's demographic data, categorize the user accordingly, and then generate a score category that can be used for the purpose of appropriately measuring the user's activities.
[0054] As mentioned above, Figure 3 As depicted, the metric may be determined and / or measured in real time according to steps 335 and 340 while a user session is ongoing. However, it should be understood that this is merely illustrative. That is, in other embodiments, the metric may be determined and / or measured according to steps 335 and 340 at other times, such as after the user session is completed. In such embodiments, the metric may be determined and / or measured by evaluating recorded user activity, which may be stored as session data, as described in more detail herein. It should be understood that when such a metric is determined and / or measured after an activity has been completed, such metric determination and / or measurement is still based on actual activity that occurred during the user session, rather than based on activity that occurred after the user session, such as requesting the user to recall his / her interactions with the software program to provide feedback.
[0055] Once the user has completed the session with the software program, the user can log off or sign out of the software. For example, the user can click a "log out" button or similar button, close the software program, close the browser window running the software program, etc. In some embodiments, the software program can determine that the session has been idle for a period of time and can automatically log the user out of the software program, provide prompts about whether the user wishes to continue using the software program, etc. At step 345, a determination can be made as to whether the user wants to log out of the software program. If the user is still actively using the software or has not yet completed a logout request, the process can be repeated at step 330 until a logout request or an affirmative logout determination has been received. At this point, the process can proceed to step 350.
[0056] In some embodiments, the operation logic 41, session logic 42, and / or monitoring logic 43 ( Figure 2 ) to complete one or more parts of steps 325-step 345. In some embodiments, instructions included in other logic (including other logic not specifically described herein) can be used to complete one or more parts of steps 325-step 345.
[0057] In some embodiments, it may be desirable to obtain post-software interaction input from the user in order to quantify various portions of the recorded user activity, determine relevant metrics, determine irrelevant metrics, etc. In such embodiments, a survey request may be sent to the user at step 350. For example, once the user logs out of the software program, a pop-up box or the like may be presented to the user, wherein the pop-up box asks the user whether or not he wishes to participate in a survey regarding the user's experience. Such a request may be particularly useful in embodiments where completion of a post-software interaction session survey is optional.
[0058] At step 355, it may be determined whether the user has consented to the survey. If so, the process may proceed to step 360. If not, the process may proceed to step 375. Such steps will be described in more detail below.
[0059] It should be understood that in some embodiments, after the user session has been completed, the user can be asked to answer the survey. In such embodiments, steps 350 and 355 can be omitted and the process can proceed to step 360 immediately.
[0060] At step 360, a survey may be provided to the user, and at step 365, a response to the survey may be received from the user. The survey is typically provided after the user has completed an interactive session with the software program. Thus, the survey may ask questions about the user's opinions throughout the entire interaction of the user's interaction with the software program. The type of survey and the content contained in the survey are not limited by the present disclosure and may be any survey or content therein. In some embodiments, the survey may ask specific questions about a particular aspect of the software program. In other embodiments, the survey may ask more generalized questions about the software program as a whole. Responses may be received in the form of text, a selection of one or more pre-prepared answers, and the like. Other types of survey responses are contemplated and included within the scope of the present disclosure.
[0061] Although Figure 3 Providing a survey and receiving survey responses are depicted as occurring after the user has completed a session, but the disclosure is not limited thereto. Additionally, the disclosure is not limited to individual question-and-answer sessions. That is, in some embodiments, at least a portion of a survey may be conducted during a user's interaction with the software. For example, a user may complete a particular task while interacting with the software program, which may result in survey questions being presented in response to the completion of such a task. For example, if a user clicks a particular button, a survey may be immediately displayed to the user, asking the user why he or she selected that particular button.
[0062] Once the survey responses are received, the survey responses may be stored at step 370. In some embodiments, the survey responses may be stored as survey data, which may be included in the data storage component 36 ( Figure 2 ), as described herein. For example, the survey responses may be stored as part of the signal score data 38c and / or other data 38e ( Figure 2 ).
[0063] At step 375, once all of the survey questions have been received or if the user declines to complete the survey, the session may end at step 375. It should be appreciated that in some embodiments, the session may be completed using the operational logic 41, session logic 42, monitoring logic 43, and / or survey logic 44 (described herein). Figure 2 ) to complete one or more parts of steps 350-step 375. In some embodiments, instructions included in other logic (including other logic not specifically described herein) can be used to complete one or more parts of steps 350-step 375.
[0064] At step 380, any data generated or obtained from recording the user session may be stored. For example, in some embodiments, a record of a user session including all interactions between a user and a software program may be stored in one or more portions of data storage component 36, including, but not limited to, session data 38a, user activity data 38b, and other data 38e. Figure 2 ). Such data may be stored so that it can be retrieved in the future for analysis, prediction, scoring, etc., as described herein.
[0065] At step 385, any data generated or obtained from determining and / or measuring the one or more metrics may be stored. For example, in some embodiments, data related to the one or more metrics may be stored in one or more portions of the data storage component 36, including but not limited to session data 38a, user activity data 38b, signal score data 38c, and other data 38e ( Figure 2 ). Such data may be stored so that it can be retrieved in the future for analysis, prediction, scoring, etc., as described herein.
[0066] The recorded data can then be used for machine learning and / or predictive analysis purposes, such as, for example, combining the interaction session scores to obtain a combined score, determining overall user satisfaction (e.g., whether the user is generally satisfied with the software program) based on the combined score, determining user satisfaction for a particular category of users, reporting user satisfaction, and / or predicting user satisfaction with improvements to the software program (including maximum improvement, which represents an overall maximum improvement score). Thus, at step 390, the stored data (including survey data, recorded session data, and metric data) can be provided for machine learning and / or predictive analysis. That is, the stored data can be provided as input for one or more machine learning algorithms and / or predictive analysis algorithms. The machine learning and / or predictive analysis can use the stored data to determine the nature of the user's interaction with the software program, including positive interactions, negative interactions, etc., so that the data can be used to determine how the software program should be implemented, as described in more detail herein. For example, in some embodiments, the recorded data can be used to determine a product success score (PSS). Figure 4 An illustrative overall flow diagram of PSS determination in accordance with an embodiment is depicted.
[0067] like Figure 4As shown, at step 410, all of the data obtained from each completed session (i.e., each recorded interaction between the user and the software program, including the combined score) can be retrieved from the memory. For example, the information can be retrieved from one or more portions of the data storage component 36, including but not limited to session data 38a, user activity data 38b, signal score data 38c, and other data 38e ( Figure 2 ). At step 420, the retrieved data may be fed into a supervised learning model for use by the machine learning library to determine the success of the user interaction, including both the interaction as a whole and the specific interactions with various parts or features of the software. One such example of a machine learning library that utilizes a supervised learning model includes, but is not limited to, the Python scikit-learn machine learning library (provided by various sponsors at http: / / scikit-learn.org / stable / ). The supervised learning model used by the machine learning library is not limited by the present disclosure, and may generally be any machine learning model that can be used to predict how a user will respond in a particular situation, such as a proposed software update.
[0068] An illustrative example of a supervised learning model that can be used is a decision tree model. A decision tree model is a computational model that contains multiple branching operations. Each branching operation includes a test or query that asks a question and the tree branches in two or more directions depending on the answer to the question. Figure 5 and Figure 6 The method of generating an illustrative decision tree model and the generated decision tree model are depicted respectively.
[0069] like Figure 5 As shown, session data may be retrieved at step 505. For example, data store 36 ( Figure 2 ) and information can be retrieved therefrom. At step 510, various attributes can be determined from the session data. The attributes are not limited by the present disclosure and can be any attribute of a particular session that is extracted from the particular session. For example, the attribute can be a particular user action, a particular user response, etc. The attribute generally relates to the action, interface response, etc. from which the metric is determined, as previously described herein. Thus, at step 515, the metric is extracted from the attribute. At step 520, the metric is extracted, which can be classified as having one of two results. For example, Figure 6As shown, specific metrics that may have one of two results include instances where the number of searches is less than / not less than 1 (box 605), instances where the session duration is less than / not less than 2 minutes (box 610), instances where the number of document views is less than / not less than 1 (box 615), instances where the number of directory views is less than / not less than 2 (box 620), instances where the session duration is less than / not less than 1 minute (box 630), instances where the number of filters selected is less than / not less than 3 (box 635), and / or instances where the number of emails sent is less than / not less than 2 (box 665).
[0070] Reference again Figure 5 At step 525, a decision tree may be constructed from the selected metrics. Figure 6 In the example, the decision tree can be arranged logically. Therefore, each of the selected metrics can represent a specific node in the tree (e.g., boxes 605, 610, 615, 620, 630, 635, and 665). At each node, there may be two answers, which will result in two branches extending from each node. However, it should be understood that for the sake of brevity, each node shows only two possible answers. That is, for some selected metrics, more than two possible answers (and therefore more than two branches) can be extended from each node. As a result of the generation of the decision tree, the machine learning algorithm described herein can traverse the decision tree in response to the questions raised at each node to ultimately obtain a successful result (e.g., boxes 625, 645, 655, 660, and 675) or an unsuccessful result (e.g., boxes 640, 650, and 670). It should be understood that the decision tree including successful and unsuccessful results is merely illustrative, and the present disclosure is not limited thereto. This article is about Figure 8 Another illustrative example of a decision tree is shown and described.
[0071] Decision trees can be used to accomplish Figure 4 The PSS score is determined, for example, as described Figure 7 More specifically, session data for a particular session may be obtained at step 705, and the decision tree may be applied at step 710 to determine whether the particular session is successful at step 715. That is, the decision tree may be traversed based on information contained within the session data. If traversing the decision tree results in an unsuccessful session, the session may be classified as unsuccessful at step 720. If traversing the decision tree results in a successful session, the session may be classified at step 725.
[0072] The decision tree may be applied to each session completed in a particular time period (e.g., a particular day, a particular hour, etc.). Thus, at step 730, it may be determined whether additional sessions need to be analyzed using the decision tree. If so, the process may return to step 705 for each session. If not, the process may proceed to step 735, and thereby record successful and unsuccessful sessions and calculate the PSS therefrom at step 740. For example, as previously described herein, the PSS may be calculated by subtracting the number of unsuccessful sessions from the number of successful sessions.
[0073] Figure 8 An illustrative decision tree that can be used to purchase a vehicle is depicted, which is depicted only to show how the nodes of the decision tree are determined and arranged in a logical flow. Therefore, it should be understood that Figure 8 The decision tree provided in is only for illustrative purposes of showing how decision trees work and is not related to the embodiments described herein. Such decision trees show the decision path when a user decides to buy a car. For example, a company that provides a software program for buying a car can build such a decision tree based on observing the historical interactions of users with the software. By building such a model, the company can learn the specific matters they need to change to improve their business. In this way, at node 805 (neutral node), a given car under consideration is selected. The next level can determine whether the car has been road tested and whether it has been proven to be safe. If the car has not been road tested and has not been proven to be safe, the historical mark at node 810 (negative node) can indicate that the user will not or is unlikely to buy the vehicle. If the car has been road tested and proven to be safe at node 815 (neutral node), the decision tree can proceed to the next level, where the possibility of purchase is determined based on mileage. If the mileage is low (e.g., less than about 50,000 miles), the decision tree at node 825 can indicate that the user will buy or may buy the car (positive node). If the mileage at node 820 (neutral node) is high (e.g., greater than about 50,000 miles), the decision tree can proceed to the next level where the likelihood of purchase is determined based on the age of the vehicle. If the car is greater than a certain age (e.g., about 5 years) at node 830 (negative node), the historical marker can indicate that the user will not or is unlikely to purchase the vehicle. If the car is less than a certain age at node 835 (positive node), the historical marker can indicate that the user is likely or will purchase the car. Such decision trees can be similarly used to determine the success of one or more interactions of a user with a software program, as mentioned in various embodiments described herein.
[0074] The success of a particular user interaction session with a software program can be identified, for example, by a combination of metrics measured within the software program, the combination of metrics being utilized by the various modeling processes described above (such as decision tree modeling). For example, one of the metrics described herein can be used to determine a value, such as the degree to which a user engages with a software program and / or a particular portion of a software program. In another example, one of the metrics described herein can be used to determine work, such as how much effort a user must expend to collaborate with / navigate through the software program to achieve his / her goals.
[0075] Other examples of supervised learning models include, but are not limited to, support vector machines (SVMs), logistic regression, perceptrons, decision trees (as described herein), random forests, gradient boosted trees, etc. Such supervised learning models are generally understood and will not be described in more detail herein. For each model, one or more hyperparameter values may be tested, such as, for example, grid search, random search, and Bayesian hyperparameter optimization.
[0076] Reference again Figure 4 , at step 430, the number of final result nodes (e.g., positive, neutral, or negative) reached by each user after modeling is combined and summarized. At step 440, a sigma score is determined based on the summarized results of the final nodes reached. The sigma score can be, for example, the percentage of positive nodes reached as final results minus the percentage of negative nodes, where neutral nodes provide no weight. Thus, for 5 user interaction sessions conducted using the software program, where two results were positive final results, one result was a negative final result, and two results were neutral, the sigma score would be 40% (2 out of 5) minus 20% (1 out of 5), which is equal to 20 points.
[0077] like Figure 4 The presented methodology for obtaining the Sigma Score / PSS is illustrative only. Therefore, other methodologies for obtaining the Sigma Score / PSS now known or later developed may also be used without departing from the scope of the present application. In some embodiments, the PSS may be presented as a report, such as in Fig. 9 The report may provide a PSS for one or more functions of the software program and may be further categorized based on various user characteristics, specific time periods, etc. For example, program instructions for generating a report may be included in the report logic 46 ( Figure 2 )Inside.
[0078] It should be understood that as software programs evolve, the learning models must be updated to ensure that they accurately reflect the responses of users to the software programs. Therefore, the process described herein can be used to continuously update the learning models to ensure that the machine learning process has the latest information when making predictions.
[0079] Once data obtained from real-time monitoring of interactions between a user and a software program is used in a machine learning library, the learned machine can be used to predict how future interactions between the user and the software will occur in order to more accurately measure user activity relative to the determined metrics, as described in more detail herein.
[0080] In addition, the data may be used to provide reports on user satisfaction with software programs and / or portions of software programs, to test prospective user satisfaction with updated versions of software programs or new software programs, etc. For example, Fig.10 Illustrative example methods of monitoring and determining predictive indicators of user interactions with a software program are depicted.
[0081] At step 1005, the system may receive a proposed change to a software program, such as a change to a particular interface, the addition or removal of a feature, etc. At step 1010, the system may use machine learning data to implement modeling of the proposed change to the software program. That is, the system may input the proposed change as data into a machine learning algorithm so that the algorithm may predict a particular user response to the proposed change.
[0082] At step 1015, the system can determine the predicted impact of the proposed change based on, for example, the output received from the machine learning. To verify the accuracy of the machine learning and / or to update the machine learning to ensure future accuracy, the proposed change can be implemented to the software program at step 1020 and provided to one or more users at step 1025. Figure 3 The described process provides the software program with results for determining user satisfaction with the proposed change. At step 1030, a comparison of the predicted impact can be completed, and at step 1035, a report on the comparison and / or results can be provided.
[0083] It should be understood that the system, method and computer-readable medium of the embodiments described herein measure the interaction of the user and the software product in real time to determine the user's satisfaction with the software product. More specifically, the system, method and computer-readable medium described herein measure the interaction of the user by recording each individual user interaction session that occurs between the user and the software product. During the recording of the session, the system, method and computer-readable medium described herein determine and observe one or more metrics, which are used as indicators of the user's satisfaction with the software product. The observed metrics are then combined with the user's post-session feedback (if any) to obtain a score. The score can then be summarized with other scores, and the summary can be used to determine how the user responds to the specific functions of the software product, and can be used to predict how the user may respond in the future (to the same function or a proposed new function).
[0084] Although specific embodiments are shown and described herein, it should be understood that other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. In addition, although various aspects of the claimed subject matter have been described herein, these aspects need not be utilized in combination. Therefore, the appended claims are intended to cover all such changes and modifications within the scope of the claimed subject matter.
Claims
1. A method for collecting and processing interaction data for use as input to one or more of a machine learning algorithm and a predictive analysis algorithm, the method comprising: recording, by the processing device, one or more electronic interactions between the user and the software program via the user interface to obtain recorded session data; estimating, by the processing device, in real time, satisfaction of the user with the software program based on one or more metrics determined from at least one of one or more electronic interactions between the user and the software program; measuring, by the processing device, the user's interactions with the one or more electronic devices based on the one or more metrics to obtain metric data; as well as providing, by the processing device, the recorded session data and the metric data as inputs to the one or more of the machine learning algorithm and the predictive analytics algorithm, wherein the one or more of the machine learning algorithm and the predictive analytics algorithm scores individual interaction sessions to determine a nature of one or more electronic interactions between the user and the software program and the satisfaction of the user; combining the scored individual interaction session with a plurality of other scored individual interaction sessions to obtain a combined score; predicting, based on the combined score, one or more improvements to the software program that result in the greatest increase in user satisfaction, the one or more improvements comprising at least one of: an update to the software to replace old software, an update to modify an interface of the software, an update to remove a feature of the software, or an update to add a new feature to the software; as well as The software program is implemented with the one or more improvements to verify the accuracy of the machine learning algorithm and to update the machine learning algorithm.
2. The method of claim 1, further comprising: determining an overall user satisfaction with the software program and one or more of the features of the software program based on the combined score; as well as Reports overall user satisfaction.
3. The method of claim 1, further comprising: determining user satisfaction with the software program and one or more of the features of the software program based on the combined scores for a category of users; and The user satisfaction of the category of users is reported.
4. The method according to claim 1, characterized in that Measuring the one or more actions taken by the user includes one or more of the following: weighting one or more user tasks; determining user satisfaction with completion of the one or more user tasks; as well as Work versus utility is differentiated for each of the one or more user tasks.
5. The method according to claim 1, characterized in that The individual interactive session begins when the user logs into the software program and ends when the user logs out of the software program.
6. The method according to claim 1, characterized in that Measuring the one or more actions taken by the user is based on the one or more scored categories.
7. The method according to claim 1, characterized in that One or more of the machine learning algorithm and the predictive analysis algorithm utilize a decision tree logic model.
8. The method of claim 1, further comprising: receiving, by the processing device, a plurality of scores from the one or more of the machine learning algorithm and the predictive analytics algorithm; A sigma score is determined, by the processing device, from the plurality of scores.
9. A system for scoring individual interaction sessions between a user and a software program, the system comprising: Processing equipment; as well as A non-transitory processor-readable storage medium, the non-transitory processor-readable storage medium comprising one or more programming instructions stored thereon, which, when executed, cause the processing device to perform the method of any one of claims 1-8.
10. A non-transitory computer-readable storage medium operable by a computer to score individual interaction sessions between a user and a software program, the non-transitory computer-readable storage medium comprising one or more programming instructions stored thereon for causing a processing device to perform the method of any one of claims 1-8.
Citation Information
Patent Citations
Systems and methods for scoring user reactions to a software program
US20190295105A1
Method and apparatus for estimating user characteristics based on user interaction data
CN102859967A
Method and system for evaluating user satisfaction with respect to a user session
CN105095334A
Metric for automatic assessment of conversational responses
CN107710192A
Satisfaction metrics and methods of implementation
US20070192163A1
